Setting Bones From Matrix

If you are a new Irrlicht Engine user, and have a newbie-question, this is the forum for you. You may also post general programming questions here.
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Setting Bones From Matrix

Post by kklouzal »

Hello I'm trying to set bones from a 4x4 matrix but I've never worked with matrices before..

I'm not sure how to get from Float4x4 to BoneNode->Set Position/Rotation/Scale.
More specifically I don't know how to manually set the transformation for a bone.

Code: Select all

 
        size_t models = models_.Count();
        size_t bone = 0;
        while (bone < models)
        {
            ozz::math::Float4x4 mat = models_[bone];
            irr::scene::IBoneSceneNode* BoneNode = Node->getJointNode(bone);
            //BoneNode->set
 
            bone++;
        }
ozz::math::Float4x4 (provided as model-space transformation)
// Declare the 4x4 matrix type. Uses the column major convention where the
// matrix-times-vector is written v'=Mv:
// [ m.cols[0].x m.cols[1].x m.cols[2].x m.cols[3].x ] {v.x}
// | m.cols[0].y m.cols[1].y m.cols[2].y m.cols[3].y | * {v.y}
// | m.cols[0].z m.cols[1].y m.cols[2].y m.cols[3].y | {v.z}
// [ m.cols[0].w m.cols[1].w m.cols[2].w m.cols[3].w ] {v.1}
OZZ is a skeletal animation library that provides support for blending between multiple animations.
http://guillaumeblanc.github.io/ozz-animation/

So I'm trying to take the output from that library and apply it to the bones rendered through Irrlicht.
This might be a task better suited for a hardware skinning shader?
I've never worked with shaders before either so I'd like to do it this way then look into using a shader in the future.
Last edited by kklouzal on Mon Jan 29, 2018 10:40 am, edited 1 time in total.
Dream Big Or Go Home.
Help Me Help You.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Setting Bones From Matrix

Post by Mel »

Skinning is tricky, as the bones use to work on a local frame of reference, but, to perform skinning, you have to do it in the model frame of reference, thus, you needed somewhere the inverse of the local transform in order to provide a proper skining matrix to the vertices, and you never go back to the position/rotation/scale, after you've obtained your matrix.

Take a look to this series of videos, they explain on most detail how skinning works, and they provide code to perform skinning in software, passing it to hardware is as easy, or as hard, as setting the matrices and the vertex data in a shader.
https://www.youtube.com/watch?v=f3Cr8Yx3GGA
https://www.youtube.com/watch?v=F-kcaonjHf8
https://www.youtube.com/watch?v=cieheqt7eqc
https://www.youtube.com/watch?v=z0jb1OBw45I
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

That tutorial series is very helpful.

Thankfully OZZ provides both the local space transforms and the model space transforms. I just need to know what I have to do to set the transform I get for a specific bone in OZZ onto the specific bone in irrlicht.

There doesn't seem to be a function to directly set the transformation matrix for a bone (only retrieve it).
Dream Big Or Go Home.
Help Me Help You.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Setting Bones From Matrix

Post by Mel »

It is complex to get back the position/rotation/scale from a matrix, In shader skinning you can just pass the matrices, and forget about PRS, otherwise, you would have to hack through the engine to be able to set the matrices directly... (and somehow, break the whole animation system in the process... ( ͡° ͜ʖ ͡°) )
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

I figured it would be a bit difficult to pull PRS out of the matrix. I started looking into how shaders work.
I found this vertex shader that will hopefully get me started.

Code: Select all

uniform mat4 bones[100];
uniform int use_skinning;
 
void main(void)
{
    mat4 mfinal = gl_ModelViewMatrix;
 
    // skinning
    if(use_skinning==1)
    {
        vec3 weights= gl_Color.xyz;
        vec3 boneid = gl_Vertex.w * vec3( 1.0/128.0 , 1.0 , 128.0 );
        boneid = (boneid - floor(boneid))*128.0;
 
        mat4 mskin  =   bones[int(boneid.x)]*weights.x+
                bones[int(boneid.y)]*weights.y+
            bones[int(boneid.z)]*weights.z;
        mfinal = mfinal * mskin;
    }
 
    gl_Position = gl_ProjectionMatrix * mfinal * vec4(gl_Vertex.xyz,1.0);
}
Simple pass-through fragment shader (just displays an unlit, presumably untextured, model)

Code: Select all

#version 330
 
uniform sampler2D tex;
uniform vec2 tex_size;
 
layout(location = 0) out vec4 out_color;
 
void main()
{
    vec4 in_color = texture(tex, gl_FragCoord.xy / tex_size);
    out_color = in_color;
}
Now we convert from OZZ Matrix4x4 to IRR matrix4

Code: Select all

        //  Convert from OZZ Float4x4 into IRR matrix4
        size_t bone = 0;
        while (bone < models_.Count())
        {
            ozz::math::Float4x4 mat = models_[bone];
            Bones_[bone].setM((mat.cols)->m128_f32);
            bone++;
        }

Code: Select all

 class SkinningCallback : public irr::video::IShaderConstantSetCallBack
{
public:
 
    //  Matrix array of bone translations in model-space
    irr::core::matrix4 *Bones_;
    irr::f32 useSkinning;
 
    void OnSetConstants(irr::video::IMaterialRendererServices* services, irr::s32 userData)
    {
        services->setPixelShaderConstant("bones", Bones_->pointer(), 100);
        services->setPixelShaderConstant("useSkinning", &useSkinning, 1);
    }
};

Code: Select all

    void CreateSkinningShader(irr::IrrlichtDevice* device, irr::video::IVideoDriver* driver, irr::core::matrix4* TranslationMatrix, irr::scene::IAnimatedMeshSceneNode* SceneNode)
    {
        irr::video::IGPUProgrammingServices* gpu = driver->getGPUProgrammingServices();
        irr::s32 mtlSkinningShader = irr::video::EMT_SOLID; //  Fallback Material
 
        SkinningCallback* callback = new SkinningCallback;
        callback->Bones_ = TranslationMatrix;
        callback->useSkinning = 1;
 
        irr::io::path VertPath = "skinning.vert";
        irr::io::path FragPath = "skinning.frag";
        mtlSkinningShader = gpu->addHighLevelShaderMaterialFromFiles(
            VertPath, "main", irr::video::EVST_VS_1_1,
            FragPath, "main", irr::video::EPST_PS_1_1,
            callback, irr::video::EMT_SOLID, 0, irr::video::EGSL_DEFAULT);
 
        SceneNode->setMaterialType((irr::video::E_MATERIAL_TYPE)mtlSkinningShader);
    }
From what I understand, the IShaderConstantSetCallBack allows me to initially set parameters to be passed into the shader via setPixelShaderConstant.

Am I able to directly pass ozz::Range<ozz::math::Float4x4>* models_ into the shader with setPixelShaderConstant? (I think I must first convert from OZZ Float4x4 into IRR matrix4).
How do I update the value of this variable later? Or does simply changing the value (since it's a pointer/reference) allow the shader to receive the updated value?
In the shader code, where does it get the variable 'boneid' from?
Finally, what about the fragment shader? I want to continue to let Irrlicht handle the lighting/texturing or do I now have to do this myself since we've overridden the material type with this shader?

The application compiles and displays an unlit and presumably untextured model in it's default T-Pose. I made OZZ step through the animation but the model does not change.
Some quick debugging shows that each call to setPixelShaderConstant return false.
Dream Big Or Go Home.
Help Me Help You.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Setting Bones From Matrix

Post by Mel »

How you update is mostly up to you and how you get the data, just keep in mind the size of a matrix would be 16 floats, not just 1, if the count of elements you pass round up to 4bytes, then, the size of 100 matrices is 400, not 100. Also, is better to set the constants via identifier index, more than the identifiers themselves (that's slow) And the pixel shader needs a texture, it is reading one... only to get ignored... To perform lighting, you must also rotate the normal (and maybe bitangent and tangent) vector/s, which implies multiplying the normal by the 3x3 version of the skinning matrix, although if i remember properly, you can achieve such effect just multiplying the skinning matrix by the normal as a vec4, just setting the last component to 0.

Also check that the row/column order and the handedness is what Irrlicht is expecting, or else, results may be unexpected (Irrlicht expects left handed, row major matrices, i think)
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

Okay thank you very much, this is extremely helpful information.

I see the callback is called each frame so the constants are once again set each frame.

Since setting the constants by index is not only better on performance but also failing when trying to do it by name, how do I know which index maps to which variable inside the shader? Is it just the order in which I'm declaring the variables?

Code: Select all

uniform mat4 bones[100];     // index 0
uniform int use_skinning;     // index 1

Code: Select all

uniform int use_skinning;     // index 0
uniform mat4 bones[100];     // index 1
I'm also assuming the index starts at 0 here as most things do.

EDIT
Trying to use the index method to set a constant outputs
Cannot set constant, use high level shader call
It seems like I must use the other method to set a constant.

EDIT_EDIT
I was able to successfully set the use_skinning constant however the other will not set. I've verified the matrix4 is being populated but I for some reason cannot get (success = services->setPixelShaderConstant("bones", Bones_->pointer(), 400);) to return true..

EDIT_EDIT_EDIT
I tried to simplify the problem by attempting to set a single mat4 instead of an array but it still returns false..

Code: Select all

        irr::core::matrix4 TestMat;
        success = services->setPixelShaderConstant("boneee", TestMat.pointer(), 16);
I'm obviously doing something wrong here I just don't know what it could be.
Dream Big Or Go Home.
Help Me Help You.
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

I found two posts that refer to a bug regarding ATI/AMD cards and setting arrays. It states that you have to manually set each position in the array? Is this still the case?
I'm able to set the first position in the array like this:

Code: Select all

services->setPixelShaderConstant("bones[0]", Bones_->pointer(), 16);
But trying to go further, bones[1], bones[2], etc.. still fails.

EDIT
This is probably my last update for today but it seems like specifying the "[0]" is a special/hidden way to access the constant and it does fill the entire array.
Now I've got to look at my actual shader code cause the entire model moves instead of the individual joints.

I could be doing this all wrong :P kinda flying by the seat of my pants here :D
Dream Big Or Go Home.
Help Me Help You.
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

Look at this abomination I have created! :D
https://youtu.be/y8faIL9cNTI

I think the vertices are all crazy because of the left-hand/right-hand coordinate system mismatch.
Either that or the way I pull each float out of the 4x4 matrix is wrong.

Code: Select all

    void OnSetConstants(irr::video::IMaterialRendererServices* services, irr::s32 userData)
    {
        //  Construct our array of mat4
        irr::f32* Uniforms = new irr::f32[MaxBones * 16];
        for (size_t bone = 0; bone < models_->Count(); bone++)
        {
            ozz::math::Float4x4* mat = &models_->operator[](bone);
            const size_t pos_base = bone * 16;
            size_t pos_elem = 0;
            for (irr::u32 row = 0; row < 4; row++)
            {
                ozz::math::SimdFloat4* mat_row = &mat->cols[row];
                for (irr::u32 col = 0; col < 4; col++)
                {
                    Uniforms[pos_base + pos_elem] = (irr::f32&)mat_row->m128_f32[col];
                    pos_elem++;
                }
            }
        }
 
        const bool success = services->setVertexShaderConstant("bones[0]", Uniforms, MaxBones*16);
        //printf("Set Constant '%s' = %d\n", "bones", success);
 
        delete[] Uniforms;
    }
On a positive note, it is walking!
Dream Big Or Go Home.
Help Me Help You.
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: Setting Bones From Matrix

Post by devsh »

You can have a look at my base scengraph class from IrrBAW, it does a pretty good job of going between PosRotScale to Matrix and back.
https://github.com/buildaworldnet/Irrli ... ceneNode.h
I do however use a 4x3 matrix instead of a 4x4 since for anything but projection your matrix operation is a 3x3 mult + add

Looking at your results on youtube, you may have to transpose your matrix (swap columns with rows)
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

I kinda like the monster I created. ;)
I'll try to swap the columns and rows to see if that helps.

As a side note, I created 25 of these objects to see what performance would be like and my fps hovered between 59-60.
5 more objects brought me down to 54fps, that's only 30 animated objects which is quite low..

I might have to ditch this all together and just deal with not being able to blend animations.

EDIT
Assuming I did it properly the vertices got even worse after flipping columns and rows

Code: Select all

        irr::f32* Uniforms = new irr::f32[MaxBones * 16];
        for (size_t bone = 0; bone < models_->Count(); bone++)
        {
            ozz::math::Float4x4* mat = &models_->operator[](bone);
            const size_t pos_base = bone * 16;
            size_t pos_elem = 0;
            /*for (irr::u32 row = 0; row < 4; row++)
            {
                ozz::math::SimdFloat4* mat_row = &mat->cols[row];
                for (irr::u32 col = 0; col < 4; col++)
                {
                    Uniforms[pos_base + pos_elem] = (irr::f32&)mat_row->m128_f32[col];
                    pos_elem++;
                }
            }*/
            for (irr::u32 col = 0; col < 4; col++)
            {
                for (irr::u32 row = 0; row < 4; row++)
                {
                    ozz::math::SimdFloat4* mat_row = &mat->cols[row];
                    Uniforms[pos_base + pos_elem] = (irr::f32&)mat_row->m128_f32[col];
                    pos_elem++;
                }
            }
        }
Dream Big Or Go Home.
Help Me Help You.
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: Setting Bones From Matrix

Post by devsh »

Fine then don't invert rows and columns XD

The bone matrix needs to have a Pose Matrix premultiplied in.

I.e. the skinning is done relative to the origin of the bone, rotations only happen around 0,0,0

So:
vertices need to be moved to a space where (0,0,0) corresponds to bone origin (end or start) [the step you're missing]
Then they need to be rotated with the bone's transformation matrix and moved with the bone to a global position

From IrrBAW

Code: Select all

 
                                        boneDataForInstance[j].SkinningTransform = concatenateBFollowedByA(attachedNodeInverse,concatenateBFollowedByA(bone->getAbsoluteTransformation(),referenceHierarchy->getBoneData()[j].PoseBindMatrix)); //!may not be FP precise enough :(
 

You're getting 50FPS with 30 objects because Irrlicht reskins every frame all the time because it shares the same mesh with all nodes, I wrote about this in my Hardware Skinning post.
Mel
Competition winner
Posts: 2292
Joined: Wed May 07, 2008 11:40 am
Location: Granada, Spain

Re: Setting Bones From Matrix

Post by Mel »

As a side note, I created 25 of these objects to see what performance would be like and my fps hovered between 59-60.
5 more objects brought me down to 54fps, that's only 30 animated objects which is quite low..
Perhaps you're not setting the meshes static on the video card?
I kinda like the monster I created. ;)
Ain't them just cute? when they drag and leave vertices all over the place in impossible forms, corrupting even the purest of the scenes? :lol:
"There is nothing truly useless, it always serves as a bad example". Arthur A. Schmitt
kklouzal
Posts: 343
Joined: Sun Mar 28, 2010 8:14 pm
Location: USA - Arizona

Re: Setting Bones From Matrix

Post by kklouzal »

Code: Select all

Mesh->setHardwareMappingHint(irr::scene::EHM_STATIC, irr::scene::EBT_VERTEX_AND_INDEX);
I was experimenting earlier with the different hint types and had it commented out >.<
Now with 50 objects I'm at 274fps which is much better!

Let me see if I can decipher devsh's last post and get this explosion of vertices fixed.

EDIT:
My transformations are already in model-space I think I'm already doing it as you said?

Code: Select all

uniform mat4 bones[100];
 
void main(void)
{
    mat4 ModelTransform = gl_ModelViewMatrix;
     
    int BoneID = int(gl_Color.r * 255);
    mat4 vertTran = bones[BoneID - 1];
    
    BoneID = int(gl_Color.g * 255);
    if(BoneID > 0)
        vertTran += bones[BoneID - 1];
 
    BoneID = int(gl_Color.b * 255);
    if(BoneID > 0)
        vertTran += bones[BoneID - 1];
        
    BoneID = int(gl_Color.a * 255);
    if(BoneID > 0)
        vertTran += bones[BoneID - 1];
        
    ModelTransform *= vertTran;
    
    gl_Position = gl_ProjectionMatrix * ModelTransform * gl_Vertex;
}
It is in the shader code we are working relative to (0,0,0) then finally move into world space by multiplying gl_ProjectionMatrix no?

I feel like the BoneID is not being calculated properly since the model is orientated properly and seems to be generally moving correctly according to the animation (from what I can decipher from the explosion of verts)
Last edited by kklouzal on Wed Jan 31, 2018 8:55 pm, edited 1 time in total.
Dream Big Or Go Home.
Help Me Help You.
devsh
Competition winner
Posts: 2057
Joined: Tue Dec 09, 2008 6:00 pm
Location: UK
Contact:

Re: Setting Bones From Matrix

Post by devsh »

What this C++:

Code: Select all

concatenateBFollowedByA(attachedNodeInverse,concatenateBFollowedByA(bone->getAbsoluteTransformation(),referenceHierarchy->getBoneData()[j].PoseBindMatrix));
does, is apply 3 matrices:
1) move vertices and bone to (0,0,0) and align bone to some axis (this matrix is the pose bind one and is static for the mesh since load)
2) move vertices and bone to global position (your dynamic matrix)
3) but the global bone matrix implicitly contains the animated node global transformation, so we need to take it away by multiplying in the inverse of the animated node global transform

(step 3 is necessary because of having a scenegraph)

You can take step 3 away if you only multiply by the projection-view matrix in the vertex shader.
But openGL only offers gl_ProjectionMatrix and gl_ModelViewProjectionMatrix.

Essentially you need to make sure #1 gets applied, and that #2 is absolutely correct and global meaning it has the bone world space position or at least model space position.

You can check this by drawing lines or boxes with the world transformation matrix set to matrix #2 .


EDIT: Your shader is wrong, it doesnt apply the weights to the matrices you are summing.

EDIT2: Ideal equation is Proj*(View*(WeightedGlobalBoneMatrix*vertex))
Post Reply