Animation system with bones and Kinect

Tekkai · Post by **Tekkai** » Thu Apr 21, 2011 5:59 pm

Hi,

I'm working on a project to have kinect and irrlicht integrated. Nowadays I can receive all the joints positions and rotations, and now I will have to fight to know how to integrate it correctly with a model made by myself.

I have started with something "really simple". My model have a bone called "torsoBone" which will be the root bone for the model. That means that if I move this bone, all the model have to be moved.

The problem I'm having now is that by some reason I cannot understand, if I apply the data received from Kinect to the model (model->setPosition()) it works correctly, but if I apply the same values to the bone position, I have the axis changed. Z turns X and so on...

Anyone can help me with that? Why is this happening? I thought about Left and Right Handed Axis, but the exporter I'm using, when I turn on the right handed axis, it doens't export anything. I tried also to change between directx9 and OpenGL but with the same result...

I don't want to fix it manually, because I think that if I do that, I will have to fight a lot when I have to work with the rotations given by Kinect.

Thx!!!

mongoose7 · Post by **mongoose7** » Fri Apr 22, 2011 4:02 am

I found when loading a cal3d model that I had to transpose the quaternions.

Tekkai · Post by **Tekkai** » Tue Apr 26, 2011 9:52 pm

Hey,

Nowadays I'm not using the quaternions, just using the core position. Any thoughts?

Pau

mongoose7 · Post by **mongoose7** » Wed Apr 27, 2011 2:42 am

So, without animation, the skeleton does not sit inside the mesh? How do you set Joint.LocalMatrix?

Or is it with animation that the skeleton is not in the mesh? Or is the skeleton in the mesh but the mesh is wrongly aligned? So you set a position key but not a rotation key? This is the paradox of skinned meshes - only the root node can have a position. Every other node must have the position original set in the skeleton, I think. That is, it is the rotation that gives the correct position of the skeleton, though the position has to be set correctly anyway.

Tekkai · Post by **Tekkai** » Wed Apr 27, 2011 2:30 pm

Hey!

Thx for the answer. The mesh has an skeleton set by 3DStudio Max. I have exported the mesh to .X, and the skeleton is exported in the same file. If I access to the "position" of the mesh and move it, it works propertly, but if I access to the mesh through the root bone, and set the position to the root bone, the axis are changed.

Pau

mongoose7 · Post by **mongoose7** » Thu Apr 28, 2011 2:10 am

I think you misunderstand me. The model is loaded into Irrlicht with the skeleton. Irrlicht now has the mesh and the skeleton. Using the debug settings, you should see if Irrlicht shows the skeleton as being inside the mesh.

Anyway, as I said, you cannot morph the mesh by positioning the bones/joints. You have to rotate them.

But this is only the first problem. I guess you want to animate a mannequin in real time. Well, Irrlicht reads animations and prebuilds the matrices. So you are going to have to do all this work yourself. It may be easier to build an SDL application and just copy/paste some animation code from Irrlicht. Hmmm, you might look at bioviewer.

hybrid · Post by **hybrid** » Thu Apr 28, 2011 8:28 am

In which way would that be easier? Even if you couldn't use anything from Irrlicht, so it would be for SDL. So nothing lost. But Irrlicht supports ragdoll animation and also full manual positioning of skeletons. Which means that it's much easier than without any support to do such things.

mongoose7 · Post by **mongoose7** » Thu Apr 28, 2011 9:07 am

There was a doubt in my mind whether manually positioning the skeleton would cause the mesh to follow. I guess you are saying that it will. Therefore, the OP should be able to take the locations from Kinect and put them straight into the Joint.LocalMatrix. So the only question remaining is why it is not working for him.

But I can't see how it can work like this. I believe that you must set rotations in order that the mesh should follow the bones. Can you tell me why this would not be the case?

Why use SDL. Well, I'd use glut myself, but the nice man who wrote bioviewer has basically done everything that is required except calculate the rotations, as they are already in the BVH file. In fact, a BVH file contains 1. a skeleton, 2. the translation of the root node, 3. the rotations of all the bones.

Is there example code for animating rag dolls?

hybrid · Post by **hybrid** » Thu Apr 28, 2011 11:18 am

See http://www.irrlicht3d.org/wiki/index.ph ... tionSystem

mongoose7 · Post by **mongoose7** » Thu Apr 28, 2011 12:13 pm

Thanks. But I never believed what this page said.

For example 'Joint Control', If you don't set JointMode to EJUOR_CONTROL, your mesh will play the inbuilt animation. If you do set it, it won't, but then, I don't think a call to animateJoints() will play the animation, because JointMode is set to EJUOR_CONTROL.

I also don't believe "Then move the bones after optionally calling animateJoints() (so that's it's not overwritten by the built-in animation)." I cannot see how it could be optional. I understand that once you have called it for a frame (well for a time value), further calls will be ignored, so it may have the side-effect you want, but I should think you *do* have to call it. Unless the intention is that you should inspect the internal state to see if a call is necessary? But the fact that further calls will be ignored looks like an internal optimisation and not an element of the interface. This may be just me, though.

"You move them like any other scene node" shows "Head->setRotation(core::vector3df(3,2,1));", so I don't think you can just set the position of the joints. It looks as if you have to set the rotations as well. I just didn't believe that Irrlicht would work out the rotation based on the position. You may be able to do it for rag dolls but not for a real animation.

So, when I got to "Ragdolls", I had completely lost faith in the description. It also doesn't sit very well with what had gone before. But, can we take it step-by-step?

We set the JointMode to EJUOR_CONTROL. OK, this will disable internal animation as far as I can see.
Set RenderFromIdentity = OK, no transformation will be applied. But haven't we disbaled animation?
Set SkinningSpace()? What does this do?
We now set the position and rotation.

There just seems too many things to do. Since we are going to set the transformations ourself, it would seem that we just need to disable the internal animation. Setting the JointControl should do this, so setting the skinning space seems to achieve nothing. Or does it cause the generation of the global matrices.

Well, I could look at the code, but that is where I came in. Without a bit of clarity in the description, one is rather left looking at the code. Which then makes all the documentation somewhat irrelevant, as it is just generated from the headers. Oh, well.

The last line is "(TODO: Please add a sample here)." It would have been nice.

Anyway, I'd appreciate any light you can shed on the above.

Thanks.

shadowslair · Post by **shadowslair** » Thu Apr 28, 2011 8:25 pm

Too much talking pal. Reading all this one can say the animation system is absolutely unusable, which is not true. Who told you that EJUOR_CONTROL disables the animation? There are already some samples around, too tired to search them right now for you, but (again) I will post a link for I assume "right" way to do original-animation-playing-and-then-manually-locally-rotated-joints as I tend to do each week: http://irrlicht.sourceforge.net/phpBB2/ ... ht=#247382 . It should™ work. As for your Kinect and ragdoll experiences the only thing I can say is that it may be a bit difficult to make them working, without having some idea of the way Irrlicht handles skinned meshes` joint transformations and the way Luke made all this work. You won`t get it easily working without getting your hands dirty.

PS: You can show us some code what you`ve tried and failed.

mongoose7 · Post by **mongoose7** » Fri Apr 29, 2011 1:46 am

OK. thanks, I'll give it a try over the weekend. It's actually Tekkai's problem. I guess he can talk for himself, but I think he gets joint locations from Kinect and he wants to apply them in Irrlicht. My point was that he would have to change locations to rotations to correctly animate a mesh.

Tekkai · Post by **Tekkai** » Sat Apr 30, 2011 11:26 am

Thx everyone, I've been working on it and I just realized what was my error. I was working with a bone that wasn't the root. I applied the kinect root position to the root bone and seems to work pretty well (Have to change Z by -Z but the rest of axis are ok).

Now I'm working with the other bones. Actually I'm calculating the rotation between two bones. I attach an image of what I want to do, see if that seems correct to all of you. I promisse that if I get the code working I'll put the entire class and a example of use so everyone can use it.

And that's the code I'm using to calculate the rotation between the lupperarm bone and the lforearm bone, but by some reason, I'm not getting good results, I'm still working on it.

Code: Select all

    
    core::vector3df lLeftForeArmVector( mJointLeftElbowPos.position.X - mJointLeftHandPos.position.X ,
                                        mJointLeftElbowPos.position.Y - mJointLeftHandPos.position.Y ,
                                        -(mJointLeftElbowPos.position.Z - mJointLeftHandPos.position.Z) );
    
    core::vector3df lLeftUpperArmVector( mJointLeftShoulderPos.position.X - mJointLeftElbowPos.position.X ,
                                         mJointLeftShoulderPos.position.Y - mJointLeftElbowPos.position.Y ,
                                         -(mJointLeftShoulderPos.position.Z - mJointLeftElbowPos.position.Z) );

    core::quaternion lLeftForeArmQuaternion(lLeftForeArmVector.X, lLeftForeArmVector.Y , lLeftForeArmVector.Z , 0 );
    core::quaternion lLeftUpperArmQuaternion(lLeftUpperArmVector.X, lLeftUpperArmVector.Y , lLeftUpperArmVector.Z , 0 );

    core::quaternion lElbowRotationQ = lLeftForeArmQuaternion * lLeftUpperArmQuaternion;
    core::vector3df lElbowRotationV;
    lElbowRotationQ.toEuler(lElbowRotationV);

And thats the output

Code: Select all

    Hand Pos:   X: -1048.66		Y: 934.008	Z: -2635.97
    Elbo Pos:   X: -1109.25		Y: 649.297	Z: -2677.76
    Shou Pos:   X: -823.775		Y: 690.626	Z: -2639.17

    Fore Vec:   X: -60.5909		Y: -284.711	Z: -41.7886
    Uppe Vec:   X: 285.477		Y: 41.3284	Z: 38.5891

    Fore Qua:   X: -60.5909		Y: -284.711	Z: -41.7886	W: 0
    Uppe Qua:   X: 285.477		Y: 41.3284	Z: 38.5891	W: 0

    In Degrees ==> Elbo Rot:   X: -7.70657		Y: 90	Z: -138.546

I think that the data I'm getting are OK, but I cannot get the correct rotation. If anyone can have a look to that will be really helpful

Tekkai · Post by **Tekkai** » Sat Apr 30, 2011 12:25 pm

Some good news

I just realized that there is on rotationFromTo() function so I'm doing:

Code: Select all

    
    lElbowRotationQ.rotationFromTo(lLeftForeArmVector,lLeftUpperArmVector);
    lElbowRotationQ.toEuler(lElbowRotationV);

And it seems to work pretty well, that's one of the outputs:

Code: Select all


// T Position

Hand Pos:   X: -1336.74		Y: 591.158	Z: -2887.27
Elbo Pos:   X: -1031.08		Y: 607.529	Z: -2864.32
Shou Pos:   X: -714.31		Y: 681.802	Z: -2850.8

Fore Vec:   X: 305.655		Y: 16.3704	Z: 22.9487
Uppe Vec:   X: 316.774		Y: 74.2733	Z: 13.5234

Fore Qua:   X: 305.655		Y: 16.3704	Z: 22.9487	W: 0
Uppe Qua:   X: 316.774		Y: 74.2733	Z: 13.5234	W: 0
Elbo Rot:   X: -0.691717		Y: 1.87301	Z: 10.0842

Code: Select all


//Fork Position

Hand Pos:   X: -954.03		Y: 926.357	Z: -2734.62
Elbo Pos:   X: -1019.53		Y: 632.731	Z: -2746.91
Shou Pos:   X: -720.749		Y: 659.161	Z: -2753.47

Fore Vec:   X: -65.502		Y: -293.626	Z: -12.2842
Uppe Vec:   X: 298.783		Y: 26.4297	Z: -6.55957

Fore Qua:   X: -65.502		Y: -293.626	Z: -12.2842	W: 0
Uppe Qua:   X: 298.783		Y: 26.4297	Z: -6.55957	W: 0
Elbo Rot:   X: -2.1247		Y: -4.55134	Z: 107.561

But when I draw in irrlicht, my forearm is moving in "depth". Does make sense for any of you¿?

mongoose7 · Post by **mongoose7** » Sat Apr 30, 2011 1:22 pm

If you replace Z by -Z you may have to transpose the rotation matrices, as per my first post.

Try something like

lElbowRotation.quaternion.W = -lElbowRotation.quaternion.W.

WAK! Your quaternions have W = 0. This is a rotation through 180 degrees!