You discovered a bug in the engine, and you are sure that it is not a problem of your code? Just post it in here. Please read the bug posting guidelines first.
Rows and columns in computer matrices tend to be different than in math (edit: or maybe depends on the library, I think Irrlicht was heavily influenced by Direct3d back then). As computer matrices tend to use row major instead of column major like the math guys. It's kinda meant to transform a 4 element vector by the matrix I guess. Which it does. If you use transposed matrix instead you get your result.
But maybe stupid name - I just realize someone managed to persuade me to add transformVec4 a few years ago because neither of us realized at that point that this function exists already and does the same.