
Yes sorry, I´ve been pulling my hair out over this one, but now have it fixed. It was soooo painful!
There were multiple problems which made the whole thing appear very confusing.
I had to simplify everything in my code, and work up from the ground up very slowly.
Using ARGB8 depth buffer, 8-bits of depth caused quite large changes in depth across objects´ surfaces - I had to check each pixel value to find that.
So I tried to the f16r texture, and that solved it. I was pleased to see that the float texture, just 16-bits on R, worked well.
Is there any performance hit on using f16r over argb8? I guess that the stream processors or whatever they are called work first in floating point, and in some case emulate integer math, or so I read, and that only PS-4 has true integer processor instructions.
The next problem was that the HLSL PS-3 compiler refused to recognise "break". Grantye kindly told me that the compile flag for HLSL in Irrlicht for branching was disabled, though yet to find that.
For now I worked around setting the for loop counter high to break.
Then, the most frustrating, was that the "%" modulos operator in HLSL does not, unless I got that wrong too, work the same as it does in VS C++!
That really caused me a headache... I have no debugger for HLSL DX9, I read that the PIX debugger with the Windows 2007 SDK works on Windows 7 for HLSL debugging, will give that a go soon.
So I changed to use the fmod function with round, and the math in my shader started working!
Halleluiah! What a set of headaches but now the depth display of the fish in my virtual aquarium works! Makes me realize that my brain is at times full of mud, and that the five Ps are very important: Planning Prevents Pathetically Poor Performance!
Any news on the DX10/11 driver, then debugging would be a little easier!
I would go for Mantle really, but with only a 23% speed improvement so said, is it worth it?