D3D9 Driver Reset failure and memory corruption when ALT+TAB

You discovered a bug in the engine, and you are sure that it is not a problem of your code? Just post it in here. Please read the bug posting guidelines first.
Post Reply
HellFlip
Posts: 4
Joined: Mon Aug 20, 2012 9:57 pm

D3D9 Driver Reset failure and memory corruption when ALT+TAB

Post by HellFlip »

Hello,

Sorry to open once again this recurring discussed issue on D3D9 Reset() failure. But here I have a working solution, this involved some little changes in D3D9 driver.

The CD3D9Driver::reset() method has strange logic of recreating textures and depth buffer even when D3D9 Reset() is failing (typically with D3DERR_DEVICELOST Resetting failed due to device lost). So under this condition of Reset failure, those next creations of textures and depth buffers will necessarly fail. Still, this is not the problem by itself, but the structure that then lead to the instability.

One first problem is that the calls for recreation of DepthBuffers being not tested, and because they are failing (since the reset is not working), this result in illegal write memory access and memory corruption.

Second problem is : while before releasing the D3D9 Surface texture and DepthBuffers->Surface, they are tested if null, and only after being released, those pointers are not set to null after being released. The problem is that it happens that the reset() function is is being called multiple time and then would retry to release. Therefore leading to instability. (this occurs after D3DERR_DEVICELOST occurs).

The result is memory corruption occurring, and leading to executable crash.

Note: those mentioned illegal memory access where tracked and found by running "Application Verifier" + Visual Studio debugger

My solution to this problem was :
1 - reorder the code:
original: a. Release All b. Reset c. Recreate all d. test if Reset failed(and exit)
changed: a. Release All b. Reset c. Test if Reset failed(and exit) d. Recreate all
2 - Once Texture and DepthBuffers are released, they are set to null

With this in place, even when D3DERR_DEVICELOST happen, there isn't anymore crash or memory corruption reported by "Application Verifier".

With this solution in place, while no more crash, another unrecoverable Reset failure can happen:
after a first Reset() failure returning D3DERR_DEVICELOST, all subsequents are returning D3DERR_INVALIDCALL, indefinitly. This problem was workarounded in the application by disabling rendering when device->isWindowMinimized() returns true, this prevented Reset() failure with D3DERR_INVALIDCALL.

Then after, the ALT+TAB was heavily stressed in my app and no more crash was observed.

Remark for reproducing instability:
- use a fullscreen resolution not same than the laptop desktop resolution, with the same this is pretty stable.
- use render targets in the code (and possibly quite much shaders also can help)

Reproduced using Irrlicht 1.7.1 (but from 1.8 and SVN code, this code has not changed)

The modified code:

Code: Select all

CD3D9Driver.cpp
 
bool CD3D9Driver::reset()
{
    u32 i;
    os::Printer::log("Resetting D3D9 device.", ELL_INFORMATION);
 
    for (i=0; i<Textures.size(); ++i)
    {
        if (Textures[i].Surface->isRenderTarget())
        {
            ((CD3D9Texture*)Textures[i].Surface)->releaseDX9Texture();
        }
    }
    for (i=0; i<DepthBuffers.size(); ++i)
    {
        if(DepthBuffers[i]->Surface) {
            DepthBuffers[i]->Surface->Release();
            DepthBuffers[i]->Surface = 0;
        }
    }
    // this does not require a restore in the reset method, it's updated
    // automatically in the next render cycle.
    removeAllHardwareBuffers();
 
    DriverWasReset=true;
 
    HRESULT hr = pID3DDevice->Reset(&present);
 
    if (FAILED(hr))
    {
        if (hr == D3DERR_DEVICELOST)
        {
            DeviceLost = true;
            os::Printer::log("Resetting failed due to device lost.", ELL_WARNING);
        }
#ifdef D3DERR_DEVICEREMOVED
        else if (hr == D3DERR_DEVICEREMOVED)
        {
            os::Printer::log("Resetting failed due to device removed.", ELL_WARNING);
        }
#endif
        else if (hr == D3DERR_DRIVERINTERNALERROR)
        {
            os::Printer::log("Resetting failed due to internal error.", ELL_WARNING);
        }
        else if (hr == D3DERR_OUTOFVIDEOMEMORY)
        {
            os::Printer::log("Resetting failed due to out of memory.", ELL_WARNING);
        }
        else if (hr == D3DERR_DEVICENOTRESET)
        {
            os::Printer::log("Resetting failed due to not reset.", ELL_WARNING);
        }
        else if (hr == D3DERR_INVALIDCALL)
        {
            os::Printer::log("Resetting failed due to invalid call", "You need to release some more surfaces.", ELL_WARNING);
        }
        else
        {
            os::Printer::log("Resetting failed due to unknown reason.", core::stringc((int)hr).c_str(), ELL_WARNING);
        }
        return false;
    }
 
    // restore RTTs
    for (i=0; i<Textures.size(); ++i)
    {
        if (Textures[i].Surface->isRenderTarget())
            ((CD3D9Texture*)(Textures[i].Surface))->createRenderTarget();
    }
 
    // restore screen depthbuffer
    pID3DDevice->GetDepthStencilSurface(&(DepthBuffers[0]->Surface));
    D3DSURFACE_DESC desc;
    // restore other depth buffers
    // dpeth format is taken from main depth buffer
    DepthBuffers[0]->Surface->GetDesc(&desc);
    // multisampling is taken from rendertarget
    D3DSURFACE_DESC desc2;
    for (i=1; i<DepthBuffers.size(); ++i)
    {
        for (u32 j=0; j<Textures.size(); ++j)
        {
            // all textures sharing this depth buffer must have the same setting
            // so take first one
            if (((CD3D9Texture*)(Textures[j].Surface))->DepthSurface==DepthBuffers[i])
            {
                ((CD3D9Texture*)(Textures[j].Surface))->Texture->GetLevelDesc(0,&desc2);
                break;
            }
        }
 
        pID3DDevice->CreateDepthStencilSurface(DepthBuffers[i]->Size.Width,
                DepthBuffers[i]->Size.Height,
                desc.Format,
                desc2.MultiSampleType,
                desc2.MultiSampleQuality,
                TRUE,
                &(DepthBuffers[i]->Surface),
                NULL);
    }
 
    DeviceLost = false;
    ResetRenderStates = true;
    LastVertexType = (E_VERTEX_TYPE)-1;
 
    for (u32 i=0; i<MATERIAL_MAX_TEXTURES; ++i)
        CurrentTexture[i] = 0;
 
    setVertexShader(EVT_STANDARD);
    setRenderStates3DMode();
    setFog(FogColor, FogType, FogStart, FogEnd, FogDensity, PixelFog, RangeFog);
    setAmbientLight(AmbientLight);
 
    return true;
}
 
CD3D9Texture.cpp
 
void CD3D9Texture::releaseDX9Texture()
{
    if (Texture) {
        Texture->Release();
        Texture=0;
    }
}
 
CD3D9Texture.h
    //! release the DIRECT3D9 Texture
    void releaseDX9Texture();
 
robmar
Posts: 1125
Joined: Sun Aug 14, 2011 11:30 pm

Re: D3D9 Driver Reset failure and memory corruption when ALT

Post by robmar »

Looks interesting, I´ll give this a try and report any findings.
Post Reply