Updated 7/3/11 -Edge detection+Antialiasing on RenderTarget!

stefbuet · Post by **stefbuet** » Tue Jul 27, 2010 7:06 pm

[Updated 7/03/2011]
-new edge detection filter also based on normal & depth but now constant edge size even if you zoom out, more exact detection thanks to a 9x9 kernel instead of old 2*2 (cross) and noise removal with sin function.
-----

Here is a shader I've done which is using depth and normals to find edges.
Depth can't be used alone because edges which are inside of an object have a continuous depth, that's why I'm also using normals.

Please, if you find a faster/better may to do it, post it

Note : on the screenshot, the floor have strange white stuff, it's because it contain grass

Result :

VS:

Code: Select all

// simple vertex shader
// Author:Stefbuet
// Date:20/04/2010

//This shader can be used by a wide range of pixel shader.
//It is just giving to the PS the texture coords, vertex pos & color.

varying vec4 textureCoord;

void main() {
    
    //texture coordinate'll be interpolated to the PS
    textureCoord=gl_MultiTexCoord0;

    //don't change any vertex position & color
    gl_FrontColor=gl_Color;
    gl_Position=ftransform();
    

}

PS:

Code: Select all


// edge detection pixel shader
// Author:Stefbuet
// Date:14/06/2010
// Last update:07/03/2011

//this shader is doing an edge detection with depth and normal map


uniform sampler2D depthMap;
uniform float SCREENX, SCREENY;

varying vec4 textureCoord;

vec2 offsetArray[8];

void main() {

   vec2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=texture2D(depthMap, -textCoord).w;
   vec3 normal=texture2D(depthMap, -textCoord).xyz*2.0-vec3(1.0,1.0,1.0);

   offsetArray[0] = vec2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = vec2(1.0/SCREENX, 0.0);
   offsetArray[2] = vec2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = vec2(0.0, 1.0/SCREENY);
   offsetArray[4] = vec2(0.0, -1.0/SCREENY);
   offsetArray[5] = vec2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = vec2(-1.0/SCREENX, 0.0);
   offsetArray[7] = vec2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, texture2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-vec3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       gl_FragColor=vec4(1.0,1.0,1.0,1.0);

   }
   else {  
     
       gl_FragColor=vec4(0.0,0.0,0.0,0.0);
     
   }


}

You have to give to the pixel shader the following data :
depthMap : Rendertarget : XYZ = normal, W = depth (view space)
SCREENX, SCREENY : depthMap size, must be floats.

This is a postprocess effect, so you might apply this shader to the quad you're using to do this post processing and put the depthMap texture on it.

Now, here is the antialiasing stuff :
We blur detected edge with a simple 3x3 blur kernel.

Result of the effect :

new pixel shader :

PS :

Code: Select all

// antialiasing pixel shader
// Author:Stefbuet
// Date:14/06/2010
// Last update:07/03/2011

//this shader is doing an edge detection with depth and normal map
//then it blur edges with a more or less good quelity filter (user selected)

uniform sampler2D depthMap;
uniform sampler2D renderMap;
uniform float SCREENX, SCREENY;

varying vec4 textureCoord;

vec2 offsetArray[8];

void main() {

   vec2 textCoord=textureCoord.xy;

   textCoord.x*=-1.0;
   
   float currentDepth=texture2D(depthMap, -textCoord).w;
   vec3 normal=texture2D(depthMap, -textCoord).xyz*2.0-vec3(1.0,1.0,1.0);

   offsetArray[0] = vec2(1.0/SCREENX, 1.0/SCREENY);
   offsetArray[1] = vec2(1.0/SCREENX, 0.0);
   offsetArray[2] = vec2(1.0/SCREENX, -1.0/SCREENY);
   offsetArray[3] = vec2(0.0, 1.0/SCREENY);
   offsetArray[4] = vec2(0.0, -1.0/SCREENY);
   offsetArray[5] = vec2(-1.0/SCREENX, 1.0/SCREENY);
   offsetArray[6] = vec2(-1.0/SCREENX, 0.0);
   offsetArray[7] = vec2(-1.0/SCREENX, -1.0/SCREENY);

     
   //depth
   float depthDif=0.0;
   float minD=0.005;

   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[0]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[1]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[2]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[3]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[4]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[5]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[6]).w-currentDepth));
   depthDif+=min(minD,abs(texture2D(depthMap, -textCoord+offsetArray[7]).w-currentDepth));

   depthDif/=(1.0+5.0*currentDepth);
   if(depthDif<0.01) depthDif=0;
   depthDif*=25.0;
   depthDif=min(1.0, depthDif);
   depthDif=sin(depthDif*2.0);

   //normals
   float t=1-abs(dot(normal, texture2D(depthMap, -textureCoord.xy+offsetArray[0]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[1]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[2]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[3]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[4]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[5]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[6]).xyz*2.0-vec3(1.0,1.0,1.0)));
   t+=1-abs(dot(normal, texture2D(depthMap, -textCoord+offsetArray[7]).xyz*2.0-vec3(1.0,1.0,1.0)));
   
   t/=15.0;
   t/=(1.0+2.5*currentDepth);
   if(t<0.16) t=0.0;
   t=sin(t*3.74);

   depthDif+=t;

       
   if(depthDif>0.2) {
       
       vec4 finalColor=texture2D(renderMap, -textCoord.xy);
       for(int i=0; i<8; i++)
           finalColor+=texture2D(renderMap, -textCoord.xy+offsetArray[i]);
       
       finalColor/=8.0;
       gl_FragColor=finalColor;

   }
   else {  
     
       gl_FragColor=texture2D(renderMap, -textCoord);
     
   }


}

You have only to give a new data to the shader:
renderMap : render target which contain your scene.

This method is used by Stalker to do antialiasing.
See http://http.developer.nvidia.com/GPUGem ... ter09.html after the deferred lighting part.

I saw that bluring far away things may let them too blurred. Maybe modulate the blur in function of the depth of pixels is a good idea, but I haven't tested it yet!

Hope you will enjoy it

slavik262 · Post by **slavik262** » Tue Jul 27, 2010 9:11 pm

Cool. I was thinking of writing a similar effect - I'll definitely use this as a basis for mine.

andres · Post by **andres** » Tue Jul 27, 2010 9:14 pm

Very good. Thanks!

wing64 · Post by **wing64** » Wed Jul 28, 2010 8:42 am

Good job.

greenya · Post by **greenya** » Wed Jul 28, 2010 10:12 am

Second screenshot pair:
anti-aliasing stuff:
48 fps with OFF and 230 fps with ON --- incredible indeed

slavik262 · Post by **slavik262** » Wed Jul 28, 2010 11:59 am

greenya wrote:Second screenshot pair:
anti-aliasing stuff:
48 fps with OFF and 230 fps with ON

How is that possible?

ACE247 · Post by **ACE247** » Wed Jul 28, 2010 1:18 pm

Whats going on here? 48 without and 230 with??? Some hardware fps boost.
And i see you didn't mix them up, since the blurred vs unblurred is clearly visible.

DtD · Post by **DtD** » Wed Jul 28, 2010 4:18 pm

lol on the FPS. Looks great!

ACE247 · Post by **ACE247** » Thu Jul 29, 2010 3:40 pm

Hmm.. I just got the idea that this might be usefull for doing something kinda like soft particles for when they intersect with other geometry like the terrain to blur the intersect edges out. Or am I wrong?

booe · Post by **booe** » Thu Jul 29, 2010 4:58 pm

Hum, stefbuet, you must have unwittingly switched the "on"/"off" labels.
The "on" label should belong to screenie with 48 fps, and "off" label to one with 230.

ACE247 · Post by **ACE247** » Thu Jul 29, 2010 5:43 pm

No he hasn't, you can clearly see the smoothed out edges in the different screenshots.

slavik262 · Post by **slavik262** » Thu Jul 29, 2010 6:16 pm

ACE247 wrote:Hmm.. I just got the idea that this might be usefull for doing something kinda like soft particles for when they intersect with other geometry like the terrain to blur the intersect edges out. Or am I wrong?

It's funny you should say that; I was looking at some soft particle code the other day from the DirectX SDK. Soft particles work by rendering the particles to a seperate RTT. You also render the depth of the particles, and then when you add the particles back into the final frame, you compare the depth of the frame against the depth of the particles. You only draw the particles if they are in front of the rest of the frame, so that way they don't clip into geometry.

stefbuet · Post by **stefbuet** » Thu Jul 29, 2010 6:24 pm

Ok, here is the FPS answer :
3dsmax was opening when I took the screenshot.

The shader don't change the FPS, however as I'm on a geforce 260GTX, it might change a little for lower spec graphic card.

slavik262 · Post by **slavik262** » Thu Jul 29, 2010 7:26 pm

Since it's completely dependant on the pixel shader pipeline, performance is mostly a hardware question. On a decent graphics card it won't slow anything down, but it will choke on crap hardware (which is why games usually make post-processing optional)

devsh · Post by **devsh** » Tue Nov 30, 2010 2:39 pm

you should be careful with normal map edge detection cause otherwise you will end up blurring bumpmapped surfaces so be sure to attenuate your normal found edges by the depth edge detection.. I could think of a very complicated algo but it would be slightly inefficient... I made my own that also takes the color buffer into account so I might share it in some time