If you use ADD_COLOR, you have to change the pixel values towards black. Quite similar is ALPHA_CHANNEL, where you have to change the alpha values towards 0. With VERTEX_ALPHA you only need to change the vertex color's alpha vlaues towards zero, which is only 4 values usually. However, it's hard to combine these transparency settings. You could try and use the alpha channel and the alpha threshold if you have your alpha channel built up as a gradient. Or try with the ONE_TEXTURE_BLEND material, which offers multiple transparency settings.
For one_texture_blend you need to call the method pack_texureBlendFunc, which takes all the necessary input for this material. The alpha source is the last parameter. There you can store whether to use vertex alpha or texture alpha, or both.
i have same problem. i need transparent textured mesh, with vertex alpha change. its not impossible, because the particle system can do it.
i solved it with own shader, but i would happy with a solution without any shader. hybrid! can you write the exact setting for that?