My experience with stop-off paint is for it to be completely effective, the material has to be very clean, the application has to be completely dry, and two or three coats are needed. We used to put a heavy case (approx. the same depth as you mention) on parts that had threaded ends. Tolerances reuqired threading after heat treatment. Obviously, the threading operation could not tolerate any bleed-through of the carburization. We tried several methods, and settled on
1) burn-off treatment at 400-500 F for an hour.
2) cool and shot-blast if any residue was evident.
3) if shot-blast, repeat burn-off (shot-blast would leave contaminats)
4) apply 1st coat - dry
5) apply 2nd coat - dry
6) apply 3rd coat - dry
7) bake at 250-300 F for 2 hours
8) charge into carburize furnace before parts allowed to cool below 100F
Eventually, we got a "warmer" that would allow us to dry around 150F and then raise up to 250 for bake. Prior to that, we would just air dry for 8 hours.
Different stop-off products have different characteristics. I know of one company that performed copper plating by electrolysis to insure stop-off.
I am a little confused on your comment "The part has to be transported appx. 50 mts from furnace to quench tank." Do you mean 50 meters? If so, this doesn't make much sense since it isn't distance that is the problem, but time. If you mean 50 minutes, I don't see how you keep the parts at temperature. If it's 50 meters, you should be able to keep the transfer time to below a minute or two (thats less than 1 meter per second). Other than a little surface scaling, that shouldn't be too much of a problem.
Regarding the IGO and carburizing you are seeing in the threaded areas, this is obviously failure of the stop-off treatment. Depending on the endo gas composition, you can have both carburizing and oxidation occuring at the same time. I think this is the reason Cory suggested better control on the atmosphere.