Rigorously calculating the required separation would require a lot of information that is likely not available to you. For example, the detailed performance characteristics of the receivers and transmitters (some are better than others). So you're probably left with doing the best you can and trying it.
I believe that vertical separation tends to provide more isolation than horizontal separation (assuming vertical antennas). Stand by for confirmation from others on this point.
If you offset (in the usual manner using a horizontal bar) the lower antenna from the mast by one wavelength (only about a foot, pretty easy), you probably won't see much impact on the pattern from a practical point of view. Since one foot is so short, maybe try two or three feet instead. A three foot horizontal bar shouldn't be mechanically difficult.