Your question depends on the Tx/Rx frequencies. Isolation is measured by the return loss of the OMT of the feed in both H & V and is usually conducted in a range test. I'm speaking of microwave frequencies way above UHF (Ku-Band for example) using parabolic antennas.
The isolation is provided in part by the orthogonal design of the OMT (Ortho-Mode Transducer) itself, and can range from 30 to 35 dB between V & H (Ku-Band). But this isolation is greatly increased by the use of the transmit reject filter. So together, the isolation of 70-75 dB can be realized.
The V can be used for Tx and the H for Rx, or vice versa. The uplink & downlink frequencies themselves provide a measure of isolation. But what you don't want is an OMT that passes both Tx and Rx through the same port. Hence, the transmit reject filter.
I don't personally bother with formulas or calculations, but use instead range tests of randomly selected parabolic antennas. This involves return loss tests of the OMT on both ports. This is more practical because it's based on actual test data. The transmit reject filter simply adds to the isolation by rejecting Tx signals at the Rx side of the OMT. For the Tx side, HPAs usually have Rx reject filters already incorporated inside them.
Sometimes even manufacturers of such parabolic antennas, especially those that include feeds, include isolation data in their specifications.