The RF port in a mixer is usually the receiver port from e.g. a radar system. When there is no or low LO/RF port isolation the power from the LO port will be delivered to the rf port.
Let's futher assume there is an antenna attached to the rf port (usually you need this antenna in radar systems ) the LO power will be transmittet via this receiving antenna. Vice versa the received power from the RF port will be delivered to the LO prot, where an oscillator should be attached. And oscillators don't like it to be feeded with rf power at their output port. The same considerations are valid for the IF port isolation.
20 dB isolation means that there is only 1% transmitted. Usually anything less than 10% is considered to be neglected. So 20dB for isolation or a match can be considered to be fair enough.
I hope this will help you.

Well, the FCC gets pretty PO'd if you re-radiate the LO frequency out of your receiver antenna.  Also, the enemy likes to see you LO re-radiation if you are building a stealthy system, like a radar warning detector.

In WWII, German submarines had poor LO isolation in their communications receivers that leaked out their antennas and was detectable by allied forces since it was CW. Let's say that receiver design wasn't bomb proof.
If you want to be quiet, you shouldn't radiate out your receiver antenna.
kch

You can say the same today about automotive radar detectors.  The cops have receivers that can listen for the LO, and they know that you are using the radar detector.

I don't know why the radar detector manufacturers can not suppress the LO, using subharmonic mixers or notch filtering--maybe for cost reasons?

You remind me something very funny about this,

I used to work on a 30 years old receiver system which made of two segments, one was location finding and the other was just simple receiver...

Because, at those days it was imposible to produce a four phased match amplifiers all over P-KA bands, they connected the antenna directly to the mixers and those mixers had something like you mention - LO/RF isolation of 20dB ~ we received the LO of this system in the antennas of the other system !!!

Although its designing fail, it was help us to see "life" on the other system for quick "life" checking,
So, sometimes, it can be an advantage...

If you're designing a direct down converter then LO-RF Isolation is very important.  If you have 20dB LO-RF isolation and your LO is at 0dBm then that means you are radiating -20dBm LO power out your antenna.  The direct down conversion means the LO frequency is at the center of the desired signal channel frequency.  If your desired signal is -80dBm and that enters the antenna with your -20dBm LO, your LO will mix both the -80dBm signal and the -20dBm LO down to baseband.  The -20dBm LO will down convert to be a big DC offset that can saturate your receiver back end.

Not to mention, if you radiated -20dBm LO out your antenna the FCC would kill you.  Typically, for cellular phone applications, the maximum radiation you're allowed is around -90dBm.  Any LO you radiate out your cell phone looks like a jammer to the guy next to you (on his phone).  

So actually, from my last post, it's not just Mixer isolation you have to worry about ... but it's the Mixer iso. plus the reverse isolation of the LNA.  In situations where the desired input signal to the LNA is high power the lna is most likely in a low gain mode where it may even be switched off and bypassed with a switch.  The reverse isolation of the LNA in that case would just be the insertion loss of the switch... which by the way would be low to keep your noise figure down.  So, the isolation requirement of the Mixer in that scenario would be perhaps around 90-100dB for a direct downconverter.