If you're designing a direct down converter then LO-RF Isolation is very important. If you have 20dB LO-RF isolation and your LO is at 0dBm then that means you are radiating -20dBm LO power out your antenna. The direct down conversion means the LO frequency is at the center of the desired signal channel frequency. If your desired signal is -80dBm and that enters the antenna with your -20dBm LO, your LO will mix both the -80dBm signal and the -20dBm LO down to baseband. The -20dBm LO will down convert to be a big DC offset that can saturate your receiver back end.
Not to mention, if you radiated -20dBm LO out your antenna the FCC would kill you. Typically, for cellular phone applications, the maximum radiation you're allowed is around -90dBm. Any LO you radiate out your cell phone looks like a jammer to the guy next to you (on his phone).