Trying to use an estimate to tweak a distance indicates that you've not brought onboard the likely magnitude of the error bounds.
I think that it would be better to estimate approximately how much antenna-to-antenna isolation your transmit and receive equipment requires, then estimate approximately what the coupling will be, then compare the two to see if it's even close. If the two values are far apart, then either it's good news or bad news.
Let me try to explain why the error is likely to be large in this situation.
If you're calculating pathloss between two antennas, when they're aimed towards each other, the antenna gain on the bore sight is well specified. In those circumstances, it's feasible to calculate a reasonably good estimate using the old "32.45" pathloss formula (just Google 32.45 and you'll find it). It'll still be an estimate, and external factors will increase the pathloss on bad days.
When the coupling depends on the two antenna's side lobes, and you intentionally arrange things to try to get into the nulls, then the coupling begins to have sensitive dependence on uncontrolled details. Are you in the Near Field? Is the coupling via diffraction at the edge of the dish? Do the nulls in the side lobes rotate due to a myriad of variables? Are there multiple effects such that there is multipath, thus making the coupling a complex function of frequency?
Have you considered installing just one antenna and using a duplexer to filter the transmit and receive bands? Virtually all real world applications find it cheaper to use a duplexer rather than install and maintain a 2nd antenna. This assumes the T/R frequencies are separate (a mandatory requirement).
To calculate the rough order of magnitude coupling, use the pathloss formula and then adjust for the side lobes. Use the min (nulls) and max (peaks) side lobes to speculate on the error bounds.
Estimate your Near Field distance, tricky because the aperture doesn't really apply to side lobes.
Compare this rough estimate to what your equipment requires. Don't forget to include cable losses in your spreadsheet.
Keep in mind the error bounds. Only an issue if your numbers land in the decision middle.
Consider using a duplexer and just one dish. Look up 'Low PIM cable' for related info if your go this route.