How do you measure the impedance of a balanced transmission line?

Probably a formula is the easiest answer. If not you could make your own TDR system. You need a differential fast pulse signal, which can be obtained from an ECL gate for example. Then drive the line through a pair of series termination resistors. View the result on a high speed scope using an active probe. None of this is easy and you do need specialised gear.

You basically can’t make good measurements without some decent test gear.

Then again, if you need to know the impedance you must be driving the system fairly fast anyway. If you can’t measure the effect of changing the sending end terminations it may not be a problem anyway.

depending on your frequency band, you could calibrate into standard 300 ohm twinlead using a off-the shelf 75-to 300ohm balun & using a short, 1/6lambda, 1/3 lambda shorts or 300 ohm load and open cal standards whatever you prefer, or using tsd etc. then using the measured reflection coefficient from the 300 ohm system is straight forward to calculate the impedance. This requires a formula that every microwave engineer learns in 101 however