The auto-start functionality should be tested when the unit is down. The standard test would be to put the main pump on line (usually the turbine drive on our systems). Place the spare pump in "auto" and trip off the main pump. This will allow you to verify that the auto-start switch is working properly. The spare should start up and catch the pressure before the trip point is reached. This can also find problems if the pressure control valves and the pressure relief valves want to fight. I have seen systems where the pressure would spike up when the aux pump came up to speed, the full-flow relief valve would open to relieve the excess pressure. The spill-back pressure control valve would overcompensate for the loss of oil through the relief and the two valves would get into an oscillating hunt.
In answer to your direct question, all of this testing should be done with the compressor down. Once the compressor is up and running, it would be too risky to repeat this testing. BigInch is correct that you should be running your spare pumps on some set frequency. We run ours once per month for four hours. On most systems it would not be practical to switch back and forth to get equal run time on both pumps. Generally, only one pump is set up to auto-start. If you want to verify auto-start capability on the system while running, you could have your instrument techs isolate the auto-start switch and pump it up to test with the aux pump turned to off (not in auto).
Johnny Pellin