Is the load strictly a 3 wire load or is there also neutral connected loads? I would presume no, because in North America neutral loads are forbidden with impedance grounded systems.
You have to make sure the auto-transformer has a zero sequence path to properly operate the ground fault detection. An open delta auto-transformer made with two single phase units will not interrupt zero sequence and hence not bother the ground fault detection. If you do have neutral loads, three single phase transformers connected in Y have to have the neutrals connected to the system neutral or they will block ground fault currents and prevent the fault detection from working.
As for sizing: Take the motor service factor current and multiply by 1.25. This will be the current in the series winding. I would suggest you use 600V:120V transformers. At 510V in they will give you 612V out. If that's what you choose, use the service factor times 1.25 times another .12 to get the kVA rating needed.
You will have to address warros's concern about open circuit voltage unloaded. Is the motor controller at the load or at the feed end? If the controller is at the feed end, you don't have too much of a concern unless the motor can run with zero load. If the controller is at the motor end, you will have concerns about the open circuit voltage burning out the controls. You'll have 720V when the motor is off. I would start with putting the transformer after the contactor and seeing if the controls have enough voltage to operate.