I'd never (maybe that's a bit harsh, but I certainly can't think of any exceptions) use an ungrounded 480V delta secondary. Bad, bad news. The thought behind them was to have an ungrounded system, but there really isn't any such thing and now you will be subjected to all sorts of voltage problems during any faults. Far better would be a wye secondary with a high impedance (high resistance) ground. An arcing fault on a system grounded only through the system capacitance can develop voltages to ground several times the line-to-line voltage; limited when the system flashes-over somewhere else.
The voltages you report are certainly not unexpected with that type of system. What to do about it: 1) nothing, that type of system has been used for years; 2) not very likely, but it could be grounded as waross suggests (never seen a transformer that would allow that, but that doesn't mean they don't exist); 3) corner ground the system; or, 4) add a zig-zag or wye-delta grounding transformer and either solidly or impedance ground it that way. Option 1 leaves you with what you've always had. Options 2 and 3 provide a solidly grounded system, which many industrial user want to avoid for system reliability/availability reasons. Option 4 would provide the best solution, but certainly costs much more than any of the other options. I suppose an option 5) would be replace your new transformer with an even newer transformer and impedance ground it, but that would be far more expensive than option 4.
The advantages of option 4 (or 5) would be stable voltages to ground during normal operation, limitation of voltage rise during arcing faults, and the ability to trace the location of a single ground fault. In the ungrounded system it is far more difficult to trace down a ground fault without sequentially turning off portions of the system.