peterb's item 1 best describes the problem. Assuming that the relay function is to sense ground faults, delta-primary (or open-delta) PTs will not allow the relay to see neutral shift [zero-sequence voltage]. With a ground on the system, usually phase-to-phase voltages do not necessarily change much, and that's the only quantities that delta-connected PTs are ’passing’ to the relay. Wye-primary, wye-secondary PTs are the only practical way to do this. (ZZ-connected PTs will work, but that’s a lot of unnecessary extra work.) The PT primaries must be rated for full phase-to-phase voltage, even though connected phase-to-ground. ‘ANSI C57.13’ PTs are meant to withstand continuous 15% overvoltage, and only intended for 173% for a very short interval, which is what you’re asking them to do long-term in a ground-fault situation.
Example—take a hi-resistance-grounded 12kV system with 60:1-ratio PTs connected phase-to-ground. Across each secondary winding is about 115 volts (12,000/3^0.5/60). With balanced/symmetrical high-side voltage, each primary is seeing 6930V. Now, fault a phase and then one PT has zero primary volts but the two others are at the full phase-to-phase potential, or 1.73x ‘normal.’ You need to go to 100:1 PTs that are meant for 12,000V. Note that now each PT secondary runs about 69V under normal conditions, but a full 12kV across a primary won’t cook the winding. Make sure your relay is intended to operate satisfactorily in this situation.
A grounded-wye primary, grounded-wye secondary bank with primary coils rated for continuous phase-to-phase voltage is the only sure fix. The caution with this arrangement is that steady-state phase-to-neutral voltage is only 0.577 (1/3^0.5) per-unit of ‘normal.’
Follow the PT manufacturers recommendations for high- and low-side fusing for units mounted in switchgear.
In hi-resistance grounded systems, a grounded-wye/broken delta connection is often used with a sensitive overvoltage relay for ground sensing, (called a ‘59G function’) but this quantity can be calculated in numerical relays based on 4-wire wye PT inputs.