Hi,
at first I would check the phase currents which are connected to the relay. These should be symetrical (nearly the same magnitude and 120 degrees phase shift) under normal operation. The relay can show the measured values. If they are not symetrical maybe one CT is wrong (wrong ratio)? Or the relays input is not working correctly. You could verify the measurement with a current clamp.
Best regards,
Oliver Janke