Your equation is correct. The negative sequence current seen on the primary is equal to secondary side negative sequence adjusted by the effective turns ratio.
If you are off by 1.732, you need to account for the delta-wye connection. The actual turns ratio of the two windings is not the same as the effective turns ratio at the transformer terminals.
busbar,
With advent of digital relays, negative sequence current OC relaying is essentially "free". It can be used to some advantage, especially for protection against line-line faults, since these will have neg. seq current while 3-phase faults do not. Because the neg.seq element is insensitive to balanced load current, it can be set below transformer full load current, unlike the phase OC elements.
And the neg seq on the primary of the delta-wye transformer will also detect SLG faults on the secondary. The benefit of this is questionable, since the same relay will also provide residual OC protection against ground faults from the secondary CTs.
On the other hand, I've never actually put the neg seq elements into service on any of the relays I've set. Generally, you want some other OC element to trip first if the fault is outside the transformer differential zone, so you end up needing to coordinate with phase OC elements anyway.
I think there is a tendency to think that we must use every feature on these relays because the features are there. But the more trip functions that are active, the greater the risk of a false operation.