Restricted earth fault relays are differential relays connected to provide sensitive protection for equipment against ground faults.  The most common usage is for the protection of delta-wye transformers with a resistance grounded neutral.
One CT in the transformer neutral is balanced against the neutral of three phase CTs in the transformer output connection (or bushing CTs in the transformer circuit breaker) and connected in parallel.  The REF relay is connected across the two CT leads - this relay is a high impedance type, operating on fault voltage developed across the CT leads for an internal fault.  For an external earth fault, the zero sequence current in both sets of CTs is balanced and there is no fault voltage developed on the CT leads.
The REF scheme is a high speed differential protection, with instantaneous operation for internal faults and high stability (depending on setting) for external faults.

By extension, an unrestricted earth fault relay is a relay connected to a single CT or set of CTs to measure zero sequence current.  This is a normal time delayed overcurrent relay, which must be delayed to coordinate with other relays downstream. This would normally be a non-directional relay, but could be directional depending on the application

Suggestions:
A. To peterb posting:
1. To the first sentence: The restricted ground fault relays provide more sensitive protection for the equipment (transformers or generators) internal faults to ground; not phase to phase (to phase). They are used to protect transformers rated at or over 10000kVA. Heavy external faults may trip this relay because of cts' asymmetries. To avoid this, additional choke, capacitor and short time delay (relay) 1 to 2 seconds are needed mitigate the effect of harmonics and DC component. Finally, a stabilizing resistor is connected in series with the relay. See page 181 of A. R. van C. Warrington "Protective Relays Their Theory and Practice" Volume I., 1971.
2. External faults are protected by another differential relay (Dev. 87), which is applied regardless of transformer 10000kVA size needed for the restrictive earthing relay.
B. To 140668: There is also a good reference IEEE Std 242-1986 "IEEE Recommended Practice for Protection and Coordination of Industrial and Commercial Power Systems."

Just to correct one or two misconceptions -
- The REF scheme requires matched CT characteristics on the phase and neutral; any mismatch in saturation voltage is taken into consideration by the use of a judiciously selected stabilizing resistor; the relay can be either a current relay, with an external stabilizing resistor, or a voltage relay with an integral resistor.
- The REF scheme setting is selected on the basis of the maximum votage developed at the relay for the worst-case external ground fault, taking the CT characteristics into consideration.  Normal scheme setting is <50% of the CT kneepoint voltage.
- The relays normally used for these schemes include harmonic filters; these would not normally be provided external to the relay
- Time delays are not usually applied with this type of protection
- External faults outside the zone defined by the REF CTs require a non-restricted scheme as backup protection. Device 87 is a differential relay, with a defined protected zone - this will not operate for external faults.
The REF scheme is a subset of high impedance differential schemes, which are generally used for bus protection.  An excellent reference is the Protective Relaying Application Guide, published by the former GEC Measurements (now Alstom T&D)

Please notice that the restricted earth fault relay (87TG in IEEE Std 242-1986 Fig. 196) is supplemented by another differential relay 87T that acts on external faults to 87TG. It is self-evident that the differential relay 87T protects equipment encompassed by its loop delineated by two sets of current transformers. In case of heavy faults actually backs up 87TG.

In the normal course of events, device 87T is the transformer differential protection relay.  This will protect both windings of the transformer against phase and some earth faults, depending on the position of the fault in the winding.  
The REF scheme protects the earthed winding to which it is applied for internal earth faults, with a more sensitive setting than is possible with the transformer differential relay.  The zones of protection for both 87TG and 87T on the protected winding are usually the same, with shared CTs, so that the 87T cannot be said to be protecting for external fault conditions.

Referenced IEEE Std 242-1986 Figure 196 on page 432 indicates four different sets of CTs. Two sets of CTs pertain to 87T and two sets of CTs pertain to 87TG. This is mandatory since the 87TG has the CTs connected in Y-connection and 87T has CTs connected in Delta-connection on the secondary transformer side. The primary transformer windings have Y-connected CTs for the 87T. Obviously, there is no way the primary set of CTs for 87T can be shared with one CT of 87TG applied to grounded neutral line of the transformer. When it comes to large generators, then the CTs are also different in the Generator Percentage Differential Relay (Phase Scheme) (87) and Ground Differential Scheme (87C), using a directional Relay 87 instead, as indicated in Figure 208 on page 454 in the above reference. Somewhat simpler scheme is seen in Figure 209 on page 455 where the Percent Differential Relay (87GN) is used for the Ground Differential Protection.

Final comments on this, to try to clear the air a little -

1. While I agree that the IEEE standards are an excellent reference, they are intended to summarise elements of engineering practice, not to serve as design standards to cover all cases for all time.  An example of the changes that have been made in accepted practice since the referenced date of 1986 is the introduction into general service of microprocessor-based transformer protection relays - these allow connection of the CTs on both sides of the transformer to be made in wye, with the required phase shifting and zero sequence current filtering being done internal to the relay.
2. Even prior to 1986, it was standard practice in the world outside of the IEEE standards for CTs to be shared between the transformer differential protection and the REF protection.  This was done by the simple expedient of installing a set of carefully selected interposing CTs in the circuit.  The transformer CTs on both the Delta and Wye windings were connected in grounded wye. The CT circuit from the Wye winding was then connected to the wye-connected primary of the interposing CTs, with the REF scheme connected to the neutral of this circuit.  The interposing CT secondary was connected in delta, providing the necessary phase shift and zero sequence current filter for connection to the differential relay.
3. My experience has been that large generators are always unit connected and consequently are earthed using a distribution transformer/resistor combination.  With this situation, earth fault currents are restricted to very low values and provision of a current-based earth fault detection scheme is inappropriate - earth faults are usually detected by neutral voltage measurement.

Anyway, 140668, I hope that all of this has helped to answer your original question, even if has been rather detailed.

Suggestions to the previous posting:
1. Switching to microprocessor relays is departing from the original posting. The microprocessor relays are often synthetized version of electromechanical relays with the substantial amount of sophistication and complexity imbedded in them.
2. There appears to be other solutions possible. My posting did not exclude those. I just do not see any references in the posting. Your statements seem to be reflecting your experience, which is fine.
3. The high-resistance grounding is limited by the voltage level and charging earth current. If those are not met, the medium resistance grounding or other grounding method is applied. That is where your "always" statement is misleading and inaccurate. The "final comments" are your prerogative since this is the public forum to address different views and elucidate topics without any contractual agreements.