That is a real open ended question. You should start by reviewing the standards in your country (perhaps IEEE)You should also review the big players electrical specifications to know what your up against, try SEL, Beckwith, GE Multilin, etc.
To go into any details about designing a generator protection relay is way to difficult in this type of forum. You should do some homework first and develop some specific questions related to measurement techniques or typical operation.