The available techniques vary a lot. It depends mostly on what type of encoder you have got, what voltages and what repetition frequencies that are involved.
The standard encoder is typically fed with anything between 12 and 30 V DC, has two tracks with 90 degree phase difference and sometimes an index track. Fur such encoders, there are a set of standard measurements to be done. Check baseline (shall be 0 V +/- a few hundred millivolts), check amplitudes (shall be close to supply voltage). Then check phase difference between track A and B (shall be 90 degrees +/- five - ten degrees) and so on. Noise on low level and high level, cross-talk between channels and missing pulses are other things to look for.
If you have problems constantly, it is often easier to find the reason, while intermittent problems often are quite tough to identify.
If you have a less qualified measurement (like a prox that reacts to flags or something else) then you can often set up a 'mouse trap' in the PLC program that reacts to abnormal signals. Or you can use a transient recorder that trigs on either the signals themselves or fault signals.
There are so many different situations that it is impossible to give you a full picture. Your best bet is to contact someone that does this kind of work on a regular basis.
Gunnar Englund
--------------------------------------
100 % recycled posting: Electrons, ideas, finger-tips have been used over and over again...