If you have a system with transfer function H(s) and negative feedback loop with function G(s), then overall system is
H(s)/[1+G(s)H(s)].
As can be seen from the equation, the system will blow up when the product G(s)H(s) approaches -1. If you plot trajectory on the nyquist plot you can see how close it comes to the -1 point. Gain margin is how far below |1| the gain was when angle passed through 180. Phase margin is how much ahead of 180 was the phase when gain decreased through the magnitude |1| circle. Same info can be gained from Bode plot. There's lot's more that can be said....
