Engleosun:
There are two types of instability: structural instability and dynamic instability. The first is usually the condition when a member of a structure bifurcates ( a sudden change in the position to a lower energy state). The load can either be static or dynamic. The second is a more sharply focused on the dynamic energy levels of a structure (Motion of a structure is suddenly changed to another response. If no damping is present, unlimited growth can result).
Dynamic stability is best described using Chaos or other comparable theories. The classical dynamic problem is Mathieu instability. Most studies have been directed towards the response of shells of revolution. Unless the analysis program measures the dynamic response to a specific theory, it is difficult to determine that the observed response from the computer output can establish dynamic instability.
There are very mathematical conditions that can be expressed to determine dynamic instability-- Thompson and Hunt, "A General Theory of Elastic Stability", 1973. Others use the term "Catastrophe Theory" or "Chaos Theory."
Unless the FEM computer program allows geometric non-linear response and the introduction of imperfection geometry, one can not establish dynamic instability. On the other hand, if they do not have a criterion to determine dynamic instability, dynamic instability can not be determined.
The observed numerical response can have a condition known as numerical instability. This condition results from inaccurate time step size for the dynamic terms used in the acceleration algorithm being used. This can be detected by observing the response of one state being the negative of the next adjacent state. Things can be confusing when talking about dynamic instability.