Frequent sources of instabilities
The FAQ 'Warning stiffness matrix is singular, structure is unstable' explains how to find the indicated (mesh) nodes.
This FAQ explains some frequent sources of these instabilities.
Hinges and connections
In most of cases the instabilities are due do errors in the modeling of the connections between elements or in the definition of hinges.
A common mistake is the release of all rotational degrees of freedom aiming at removing bending moment from the beam elements, as shown in the following image:
This setup will allow the beam to rotate on its own axis, leading to instability.
The 'fix' degree of freedom should never be released on both extremities of the same beam.
Furthermore, a more elegant and effective way to remove bending from beam elements, without using hinges, is to declare them as 'axial only elements', as in the image below.
Surface support
Another common source of instability arises when using surface support without specifying a support stiffness in the horizontal direction:
Unless the structure his blocked horizontally via additional supports, the C1x and C1y value should never be equal to zero, because in this case the structure is free to move horizontally, thus leading to instability.
Instability during nonlinear analysis
In case that the linear analysis run flawlessly while nonlinear analysis stops due to instability, that means that the structure is not able to carry the applied load. In fact, the nonlinear solver can find a solution only when the structure is stable: when the structure is instable (buckling) the nonlinear solver stops. In such cases you should review/change the design of the structure so to allow it to support the applied load.
To confirm that you are facing this case, you should perform a linear stability analysis to check the value of the lowest critical alpha: if this value is close to or lower than 1, it means that the structure will not be able to carry the load when subject to the nonlinear analysis.