When the displacements (or rotations) become so large that they have significant effect on the stiffness, etc. In many cases this is quite a bit more than most people think...It really does depend on the problem...Consider..
A problem that is essentially a bar loaded axially...If the meshing is such that strains within elements can be kept "small" (which is also subject to interpretation) then a good solution can be obtained for displacements which are quite large overall....
Also, depending on the problem, rotations up to 10-15 degrees can be tolerated without totally trashing a solution (Which also depends on your individual tolerance for error)...Some people think any error over 1-2% is excessive while others may accept errors of 50-100% depending on the type of problem and known loads, material properties, etc....
Hope this helps.....
Ed.R.