ocgeo,
Couple of thoughts about Coulomb and M-O analysis for cohesionless soils:
In a static analysis, the backslope can not be greater than the soil phi angle.
In a seismic analysis, the backslope can not be greater than the phi angle minus Tan-1(Kh) (from memory).
The relationship is relative, low phi angle -> low slopes, high Kh -> low slopes. There is no one value such as 6 degrees or 30 degrees but it is the relationship between slopes, phi angle, and Kh that governs the inability to solve.
A trial wedge solution can work for a retaining wall (not considering that a slope may not be safe by itself) if the slope breaks at some point or there is a cohesive component to be considered. When the equation solutions do not work, other equilibrium methods have to be employed.
I have not messed with the slope stability method in a while but it can be a method to come up with a static or seismic thrust on a wall assuming that slopes are not infinite and/or cohesionless. All methods will not solve if there is only soil friction and an infinite condition to contend with.