Actually the problem is not the denominator being zero, but the term sin(phi-beta-theta) becomes negative when phi<beta+theta which leads to a negative value inside the square root. I have a case with friction angle of phi=35,a backfill slope of 2/1(H/V), then kh cannot be larger than 0.1g...