1. Response to your first question - I wouldn't categorically say 'never' exceed 42 HRC (about 190ish ksi UTS). This is within the useful capability of this alloy, but it just depends on the design trade off between strength and toughness for your application.
2. As metengr pointed out, tempering some alloy steels in certain temperature ranges can result in brittleness such as you describe. Depending on the metallurgical phenomenology that is causing your part to shatter, the problem could be due to either ‘temper embrittlement’ or ‘blue brittleness’. 4140 alloy is supposedly more prone to the latter. Whatever the differences are, try staying out of the 450 -700° F tempering range. Impurities such as those already mentioned are a part of the problem. (To optimize toughness on one of our alloys we specify that the sum, P + Sn + Sb + As < [X-amount].)
3. For further details and rational on embrittlement causes and cures, if you have access, check out:
ASM Handbook, Vol 1, 10th Ed., pp 432, 698
Reed-Hill, Physical Metallurgy Principles, 4th Ed., p 705-6
SAE spec AMS-H-6875A, Heat Treatment of Steel.., Table 1A.
These references show a slighly higher austenitizing temperature than you are using, so you may want to consider bumping that up. It may have a secondary effect.