I have a nonstandard keyway due to space constraints and I'm woried about compression. Is there any reason I can't stray away from standard keystock (1018 or 1045) and use something harder like 4140 Q&T?
There is no reason that you can't use a harder material, I've used hardened keys in the past with much success. Just keep in mind what mintjulep said, but you almost always have some repair considerations when a key shears, regardless of material.
This is an interesting question. However, you should be careful to consider what the industrial mechanic will replace this key with if they have a failure of some type. We recently had some issues with keys fretting and found out that our stock room was stocking 6x6 keystock material incorrectly (using 6x6 square stock c-1018. The tolerances are greater +0/-0.078, whereas keystock is +0-0.030.