Bob,
Using an end mill will help a lot, the edge cuts into the tube wall and helps prevent sliding. The more fool-proof way is to make a "drill guide" and harden it after you have the guide hole, then use an end mill in the drill guide, as even the mill will deflect as it reaches...