So I assume the bearing is falling out.
A proper press fit is going to hold the outer case, and it is not going to fall out unless there is too much thrust load.... or there is thermal expansion of the bore.
Press fits for soft material is not the same as for hard material, obviously because some materials are more maluable and they compress differently.
When I can not use a press fit, because sometimes I want the shaft pressed, and you can't press both, I use a flanged bearing and hold it in place with a screw. Or I C-Bore a bit and use a retaining ring.
I can not imagine a good press fit falling out. I don't think you are using a FN4 press fit without destroying the bearing, unless is it very large and the outer case is made of iron. FN4 fits are usually shrink fits, pressing to these are impractable. If you did happen to use a fit like this, it is not "falling" out.