You might have better luck using a thin hollow tube with a sharp edge (e.g. a hollow punch, sometimes called a washer cutter or gasket cutter).
Sanding/grinding works okay for material removal (shaping), if cutting pressure is kept low.
Beyond that, if you must machine, try freezing the part first (freeze it in the vise/collet/holder to help it stay cold longer). Freezing it may help make the rubber stiffer and not flex as much when cutting/machining. Maybe try doing the machining inside a walk-in freezer...