When turning brake drums for trucks I've seen them use belts they wrap around the part.  Heavy lead lined belts might work wonders.

You could perhaps use a smaller grooving tool and take several bites at the groove - you mentioned cnc...

How about using pressurized air instead of coolant on this operation?  In most metals you can speed it up fast enough to get into "high speed machining" relm where the chip comes off red hot and plyable.  I did this with large oil-field pipe couplings "extreme line casing" stuff.  It worked wonderfully but scared the hell out of the operators...  It was boring also, but in 8" to 12" couplings, on a 65hp lathe...  Watch the chip build up - there is significant heat available to distort all sorts of things.

Larry

Not sure how's your tool look like.  But you can try to stiffen the structure.  This will alter the mode frequency.

It might be possible to dampen the hub as Machining Dude has mentioned.  I'm picturing something along the lines of a layer of damping material (rubber) bonded to the outside of the hub.  On the outside of the damping layer, another ring of metal is bonded to the rubber.  This should give you constrained layer damping, which will maximize the effectiveness of the rubber.  Figuring out the optimum thickness for the rubber and the outer constraining ring is not easy, but you may want to just try trial and error.  The inverse of this problem (ring inside) has been used successfully to quiet noisy gear trains.

As an alternative to what is mentioned above, you could try to install a squeeze film damper on the outside of the hub.  An outer ring with properly dimensioned internal O-ring grooves at each end would be fit over the outside of the hub.  The space between the O-rings is filled with grease.  The ring-hub fit is of the same order as what would be used for a hydrodynamic journal bearing of similar size (maybe ~0.004" diametral clearance).  The O-rings act as radial springs as well as sealing the grease in.  The grease is displaced and sheared as the outer ring moves, adding damping.  A similar device is used (with circulating oil rather than grease) in the roller element bearing supports in many gas turbines.  Of course, this approach would require machining the outside of the hub to a tight tolerance for the O-rings to seal.

The dampened tools that I have seen (which are very few) have a mass in an internal cavity filled with oil so that the oil shearing adds damping.  It may be possible to modify the tool to utilize constrained layer elastomer damping as I have mentioned for the hub.  

To optimize any of the above ideas would take a detailed FEA analysis coupled with actual vibration spectrums of the existing setup.  This would not be cheap.  Hopefully, you can take one or more of the ideas and do some trials to come up with a "good enough" rather than an optimum solution.