<rant>
The phrase "minimal o-ring compression" should be a red flag, as should "maximal..". Any deviation from o-ring manufacturer's handbook specs is a sign that a dilettante has been messing with the design, and the whole thing will be a headache, forever.
</rant>
What's important to the o-ring is the radial depth of the gland, as assembled. The groove can be elliptical, or look like a polydyne cam, and still function, so long as the radial depth of the gland, not the groove, at every point, gives the right radial compression, and the perimeter gives the right circumferential compression.
<rant>
A CMM is the the worst, most error- prone, most labor- intensive possible choice for the task of measuring an o-ring groove.
</rant>
You have clearly figured that out already.
Unfortunately, the depth of the gland that's important to the o-ring is determined by two (or possibly more) parts, and you only measure one at a time, so you have to apportion the available tolerance (not much) among the features that define the real dimension of interest.
In this case, the real limit on roundness is partially determined by the clearances between the locating features of the parts in the assembly, and their allowed eccentricity relative to the o-ring faying surfaces. We can't evaluate that from here, but someone needs to do it. You can't just pull a number out of the sky.
A meteorologist can't help you. A metrologist might. He'd probably say that you need to be measuring runout relative to whatever feature locates the mating part(s).
If you have the time, put the problem into Excel, and work out what happens as the dimensions and eccentricities change. You'll probably find that the whole assembly needs to be retoleranced.
Mike Halloran
Pembroke Pines, FL, USA