When a hole is cold-worked, there is a residual compressive stress field established around the hole. This benefits the fatigue capability of the hole greatly since local tensile stresses need to be in exceess of the residual compression to create an opening displacement for a flaw. In a sense, the highly localized stress at the hole has a -R. The hole severity factor is greatly reduced as can be seen in the hole condition factor for SF computation.
This residual stress field is not huge but obviously has some radius. There are several resources I can think of that discuss this:
Exploratory Investigation into the Durability of Beneficial Cold Worked Fastener Holes in Aluminum
There is a discussion of strain and stress measurements around cold worked holes in test specimen, plotted against the radial distance from the hole.
Also useful might be:
NASA TN D-6955, An Elastic Analysis of Stresses in a Uniaxially Loaded Sheet Containing an Interference-Fit Bolt
This provides formulae for determining the stress around a hole which has interference. You can plot the stress versus radial distance.
This shows the effect dies off pretty quickly, but not immediately. Most of the beneficial stress field is contained within a distance of 0.75R from the hole edge. However, oversizing a fastener will not remove that much material.
I think the answer is that you can enlarge a hole which has been cold worked without losing all of the advantage. But the beneficial stress field will be diminished.
Keep em' Flying
//Fight Corrosion!