In my experience, if the right compaction equipment is used for the type of material being compacted, if the material is near its optimum moisture, is not segregated (not a concern with clay) and lift thickness is around 6" to 8", compaction is achieved in about 4 passes. If it takes more than six passes, one or more of the criteria above needs adjustment. Too many passes with a vibratory roller will loosen the material rather than compact it. This is one of those cases where more is not necessarily better.
To determine the correct number of passes, use a nuclear densometer to check the density of the material after every pass and make a simple graph of the density rise. At a given number of passes, the density curve will drop ("break"), much like you see in a moisture/density curve of a proctor plot. If additional passes are made after the break point, the density will increase with one pass and then decrease back down again on the next pass. Once you find the break point, you have all the compaction you're going to get with that roller regardless of the number of passes made. If the density meets specifications at that point, everyone goes home happy. If not, one or more changes must be made to the criteria items above, usually the moisture content.
One other possibility is that you never find the break point in the compaction curve. In this case, the density goes up with each additional pass, but in decreasing increments until eventually no further densification occurs. If this happens, most likely the lift thickness is too thick and it is very unlikely that the required density has been achieved.
An exception to all of the above is when a modern roller is used that utilizes “intelligent compaction” technology. These rollers have the ability to measure the stiffness of the material directly using accelerometers and reduce energy output as they near what would otherwise be the break point. By reducing energy output, they can continue to increase densification with additional passes.