Detonation is most likely around peak VE all other factors being equal.
But other factors aren't equal in my opinion. Firstly, let me say I'd like to replace the word detonation by knock in most of the previous postings.
Let's say that normal combustions has a flame front propagated by chemical reaction, and let's say that a faster mode of combustion is possible where the flame front is created by a mixture autoigniting due to increased temperature and pressure from the combustion that has already occured.
So let's say that in normal combustion the pressure wave front precedes the flame front, and in detonation the pressure wave front is the flame front.
Now, in an engine that gets closer and closer to its limits, what is going to happen? You might expect that there is a sudden transition from normal combustion to detonation. Thank goodness that isn't the case. The pressure from combustion is distributed by gases expanding. This is what causes the wave of pressure towards the edge of the cylinders. But when the expanding gas reaches the edge of the cylinders there is nowhere else for it to go and it bounces back.
Without having sufficient pressure for detonation the higher pressures achieved near the cylinder walls, where the pressure can no longer dissipate sideways, causes autoignition of gas near the cylinder wall, and a second flame front can start. So now there are two normal chemically propageted flame fronts, approaching each other, one say expanding outwards from a central spark plug, and another moving inwards from around the cylinder walls.
Let's call that knocking. Knocking occurs then, when conditions in the mixture are approaching the conditions required for actual detonation of the gas. Knocking then, is not actual detonation - thank goodness, otherwise it would be time to get a new engine.
Detonation is most likely around peak VE all other factors being equal. ?? I assume the train of thought here is that more air in means more pressure and autoignition of the mixture occurs more readily. We tend to think of pressure as something fundamental, due to it being easy to measure, but let's split it up: pressures can be higher when there is more gas physically present - a density thing; and when gases are hotter - a temperature thing. Higher volumetric efficiency is saying there are more gas molecules physically present in the cylinders. They aren't especially hotter; there's just more of them.
Now let's assume there exists an autoignition temperature. The higher volumetric efficiency then hasn't made the denser air particulary more prone to autoignition.
So what would make a mixture prone to autoignition and thus potentially make an engine more prone to knock?
Low Speed Knock
Well, the obvious way to get a gas closer to its autoignition temperture is to heat it. Let's say best BSFC is around 2000 RPM. Why is the engine less efficient at lower RPM? Heat is lost to the cylinder walls, as each cycle takes longer and the heat lost per cycle is therefore larger. A consequence of this is that the incoming fresh air in the mixture is heated more and the mixture is closer to its autoignition temperature. So more mixture heating time at low RPM causes low speed knock.
High Speed Knock
Let's say some engine redlines around 7500 RPM. What's stopping it just going faster and faster with more air and fuel giving more power until it breaks? Airflow. Suppose the size of the exhaust valve is restricting how much exhaust gas can actually get past it. Higher RPM doesn't increase flow and power as only so much gas can get past the valve. Some hot exhaust gas starts staying in the cylinder. Again, we can be heating the incoming mixture and creating conditions where end gases exceed their autoignition temperatures and knocking occurs.
Summary
There is nice good airflow around peak VE and knocking occurs when air in the cylinders is heating up for one reason (staying in there too long) or another (not getting out properly). I will now stick my neck out on the chopping board and say :
detonation is unlikely around peak VE all other factors being equal.
