No, it's not "all" about grain size, but that is part of it.
As mentioned above, forging temperatures (>2300 F) are high enough to cause a large grain structure. Also, the cooling rate from forging temperatures is often not controlled. Often, the parts are just thrown into a large bin off the forge until the bin in full. Once the bin is full, it is often just left alone until the next day and they are fully cooled. So, all of the parts have cooled slowly, but some of them could have very well spent an extended amount of time at a temperature where both austenite and ferrite are stable. Because of the large grain size, this can result in considerable amount of micro-segregation of carbon, with large ferrite grains and an austenite decomposition product that can be anything from a coarse pearlite to spheroidized carbides. What makes matters worse is that nearly all of the carbon, and, depending on the alloy content, a lot of the alloying elements, can be tied-up in these carbides.
This structure can be very difficult to austenitize since the ferrite is very low in carbon, so it has to either get above 1700F or so or absorb carbon from neighboring carbides or austenite (that has already absorbed carbon from decomposed carbides). As the ferrite grains can be large, and the carbides can be large and spheriodal, this can take a surprisingly long amount of time.
One problem this causes is that not all of the forgings in the bin will have this condition. Some of these parts will cool rapidly enough so when they will austenitize easily and the only effect will be perhaps a slightly larger grain size than would have been obtained if a normalizing operation had been performed. Others, however, may have this problematic structure. The result is that if you heat treat a batch of 100 and select one for destructive testing, it may produce good results, but others in the batch may not have responded as well.
A normalizing operation will make sure all the parts have the same starting microstructure, eliminating the problem.
If you control the process, you may be able to eliminate this problem without needing a normailzing treatment. At my first professional job, we had a set-up where the parts came off the press onto a controlled cooling conveyer and were not put into the bin until the temperature was below 800F. After rough machining, the parts were given a Q&T and had to meet very demanding tensile and impact properties. The process was tightly controlled and produced very consistent results. We were producing our own parts, however, and we only had to satisfy our own engineers, not a host of engineers from various customers.
rp