My recollection is that there is atomic repositioning once the atoms start getting incorporated at the interface. At higher temperatures, there is high atomic mobility within the existing lattice, so it's more likely that a given atom will find the correct and minimum energy position.
Don't forget that the oxide grows at the interface between the silicon and the oxide. So in addition to the above, the availability of oxygen at the interface is more limited at lower temperatures, so you might expect more voids, while at higher temperatures there may be an excess of oxygen
