We use a circular area in orifice because it is repeatable and simple. Then we used emperical results to generate the equations for metering. If you read Miller and Spinks along with AGA3, there was some work done on putting a "weep hole" at the bottom of the orifice to allow liquids to drain from time to time. The resulting data inducated that if the area of the weep hole is added to the area of the orifice and a new pseudo diameter is entered into the equations you can get resonable results. I don't recall error analysis results but +/-3% from base isn't bad.
NOW, could this idea be expanded to say a square hole, I believe it will get a resonable result. It might make for an interesting comparison.