When I was faced with a rectangular flange design this was the approach I adopted. It was very low pressure and a soft gasket. If you have high pressure you have to give the problem very serious thought because Square and pressure don’t go together !
Take the rectangular circumference or perimeter of the flange at the bolts. Consider the flange to be circular with the same circumference.
Example
Consider a 10 x 5 flange, perimeter = 30, Area = 50
Therefore Equivalent circle Dia = 9.5 Area = 71
Reduce the design pressure by the ratio of the two areas
Equivalent Pressure = Design Pressure x 0.704
Calculate as a round flange using the equivalent pressure.
It may also be a good idea to draw the flange and divide its area into the individual rectangular pressure envelopes acting on each bolt and take the worst case area. Divide up the equivalent circle (dia 9.5) by the number of bolts, if this area is less increase the equivalent design pressure accordingly.
Unfortunately this analysis assumes the flange to be a ring, which it is not.
Consider the flange to be a cantilever with the bolt loads applying the bending force.
Then consider the pitch between bolts as a built in beam subject to its share if the pressure area. You should also consider defection as this may cause the flange to leak.
Do a mock up and test your design. If you need to comply with the ASME code you may have to do a burst test.
Have fun
