H B, if you closely study and understand flange design in, for example, ASME Sec VIII, Div 1, Appendix 2, you will see that the method can be extended to non-circular flanges.
Concepts are the same. The relevant dimensions, quantities, etc. only need to be translated from a circular to a non-circular geometry. A loose type flange will be simpler to adapt the methods to than an integral type flange
If your geometry is such that you have metal-to-metal contact outside the bolt pattern (ref: ASME Sec VIII, Div 1, Appendix Y) things get more complicated; I would not recommend this approach.
Reading in Taylor Forge Bulletin 502 if you can find it or any pressure vessel design handbook will be useful as well.
Regards,
Mike
The problem with sloppy work is that the supply FAR EXCEEDS the demand