I will assume that it is a concrete beam, and if this is the case please look at PCA 318-95 Deep flexural beam chapter. It Covers the concept of design in details. Briefly, as the beam becomes deeper (comparing to the span), the strength diagram, and thus flexural forces become smaller. In such case if your opening at or near the middle of the beam span, you may need to add a horizontal member below the opening, something similar to the bottom chord of a truss. If no room for such member, you need to consider designing the beam supporters to sustain a horizontal thrust (a horizontal reaction due to the lack of not having the chord member). If your beam is continuous on both sides of the bay with the opening, you have a home run. Considering detailing the beam itself, you may design a beam above the opening (a shallow beam that is), which is supported on two small posted, one at each end, inclined and dragging the load back to main support. You do not have to detail the concrete body below the post to take the same shape. The posts are just embedded rebars, with proper ties, into the beam section.
Hope it helps!
Considering that it could be a steel section (a plate girder perhaps), then you would design based on a net moment of inertia, but also considering the stress concentrations that might develop at the corners of the cutout. Ideally, the cutout would be round; however, any shape can be done provided you accomodate the bending at that cross section and/or the net shear effect, particularly if near the supports.
