Square or rectangular tubes have problems with fatigue in the corners when subjected to pressure cycles.
Still, they might give better heat transfer in some configurations, i.e., if they can induce some extra turbulence on the shell side.
But the most likely answer is that there's no simple way to terminate the ends of square/rectangular tubes to a tubesheet.
You could use square tubes that are formed with round ends and terminate them in the usual way, but that leaves no way to replace individual tubes, e.g., pulling a square tube through a round hole.
Note that obround tubes are used extensively in automobile radiators, but they are never individually replaced, and are assembled with external fins, for which the flattened sides are a great advantage in conducting heat from the tube wall to the fin material.
Mike Halloran
Pembroke Pines, FL, USA