The strongest section in bending is that with the highest section modulus. For tension, it is the one with the highest cross sectional area.
In general, I think the square tube is stronger.
For example, a 6" sch 40 (0.280" wall) pipe has a section modulus of 8.50 in^3 and a 6" square tube with 0.250" wall has a section modulus of 10.1 in^3. So, the square tube is stronger in bending. The area of the 6" pipe is 5.58 in^2 while the tube is 5.59 in^2 so they are about equal in tension.
Note that comparisons are difficult because wall thicknesses are not always this close.
Note that round sections are equally strong for bending in all directions, while the strength of rectangular tubes in bending varies with direction. Only you know the actual load conditions.
There are many other considerations, including but not limited to: material cost, fabrication cost, erection cost, appearance, corrosion, connection details, availability, etc.
