Angular momentum is "conserved", you just aren't looking closely enough. The theory of conservation of momentum and angular momentum is the statement "a body's momentum is conserved unless outside forces act upon the body". If you compute the shear forces at the wall, there will be a tangential component (drag). These shear forces arise from the swirl flow, but in the equivalent but opposite sense, the wall acts upon the flow, thus is an outside force that changes the flow's angular momentum.
Even in a free jet, the flow will interact with the outside atmosphere eventually (unless exhausting into vacuum), and the swirl will dissipate.
That said, in cylindrical swirling flows, the core region of the flow often persists a long distance downstream; I could imagine that a square channel would cause more rapid dissipation due to the disturbances formed in the corners (additional axial vorticity that gets (robs) its energy from the core flow).
Swirl number is defined by...? I assume some average swirl velocity or momentum divided by axial velocity or momentum? Again, no, the swirl number would be preserved only in the core region, away from the wall disturbance.
