I have not seen a design guide with an equation of velocity profile as a function of angle a diverging flow. SMACNA does have loss a coefficient formula in A.21 - transition round to square but this will only give you the loss. SMACNA shows a max of 45 deg as a design standard. I look at the coil selection first, then make the duct work. Most coils are going to be rectangular. I think you would be in the range of a 60"w x 48" or 80"w x 36". Your inlet velocity is low for a 36". Use 1' to 2' straight duct in front of the coil to help equalize the velocity profile. Perforated panels upstream help but add fan hp. Make sure you have 3 to 10 duct dia. up stream of the transition.