The methods for calculating nut dilation are contained in the following reference:
"Analysis and Design of Threaded Assemblies", E.M. Alexander, 1977 SAE Transactions, Paper No. 770420.
ISO 898-2 Mechanical properties of fasteners - Part 2: Nuts with specified proof load values - Coarse thread has some background information on this subject.