We regularly transfer files from Zemax to Solidworks - and back again - without much problem. I've found that IGES seems to be the best method for what we're doing but this isn't based on any extensive investigation, simply that compared with STEP files they seemed to load quicker. We have successfully loaded individual parts and complete optical assemblies.
Where I have seen problems is with complex optical shapes where there's a lot more surface detail than would be needed in SW (e.g fresnel lenses) and as a result they takes much longer to load. I guess you could run out of memory during the transfer depending on model complexity and PC spec.