I hate to say "upgrade your PC", but that would probably help. CPU speed does not appear to be an issue. You don't say what kind of graphics card you have and what driver you are using either. Both could be issues. We also use very large assemblies with complex goemetry - another issue entirely - but our speed is good. I am running a 1.4 machine at home with XP Pro and I do not have to same problems on like size files. I'm not saying it is blinding fast, but it is not that slow. BTW: your load time is not way out of line, depending on your hardware (remember you are loading a LOT of data.)
Have you checked that:
Software Open GL is OFF. (Both in your SW Options and in your graphics driver settings). Also in the Driver settings, that the Hardware Open GL acceleration slider bar is not set to zero. (It should be set to full or the highest value that runs OK.)
Your graphics cards is appropriate for CAD work. Some work great for games but not graphics.
Your SW display performance settings are appropriate. If you are using too fine a setting it is going to be slow on large assemblies.
If your graphics shows up almost immediately and the feature tree is what is taking 3 minutes, it is probably not a big graphics issue. (Graphics/User Interface are the only multi-threaded things in SW. Has to be that way if you think about it - it's a time dependent serial solver for Parasolid feature based modelling - this is why the graphics can show up real quick compared to the feature tree. And why multiple CPU chips do not help you.)
Do you have enough system memory (and it is fast enough)?
If you are running files that large you probably need to have a Gig or more if possible. If you are running into virtual memory issues, that will kill your speed. (Check to see if the disk light is wearing its poor little heart out.) Just because the total size of your stored files is, say 500MB, does not mean that they stay that way when loaded. Plus you need room for OS, SolidWorks, etc., undo space and all manner of things.
Do you have transparency turned off? (We have not had much issue with this, but some people do - seems to depend on your hardware.)
Do you leave other applications running (like email, messaging, etc.) while you are using solidWorks?
Are you using a Gforce card with Win XP? Don't - at least with SW2003. Read the SW2003 performance thread for information on that.
I would also avoid the Fire GL2/Win XP Pro. combination. It has horrible problems particularly when connected to a network (unrelated to SolidWorks).
I know this sounds dumb, but if you are on XP, you are using Win XP Pro, not the "home edition"?
Have you defragged your disk lately? (Should not affect graphics much unless you are using virtual memory.)
That should give you a ton of stuff to check up on for now.
Good Luck.