Upgrading Insight Server is the easy part. Whether you are upgrading from ST1 directly to ST3 or from ST2 to ST3, you simply uninstall the old version and install the new version. Upgrading Windows Server, SQL Server and SharePoint is the difficult part.
We have just been through the process of upgrading from Windows 2003, SQL 2005 and SharePoint 2.0 (all 32 bit) to Small Business Server 2011, SQL 2008 and SharePoint 3.0 (all 64 bit). My day is too short to take you through the whole process but I can give you the following advice.
1. Seriously consider to invest in new hardware. 64 bit is the way to go and to fully utilize its capabilities you will need decent hardware. Also, if something goes wrong with the upgrade process you still have the current server to work from.
2. Get good IT support. Windows Server is a lot more user friendly than what it used to be but with a big upgrade like this there is always a little surprise or two.
3. From experience I will suggest the following steps. Setup a new server (Windows 2008 64 bit), install SQL 2008 64 bit, copy your current SharePoint content database, attached it to SQL 2008 (SQL 2008 with automatically upgrade the database), install SharePoint 3.0 and add the upgraded content database to one of your SharePoint web applications. Install Insight Server to the same web application.
Regards,
Theodore
Solid Edge ST3 MP3 on WinXP x64
Insight Server on Win2008 R2 & SQL 2008 R2 x64
SharePoint 3.0 x64