BillClark,
There are 3 major vibrational modes on a single: Crankshaft & conrod circular vibration, which can be balanced out by crankshaft counterweights; reciprocating piston & conrod little end vibration which requires balance shafts to eliminate, and flywheel/engine rotational vibration (rocking) as a result of engine/flywheel energy exchange during the power and compression strokes.
The only way I can think of to get rid of the last one is to run a counter-rotating jackshaft geared off the crankshaft with an identical flywheel running in the opposite direction to counteract the rotational speed variations of the flywheel. A larger single flywheel will NOT help--the energy exchange must always be there and exists between the engine and flywheel. It can only be cancelled by an identical mass running the other way. Needless to say the two gears which drive this shaft must be capable of absorbing the impact loading that this second flywheel will absorb as the engine stores and retrieves power and compression energy alternately.
The linear vibration of the piston & conrod little end can be offset by balance shafts. Perhaps if the previously mentioned counter-rotating jackshaft with second flywheel is arranged on a line at 90 degrees to the cylinder axis ("beside" the cylinder), you can use these as your balance shafts to reduce weight & complexity. The centerline between the two shafts won't jive with the cylinder axis, but can be made close enough that it will help, and is simpler/cheaper/lighter than two correctly located balance shafts plus one counter-rotating jackshaft for the second flywheel running, say, underneath the crankshaft.
Now that I think about it, perhaps you could have two properly located balance shafts, and use the one that runs opposite to the crankshaft rotation to drive the other flywheel. (Oh yeah, the gears for said multi-purpose shaft must also be rather substantial!)
Alternately, a single-cylinder opposed piston layout can be used (no cylinder head, a crankshaft at either end of the cylinder with two pistons meeting in the middle to form the combustion chamber). With the two crankshafts synchronized by a timing drivetrain to counter-rotate, each having an identical flywheel, plus identical reciprocating masses, and properly balanced crankpin/conrod rotating masses, these engines can be made to run extremely smoothly even as single cylinder units. They also feature piston operated transfer and exhaust ports at opposite ends of the lung for excellent scavenging.
Just some thoughts,
PW