Use a process simulation tool such as HYSYS - or better yet get a process engineer to do so. The calculation will use a EOS and find the phase equilibrium at the given P&T for you composition.

well total mass flow is of course the same as that going in, but are you trying to find the mass of liquid and mass of gas at the condenser outlet?

As Morten A says, this is a complex scenario and only either by experiment and testing / sampling or use of an analytical tool such as Hysys (other software is available) will you get close to the answer.

Remember - More details = better answers
Also: If you get a response it's polite to respond to it.

if you know composition, operating temperature and pressure simply solve a isothermal flash to calculate the amount and compositions of vapor and condensate (at specified conditions, i.e. heat exchanger outlet), many tools allow to do that, for std. HC you may wish to select a EOS such as std. Peng Robinson which (usually) gives good results except for properties of liquid phase but there are extended versions, see for example PRX in Prode Properties etc. also, for additional accuracy, a complex multiparameter model such as GERG etc. could be a good alternative,
as said in previous posts there are many commercial and free simulators, but you can solve also with some free online tool or Excel pages (see for example htcprops.xls distributed with Prode), many alternatives available...

To MortenA, LittleInch, chemlite

Thanks a lot! I'll look into isothermal flashing and Prode. If it's too complicated I'll use HYSYS.

From the many sources that I read (skimmed), for some reason flashing always has a liquid feed example/diagram... never a gas feed. Can I do isothermal flashing calculations with a gas feed then?

I'm sorry that I don't understand "a gas feed"
You said that you wanted to know the liquid mass and the gas mass of the cooler outlet.
If you don't use HYSYS or similar software, you can calculate manually.
You know the components, percent of the gas. So you can calculate the partial pressure of each component according to Dalton's law of partial pressure.
You can find the saturated vapor pressure of the component in the cooler outlet temperature from link such as https://webbook.nist.gov/chemistry/
So if the component's partial pressure is higher than the saturated vapor pressure, the gas will condense to liquid until the partial pressure equal to the saturated vapor pressure. You can calculate the mass of the liquid or gas now.
You can do these one by one.

Victarion, if you know feed composition, temperature and pressure (the conditions at cooler / condesner outlet) there is only one possible "equilibrium" condition (i.e. gas, liquid or vapor + liquid , not considering non equilibrium) which you can estimate solving the so called isothermal flash operation), there could be some uncertainty due to accuracy of EOS models, errors in measurements etc. etc.
at compressor discharge (before to enter the cooler / condenser) most probably you have only a vapor phase, while after the cooler / condenser there could be (or not) condensate,
if you wish to receive more information, perhaps you can provide additional details as the composition and the operating conditions...

Thank you for the responses! Because of the complexity I decided to use HYSYS with a process coworker. It works!

as suggested by others, if you are not familiar with these calc's, probably a process engineer is the best option,
for these (simple) problems you can adopt many different tools (see previous posts) and procedures as data validation and reconciliation to compare measured values vs. simulated values...