Synchronize Generator with UPS
Synchronize Generator with UPS
(OP)
I had a situation occur this weekend at our data facility. We lost utility power to the building for about 5 hours. The generator started ok and picked up the load.
Unfortunately, the UPS was not able to carry the servers during this 10-12 second ATS transfer. It was discovered both UPS battery cabinet breakers had tripped (bad batteries) as well as blown bypass fuses. The servers crashed and we lost everything.
I immediately bypassed the UPS with the MBP and got the PDU's powered. The IT group was called in and went through the process of starting this large data center back up as soon as possible. Meanwhile the UPS tech arrived, identified and replaced the failed components, and about 3 hours later was ready to get the module back on line. By the time he was ready all the servers were back online and utility power was available but left off until the UPS could handle the transfer back.
Now came the problem, we could not get the UPS to sync with the generator to allow a transfer. The frequency of the generator was up and down only fractions but not long enough to initiate a transfer. The generator is a two year old 480/277V 2MW CAT Diesel which was loaded about 50%.
It was decided to take all mechanical loads off of the generator and it helped quite a bit but still not enough time as the UPS was still leading the bypass by 7 degrees. We also had to take down the lighting in an attempt for the generator to freewheel to stabilize the frequency. A painful process.
Finally we had an ok to transfer the UPS and all went well. We restored utility power and the UPS carried the ATS transfer ok.
My question is, is there anything that can be installed to either the generator or service to prevent us from unloading the generator to this extent if this ever occurs again?
Thanks much for any help.
Unfortunately, the UPS was not able to carry the servers during this 10-12 second ATS transfer. It was discovered both UPS battery cabinet breakers had tripped (bad batteries) as well as blown bypass fuses. The servers crashed and we lost everything.
I immediately bypassed the UPS with the MBP and got the PDU's powered. The IT group was called in and went through the process of starting this large data center back up as soon as possible. Meanwhile the UPS tech arrived, identified and replaced the failed components, and about 3 hours later was ready to get the module back on line. By the time he was ready all the servers were back online and utility power was available but left off until the UPS could handle the transfer back.
Now came the problem, we could not get the UPS to sync with the generator to allow a transfer. The frequency of the generator was up and down only fractions but not long enough to initiate a transfer. The generator is a two year old 480/277V 2MW CAT Diesel which was loaded about 50%.
It was decided to take all mechanical loads off of the generator and it helped quite a bit but still not enough time as the UPS was still leading the bypass by 7 degrees. We also had to take down the lighting in an attempt for the generator to freewheel to stabilize the frequency. A painful process.
Finally we had an ok to transfer the UPS and all went well. We restored utility power and the UPS carried the ATS transfer ok.
My question is, is there anything that can be installed to either the generator or service to prevent us from unloading the generator to this extent if this ever occurs again?
Thanks much for any help.






RE: Synchronize Generator with UPS
You said that the UPSs had been repaired. If they won't carry the load for the time it takes to do an open transfer you either need better UPSs or a better repair man.
The procedure should be;
Power fails;
UPSs take up the critical loads,
Generator starts and supplies the main panel.
The UPSs sync with the generator and go on bypass.
Power is restored;
The generator breaker opens.
The UPSs take up the load.
The mains breaker closes.
The UPSs sync with the mains power and go on bypass.
I suspect that the UPSs are inappropriate for the service.
I further suspect that the UPSs took up the load, but were unable to sync with the generator so as to be able to transfer the load to the generator. Eventually the batteries were drained. (I also suspect that the batteries were drained and the bypass fuses blown as a result of a failure to transfer to the generator.)
Suggestions;
Even though the UPSs don't want to sync with the generator, they should have no technical problems maintaining the battery charge with a less than perfect frequency. I mean it's a battery charger and battery chargers are very tolerant of small frequency excursions.
If the UPSs can't tolerate small frequency excursions on the generator, don't try to put the servers on the generator. The UPSs will be going back online everytime a motor starts.
Inhibit the UPSs from transfering to generator power. Let the generator power charge the batteries and leave the UPS inverters running to supply the servers until utility power is restored.
I am surprised that dumping load stabilized the frequency. I find the frequency of sets from about 30 KW up to over 1MW is more stable with more load.
Although autogenous governor mode corrects the frequency to 60 HZ. there must be an error for the governor to respond to. That means that every time the load increases or decreases, there will be a frequency excursion that the governor will correct back to 60 Hz.
It sounds as if you have an electronic autogenous governor.
I would either have the UPS modified so that the batteries can be charged from available power regardless of frequency, or change to a set of UPSs that can handle the duty.
As for adding equipment to the generator or the service, I believe that the problem is with the UPSs they are too delicate to work and live in the real world.
respectfully
RE: Synchronize Generator with UPS
You mentioned that the generator frequency was 'up and down only fractions'. Do you know by what amount, and over what interval of time?
I don't know why reducing the load on the generator would improve the situation unless the engine speed control was having some type of problem at that particular load level. I have seen governors have problems with hunting as load approaches the limit of the engine.
Also keep in mind the voltages must match within a specified tolerance as well. Generator voltage regulators can have problems if the voltage waveform is excessively distorted. Perhaps the HVAC loads have drives that were causing waveform distortion (seems unlikely).
RE: Synchronize Generator with UPS
Keith Cress
Flamin Systems, Inc.- http://www.flaminsystems.com
RE: Synchronize Generator with UPS
How did the UPS get in to such a poor state without anyone noticing? An auxiliary switch on the battery breaker, some basic monitoring and an occasional inspection isn't much to ask.
Does you generator have utility paralleling capability? If it does then perhaps the easiest way to stabilise frequency is to synch to the utility network until you can sort out your transfer. Most big sets have the capability built in, even if it is not enabled. Of course the utility has to agree with you paralleling to their system which is usually the big hurdle.
----------------------------------
Sometimes I only open my mouth to swap feet...
RE: Synchronize Generator with UPS
waross - yes, the ATS transfer is an open transition. The entire 2500A switchboard to the facility is backed up from the generator. We have performed bi-annual load tests to this generator by simulating loss of utility power 5 times since commissioning the new genset and never had a problem with the batteries carrying the load for the 10 second open transition.
alehman & scottyUK - your senarios are correct. During these load tests we never had an issue, the batteries carried the load and once gen was on the building the inverter was back online and the UPS ran normally whether or not it was gen or utility input. The problem we had was starting the UPS module from offline with the available generator power.
I questioned this tight tolerance as well. The frequency varied between 59.8 and 60.2Hz about every 5 secs. Each time the UPS display would just toggle between "ok to transfer" and "not ready to transfer". We did not want to take the chance of transferring the rack load back to the UPS until it was steady. Unloading the generator did the trick and I can't explain why.
From what I am told the UPS and batteries are tested quarterly with the last visit in December. Shame on the IT group that handles the maintenance because the bad batteries were flagged and scheduled to be replaced next month. It goes to show how critical it is to replace ASAP I guess.
I totally agree that the UPS must have a setting to widen the variation of this slew rate from the generator. This is where I will start.
Thank you so much for the replies and education.
RE: Synchronize Generator with UPS
RE: Synchronize Generator with UPS
We have never had a problem with the inverter going to bypass while on generator power since commissioning this new generator a little of two years now. Previously a CAT 1500KW natural gas unit (now the redundant backup) also had no ill effects to the UPS when utility power had failed.
I figure we have a unique situation whereas we never had to start the module on generator power before.
We have bi-annual building load tests and run the building on generator for 7 hours each time.
Quarterly power quality spot checks at the PCC result in low harmonic distortions well within IEEE 519 guidelines, although never measured while on generator power.
Do you think it is worth attaching our power quality analyzer for a few weeks to the main service to get an overall picture?
Thanks for your help.
RE: Synchronize Generator with UPS
Keith Cress
Flamin Systems, Inc.- http://www.flaminsystems.com
RE: Synchronize Generator with UPS
Your experience with the VRLA batteries is very familiar. They have a tendency to fail suddenly without showing much sign of degredation (and usually right after the replacement warranty period expires). Usually they go to an open circuit. What kind of tests did they do on the batteries? I would recommend at least quarterly internal impedance tests on each cell. With VRLA batteries, you are better to a have several strings in parallel, so if you have an open cell you only loose one string of capacity.
RE: Synchronize Generator with UPS
RE: Synchronize Generator with UPS
I mentioned the voltage concern also, because I've seen these filters cause havoc with genset voltage regulators. When lightly loaded and the filter on, the input power factor can become leading. If there are insufficient other lagging power factor loads to absorb the VARs, the genset voltage regulator can become confused as it reduces the excitation current to zero, attempting to hold the voltage. This can also lead to false activation of loss of excitation protection.
RE: Synchronize Generator with UPS
I was on a mine construction project years ago that started on diesel power until the transmission line construction was finished. One of the generators was throwing a crank about every 6 weeks. It turned out that a time proportional controller for the electric heating in the office complex was causing the crankshaft problems. The problem was solved by jury rigging a conventional thermostat to control the heating controller until utility power was available.
The point?
A hunting governor means extra stress on the crank and bearings.
A fast cycling load will probably cause more stress on the machine.
The recommendations?
Talk to Cat about stability adjustments on the governor.
Check your loads for a fast cycling load such as time proportional electric heating. If the frequency hunting is load induced, you may be able to stretch the cycle time from 5 seconds to several minutes during the time that you are on the genset.
respectfully
RE: Synchronize Generator with UPS
I will investigate that option you mentioned and perhaps adjust the timing between cycles and also look into adjusting the floats on the holding tanks.
The IT group has scheduled complete battery replacement for this Sunday and the owners have agreed to simulate the condition we experienced last weekend in a few weeks during a maintenance window.
I do have one last question in regards to itsmoked recommendation. I agree to perform transfer tests more often than twice a year. From what I understand about these high rate VRLA batteries is that these multiple discharges will greatly reduce their life expectancy. Am I safe to assume this?
I'll keep tuned in.
Thanks much for the help.
RE: Synchronize Generator with UPS
By the way, if you have power factor correction, beware of the power factor going leading on light loads when on the generator. This will overexcite the generator and cause high voltage issues.
respectfully
RE: Synchronize Generator with UPS
One manufacturer of large UPS systems has taken the approach of sizing batteries for only a few minutes. To counter the effects of high rate, they have a periodic battery replacement program.
RE: Synchronize Generator with UPS
RE: Synchronize Generator with UPS
RE: Synchronize Generator with UPS
Your issue probably had something to do with an input filter on your UPS. Some filters will drop out under a certain load so that there aren't issues with the UPS and generator "getting along".
Mike
RE: Synchronize Generator with UPS
I don't have nearly the UPS experience the previous posters do, but the ones I've been involved with that had symptoms you described were solved by the UPS folks making slew settings wider. Some were voltage regulations issues (almost always with smaller gensets) that involved replacing voltage regulators for better quality and/or adding PMG regulator power supply.
Thanks for letting me ramble and good luck (if not already solved.)