×
INTELLIGENT WORK FORUMS
FOR ENGINEERING PROFESSIONALS

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Manhattan Partial Blackout
2

Manhattan Partial Blackout

Manhattan Partial Blackout

(OP)
Looks like they lost the west 49th st substation. And reports of a manhole fire though Con-Ed says it may not be related. Any information on your side?

RE: Manhattan Partial Blackout

(OP)
Governor promising an investigation:


https://www.youtube.com/watch?v=ySnBOkj0WQk



I agree with him and he is right when he says the system is specifically designed to prevent this. In fact the system can handle multiple failures (equipment failure + breaker failure) or failure of any two elements one after the other before load shedding or voltage reduction need to be implemented let alone a complete blackout.


There is also the fact relaying failed during Sandy at 13st st, and at Astoria 6 months ago. Several fires in the past few years at Farragut. Con Ed has some profound protective relaying issue if they are cropping up so often in such magnitude.

RE: Manhattan Partial Blackout

I'll predict there will be plenty of blame to go around. Antiquated infrastructure, lack of investment in redundancy, avoidance of disruptive construction projects, and of course, rats.

RE: Manhattan Partial Blackout

(OP)
Probably. I hope the public report that comes out is technical and no full of laymen fluff.

RE: Manhattan Partial Blackout

This is what I would bet it is. No one was providing oversight to the engineering because the utility was very short staffed or it was assumed the outside consultant would check their own work to make sure that it was always 100% correct. Problems like this accumulate and since faults are generally infrequent, it can take awhile for someone to notice that the relaying is not set correctly. Whatever the technical problem, in my opinion the causes is really a process and resource problem.

It takes a rare event to verify that your system is coordinated for a rare event. That isn't 100% true and some companies like Omicron have systems that take a system model and bombard the relay with a whole bunch of cases but I don't know of anyone that does that.

------------------------------------------------------------------------------------------
If you can't explain it to a six year old, you don't understand it yourself.

RE: Manhattan Partial Blackout

The authority rules out the following:
1) Overload
2) Ciber attack
3) Terrorism.

This site is a 40-year-old first generation of GIS with a large history of issues associated with leaking SF6 gas.
ConEd need to evaluate if this type of old installation with multiple feeders with oil-filled pipe-type cables should be adequate to maintain the reliability of the system expected for this application.

Below is an excerpt with public domain information for the 49th St Substation:

RE: Manhattan Partial Blackout

(OP)
Yup. That station has me concerned, more so than some older air insulated stations on Con Ed's territory.


Anyway, an update:


https://www.coned.com/en/about-us/media-center/new...


I hate to speculate, but could this be a system wide problem involving IEC-61850? This is really no different then the relaying failure in December. Twice primary and backup has failed in 6 months. Probability says this must be rampant.

RE: Manhattan Partial Blackout

Sounds like years of 'do more with less' cut maintenance and spend big on useless capital. These relay rooms probably flooded in the past and have never been properly recommissioned...

Just a guess

RE: Manhattan Partial Blackout

(OP)
Don't think either of these stations have flooded that I am aware of. But yes, lack of maintenance could certainly be it.


Thinking this out it looks like not only did the relaying at 65th fail, but also West 49th in that 49th didn't just cut power to 65th but also 50th and 42nd which are not tied into the 5 138kv feeders serving 46th.

In theory a primary and secondary relaying failure should simply clear a 13.8kv bus section and one transformers (if a circuit switcher is present) at 46th on over current. Worse case scenario two transformers and one 138kv cable get tripped. The remaining load can be served via 4 cables and 8 transformers no problem.

But ok, all 5 138kv feeders trip. How do the reaming 6 coming out of 49th also trip?

There is a lot that went wrong here. If any of this involves microprocessor relaying the industry needs to know.

RE: Manhattan Partial Blackout

Concerning that there was such a massive failure of the protection system. All relays are digital? Complex relaying schemes are often not adequately tested for functionality - even if the protective element detects the fault, relay programming/configuration issues results in no breaker trip signal. Pointless speculation at this stage, I suppose.

RE: Manhattan Partial Blackout

(OP)
I agree that it is somewhat pointless in that we do not have all the details on the relaying, but at the same time such a grand failure deserves a conversation. I know for a fact that if both primary and secondary 13.8kv relaying went at a 115kv-13.8kv substation around here it may (key word) clear the whole substation but not the entire 345-115kv bulk station supplying it.

RE: Manhattan Partial Blackout

Another blackout for ConEd 50K customers in Brooklyn.

RE: Manhattan Partial Blackout

(OP)
Wow! Thank you, just read up on it now. Sounds like they had to cut power to prevent further damage.

RE: Manhattan Partial Blackout

(OP)
Question.

I'm piecing together the original blackout. From all the News reports and amateur video it looks like the Times Square network never went down.

https://www.cityandstateny.com/articles/policy/ene...

Jump to 10:16 "It is a tale of two Time Squares"

https://www.youtube.com/watch?v=xOAElvuY1sI


However according to reliability reports, it appears that Time Square Network is fed via 50th st.

Anyone know if it was transferred over to another substation or has feeders from other networks? 50th also has the Hudson network, but it looks to have gone down. I'm beyond words confused.


RE: Manhattan Partial Blackout

(OP)
Interestingly, or inversely, the Empire State building was supposed to be lit but was not. Looks like Con Ed mixed the networks/supply substation around a bit.

https://www.newsweek.com/new-york-city-power-outag...

Quote:

Posts to social media showed landmarks in the city without power, including Rockefeller Center, the Empire State building and parts of Times Square.

Here is Murray Hill supplying Fashion and Empire at the time of Hurricane Sandy.



Networks supplied by Murray Hill in 2015. The number of transformers mentioned in the table are those with PQ monitoring, not the total number in the substation or network.



Murray Hill fed by Vernon at the time of Sandy.



The blue line represents 5 138kv feeders originating from Vernon supplying East 40th st and Murray Hill substation.

If wondering Queens bridge takes 138kv and steps it down to 69kv with 7 69kv feeders supplying East 63rd st.



The Vernon double ring bus, which interconnects generation to the 138kv transmission system in Queens.

38M01, 38M03, 38M04, 38M05, 38M06 feed out to 40th st and Murray Hill.

My guess for the numbering (missing 38M02) is that at one point 7 69kv cables fed out but were upgraded to 5 138kv cables.


If I have it right 38M72 is no longer "capped off" (alive on back-feed) but connected through a phase angle regulator were some power can be exchanged between West 49th 138kv and Vernon.








RE: Manhattan Partial Blackout

(OP)
FWIW, somewhere in the late 2000s 38M72 was connected to Vernon via a phase angle regulator.


Vernon substation showing the 38M72 phase angle regulator (quadrature phase shifting booster) at the top right.





A 2006 picture of Vernon without the phase angle regulator present on 38M72.

Con Edison did the right thing and replaced the maroon colored breakers with SF6 breakers.

The 3 round cylinder with radiators on some outgoing lines I believe to be iron core current limiting reactors.

However, the flood risk is hellacious with the list of essential equipment disquiet being voluminous.

RE: Manhattan Partial Blackout

(OP)
As I said, I think this is wide spread:

Quote:

“Out of an abundance of caution, we have taken preventive measures by isolating similar relay equipment at other substations,” Con Edison added. “We will analyze and test the equipment before we put it back in service. Our electrical delivery system continues to operate with multiple layers of relay protection.”


https://www.foxnews.com/us/new-york-city-blackout-...


Some relay manufacturer be it ABB or someone else has a hoard of defective equipment.

RE: Manhattan Partial Blackout

Back in the day, the PRSC used to maintain a list of know relay issues. I don't think it gets updated anymore. The mention of the interface between the "sensor" and the relay sounds like a terminal block or maybe a test switch?

I do recall seeing a demo of the GE "Brick" (61850 type merging unit), maybe it was 2008?

Utilities in my region tend to take a "wait and see" approach to newer technologies. 61850 is in that list as well as the new SEL traveling wave relay.

If this is some kind of 61850 related failure it would give all the ammunition in the world to the skeptics and ultra conservative practices seen in many utilities.

RE: Manhattan Partial Blackout

(OP)
It certainly would, and the manufacturer involved.


From another forum I learned that Con Ed disclosed what happened but sadly nothing released to the public:


https://events.vtools.ieee.org/m/198911


I think journalists really need to get involved because if people knew what I suspected there would be a lot of outrage. It is my belief that are hundreds if not a thousand relays with several latent software defects out on the Con Ed system.

RE: Manhattan Partial Blackout

A colleague commented:

Quote (If it was really the most technologically advanced in the world, the flowed connection would be of a great concern)


A good design should include primary and backup protections devices from different technologies. It is the case, why did this flaw happen?
Another factor might be associated with the O&M program and the qualification and training of the personnel. It is well knowing the downsizing and retirement of the labor forces and qualified knowledge engineers. ConEd is notorious for hire temp. and cut in training.

In my opinion, the root cause official note is vague and lock of credibility. For the high rate of energy in NYC, the customer and the general public deserve a better and honest report indicating the real root cause of the frequent power outages.

RE: Manhattan Partial Blackout

(OP)
What do they mean by "flowed" connection?


IMO I still think they are hiding a communications failure triggered by bad software in a major relay upgrade.

Just to give you an idea Con Ed has really been hitting IEC-61850 and fiber optics:


http://digitalsubstation.com/wp-content/uploads/20...


Something I do not agree with. And is just a feel good for flood prone substations.

Other countries are building plenty of brand new substations and for good reason everything is hard copper. No bricks, merging units, optical CTs, ect for the tripping and local monitoring.


RE: Manhattan Partial Blackout

There turns every day with networked solutions and you don't have to try hard to find an example of something going wrong due to miswiring.

------------------------------------------------------------------------------------------
If you can't explain it to a six year old, you don't understand it yourself.

RE: Manhattan Partial Blackout

(OP)

This is getting very interesting.


This I team investigation makes a very good case that old outdated electromechanical relays may have been the culprit:


https://www.nbcnewyork.com/news/local/I-Team--Con-...


-However-

Here they talk about "Software" leading me to think they are talking about a new microprocessor device:


https://www.nbcnewyork.com/on-air/as-seen-on/Con-E...



Me thinks its time folks file some freedom of information forms.


I know public documents have been far more transparent in the past up to including the make and settings of the equipment ie, for example the Astoria generation project relaying.



https://files.engineering.com/getfile.aspx?folder=...


https://files.engineering.com/getfile.aspx?folder=...


RE: Manhattan Partial Blackout

My experience has been that most relay manufacturers are not very forthcoming in advising customers of known issues with their devices. SEL is generally pretty good. ABB has been NOT good, at least in my limited experience. I'd put GE/Multilin somewhere in-between.

RE: Manhattan Partial Blackout

(OP)
As an example of the typical redundancy in a Con Edison area substation and network, here is how a typical 13.8kv supply substation is configured which should also hold true for 65th st.

http://engineering.nyu.edu/power/sites/engineering...


I am near 100% certain the substation in figure 1 is East 63rd st (half of it anyway).

I'd be surprised if there wasn't directional over current on the 13.8kv transformer breaker and synchronous bus breaker as a last resort backup to failed feeder relaying. I mean there really should be. The loss of a transformer and bus section at 63rd would not result in any lost load as all Manhattan networks can handle the loss of two feeders at peak load. It has to be the same for the 65th station. Even if a lost bus section at 65th st removes more then two feeders a network should still be able to survive that considering it was not a high peak demand period before the blackout.





The networks fed by East 63rd st (ignore the first two columns on the left, and make note that they do not refer to the number of transformers at the substation or in the network but rather the number of transformers with PQ monitors)




Hunter, Sutton, Turtle Bay and Roosevelt each have 12 feeders.


7 69kv feeders from Queens Bridge come into East 63rd st. Each 69kv feeder supplies two transformers or 14 in total. 7 supply 13.8kv feeder switchgear in part 1 of East 63rd st while the other 7 supply 13.8kv feeder switchgear in part two of East 63rd st.

One "part" is shown here:



Each 13.8kv main bus section has at least 4 feeder breakers, two feed one network and two feed another network.

A bus fault or feeder breaker failure results only in the clearing of 4 feeders, or two in each network.

The loss of a 69kv transmission supply feeder results in two transformers de-energizing- or one in each part of the substation.

This again is not problem as the transformer breaker senses the reverse power and opens.

In fact you could loose two 69kv feeds- 4 transformers (two in each part of the station) and not have outages.

46 MVA x 5 = 230 MVA, or 460 MVA for the whole station.

All in all the design is very redundant and over built.

Reason I am using 63rd as an example is because its not far off from most other substations in Manhattan like such as those involved in the blackout.







To give you an idea, here are the transformers at 63rd st. The colored "tubes" contain the 69kv feeds to the trafo primary.

14 arrows in total.




Another shot, the tree growing in the middle covers one of them up.

The other grates cover capacitor banks and similar equipment.





West 65th st involved in the blackout.


10 transformers, two 65MVA transformers per 138kv cable, 5 per section run in parallel.


Same operating principal.









RE: Manhattan Partial Blackout

(OP)
While piecing together the incident, I can confirm that from a hardware perspective the system is indeed designed as a second level contingency system.


Each area station has 5 or 10 65MVA transformers with no more then two sharing a 138kv supply feed.

65th St:




Astor:





West 42nd:




West 49th and West 50th st. Yellow arrows show 420MVA 345kv to 138kv auto transformers also based on a second level factor. Red arrows show 65MVA 138kv to 13.8kv units.




Stations supplied by West 49th autos:






345kv-138kv auto transformer ratings:



138kv-13.8kv transformer ratings:




If any 138-13.8kv transformer needs to be de-energized for an uncleared secondary fault only one 138kv cable is tripped. In the case of Astor it will result in a circuit switcher trip, as for the other 3 stations I am not sure but I'd say a cable trip due to their age (hence no circuit switcher).






A 138kv cable clearing will simply result in one auto transformer and 6 out of 30 138kv-13.8kv transformers being cleared. The remaining 4 autos, 138kv cables and 24 area substation transformers would remain energized and due to their 13.8kv secondaries being in parallel absolutely no network load would be lost.





The assumption above is made that the auto transformers feed the 138kv cables directly with at most a disconnect between them.

Even if the 5 auto transformers feed an elaborate 138kv ring bus at 49th st- the fault clearing topology would still remain the same and thus the outcome. Only the breakers around (for) the 138kv cable requiring de-energizing would open.

I can only conclude that:

1) In addition to the relaying failures at West 65th relay misoperations also took place at 49th st.

2) West 65 lacks directional over current or any over current for that matter on its 13.8kv transformer breakers. The uncleared 13.8kv feeder fault presented itself as an overload condition. Eventually some device caught the 5 paralleled transformers over heating and issued a trip command for all 5 clearing all the 138kv cables and the area substations they supply. But I can't see not having any 50/51 protection with secondary trip on the 65MVA units.

I'm left with more questions then answers. 3eyes ponder
















RE: Manhattan Partial Blackout

Wow - lots of great info - thanks for posting.

Dave

RE: Manhattan Partial Blackout

(OP)
Two papers that may shed more light on the event- or at least a bit on the protection system/philosophy itself.

Interestingly they do not note the 138kv-13.8kv transformers as having over current protection.

And to be clear regrading the "wave" in the 138kv cable I highly doubt thats actually a fuse.







Page 7:

https://files.engineering.com/getfile.aspx?folder=...


Paper #2:


https://files.engineering.com/getfile.aspx?folder=...

RE: Manhattan Partial Blackout

(OP)

https://legistar.council.nyc.gov/MeetingDetail.asp...=


Watching the vid presently... Now they are saying that transformer relaying maloperated at 65th st but somehow the 13kv did trip...? So if a trafo protection relay mal operated why didn't the transformer circuit switcher just trip? And how did one transformer relay maloperation result in all 5 transmission supplies tripping? And 3 systems malfunctioned? Last one I can believe.



There is absolutely no information being disclosed that answers any of the original questions. So much was said without actually saying anything.


If you want my opinion it wasn't one or two relays that malfunctioned, but at half dozen or more, at two substations. There is a lot they are hiding. A lot.

RE: Manhattan Partial Blackout

(OP)
Before I forget I just want add (stress) why this is so big- a substantial concern for an area substation triggering multiple transmission shutdowns is not just in local outages but potential system stability/thermal issues. Which is why NY-ISO is also investigating the West Side incident.

A majority of 345-138kv and 138-69kv auto transformers do not have secondary breakers and rely on their primary breakers to de-energize low side cable faults and DTT events. Because transmission cables and auto transformers alternate every other position on Con Edison's ring buses, typically between half and all transmission cables are lost when an area substation calls for the tripping of all its supply lines.


Similarly major 138kv stations alternate between lines, generators and load serving transformers. Tripping of all 5 transformers or all 5 radial area substation feeds results in at least half of all lines and gens being cleared.


Typically this puts the system beyond its normal design contingency necessitating immediate action.

Ie, clearing of all cables in a major 345kv path:




Clearing of two 345kv cables:



Wanted to make a single line of Farragut to better illustrate it but did not have time. A total area substation trip removes 5 345kv circuits on one side of the station.





Rainy:



Queens Bridge:



Flow paths that can be effected.













RE: Manhattan Partial Blackout

I looked and I see disconnects, the breakers are on the ground. I saw on your one lines from the August 2 post a maintenance scheme. The 6 transformers on the edge feed the load until the one in the middle needs to carry load to perform maintenance on one of the perimeter transformers.

RE: Manhattan Partial Blackout

(OP)
From what I've figured there are no breakers on the ground- the disconnects feed potheads transitioning to underground cables.

East 13th st is one of the few that has breakers on its outgoing area substation feeders, with Astoria West and Vernon for its generators.


A few more pics of Farragut and Queens Bridge showing the cables heading into the earth:











East 13th st is not a concern for a total transmission outage making an impact beyond directly resulting load loss:




RE: Manhattan Partial Blackout

(OP)
Con Edison had qualms about loosing an entire area substation. At first I was unsure whether or not 65th st had circuit switchers, but after reading through commission filings they should have been in place at the time of the incident.













Looking at Google maps taken in 2017 "substations beyond 2018" already had/have circuit switchers installed:









The event which started the program:

https://www.nytimes.com/1990/08/14/nyregion/con-ed...


https://www.nytimes.com/1990/08/14/nyregion/con-ed...


The result of Seaport clearing all 5 138kv feeders also de-energized the Trade Center substation and I would imagine took out 5 345kv feeders at Farragut which would certainly alarm Con Ed / NY-ISO on a hot summers day.








At the time Seaport fed the Fulton network, but Trade Center going down also removed the Park Place network, Battery Park network, World Trade Center network and the Cortlandt network. So outage was substantial.


RE: Manhattan Partial Blackout

Wow, that’s a lot of information to digest. I’m not going to read through it...
I did look on google, and while I’m not familiar with all the stations and their locations, I did see a couple and saw breakers on the ground. Some had SF6 interrupters, as indicated in your last pictures.

I’m really not familiar with what happened, but here’s a question. Some years back the north had a large power outage. It was started by a tree trimming crew, I believe. They knocked out a feeder, another breaker failed in a station, taking that station out. Stations in parallel now are asked to take on additional load instantaneously. Can’t handle it, causing 50, 51 and 81 relays to open circuits. Something like this prompt this change?

RE: Manhattan Partial Blackout

(OP)
To sum it up the West 65th street outage should not have spread (resulted in 5 138kv feeders going down) even if all 5 or 10 transformer protection relays at 65th called for trafos shutting down.

Take a screen pic of those stations, I can immediately tell you what is what.


What prompted circuit switchers being installed at Con Edison area substations was a fire at the Seaport substation in 1990. The transformers could not be shut down without de-energize all 5 138kv supply feeds. Unfortunately when those were de-energized, so was the trade center substation which also takes taps off those 5 feeders.


August 14, 2003 was a bit different. In simple terms several 345kv lines tripped out causing other lines to sag into untrimmed trees also tripping out. The remaining lines began to take on the work of the missing lines, and since more amps = more heat, those lines also sagged and tripped. Voltage in the Ohio area began to fall from all the missing equipment causing generators to shut down. The depressed pocket pulled power from its surroundings causing those to overload and further trip lines and generators. Eventually it triggered a domino effect which spread throughout the North East and Canada.





Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members! Already a Member? Login


Resources


Close Box

Join Eng-Tips® Today!

Join your peers on the Internet's largest technical engineering professional community.
It's easy to join and it's free.

Here's Why Members Love Eng-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close