Big blackout. What happened? 40

[Skogsgurra]

When I got words about the big outage, I immediatley went to my puter to find out what my engineering friends in the US had to say about it. But no Eng-Tips page available. Of course I can understand that. No power - no Internet.

Power was restored piece by piece and I now find Eng-Tips up and running again. My question is still valid: What happened?

Glad to see you again!

Gunnar Englund, Sweden

 

[SidiropoulosM]

The causes of the blackout will trouble many of us in the days to come. We should keep in mind that it took six days to determine the cause of the 1965 New York blackout.

Let's make this a long thread with as many comments and ideas as possible. Here are some topics that, in my view, can be used as a guide:

-Cascading. The Western Electricity Coordinating Council (WECC) has struggled with the definition of cascading outages for some time now. How would you define a cascading outage? What are the causes? How is cascading modeled?

-Protection. How are transmission lines tripped for power surges? What settings are used? Interaction of distance and out-of-step relaying?

-Restoration. Why does it take so long to restore power? How does black start work?

-Islanding. Is it possible to devise control systems that will prevent cascading by breaking up load areas into healthy islands?

-Security. With the advent of microprocessor and internet based control, are we becoming vulnerable to cyber-terrorism?

This is by no means an exhaustive list. I'm sure our friends here will add many more topics as the thread develops.

 

[busbar]

A general question — In the affected region, is it customary to use GPS-time synchronization for fault reorders, sequence-of-events monitoring and numerical relays?

 

[busbar]

It changes by the second, but a few links are

http://www.nerc.com/8-14-03-outage-announcement5.pdf

http://www.nerc.com/08-15-03-press-briefing-talking-points.doc

http://www.nerc.com/pub_doc/PreliminaryDisturbanceReport.pdf

and also

http://www.enernex.com/special/blackout.htm

 

[alehman]

See

http://www.nerc.com/BlackoutTable.pdf

for a discussion of previous major outages. The western interconnect is apparently designed to break itself into 5 islands in the event of a major disturbance. According to this report it worked during an event in 1996.

 

[jbartos]

Suggestion/Comment: It appears that the blacked out interconnect is still too big since it affected 50 Million people and caused large losses. It appears that some decentralization will be under consideration.

 

[SidiropoulosM]

Busbar, your NERC links show that major disturbances started a full two hours before the final cascading collapse. It's interesting, to say the least, that better human operator judgement did not come into play during this time to prevent the blackout.

Alehman, you are correct that the western interconnection broke into islands during the two 1996 blackout events. But is this designed into the system...and how? My own sense tells me that this is an inherent feature of the western system. Load centers are far apart and are connected to power sources and to other load centers via long lines. The ties are weaker than in the eastern system, where load density is high. Power surges will trip these weak ties creating self-sufficient islands.

 

[Skogsgurra]

Thanks for all views and answers so far. I have not had the time to digest them all, but if you grade the probability from unlikely(0), possible(1), probable(2), likely(3), certain(4), documented(5). How would you grade these causes?

A Overload of the complete system

B Local overload

C Stability problems due to line impedance

D Stability problems due to generator regulation

E Protection coordination

F Human inadequacy

G Lower "operator presence" in the physical plant

H Computers and HMI

I Other1

K Other2

L and so on

The list is far from exhaustive. SidiropoulosM has asked several questions that also should be contemplated and "baked in". Let us combine our knowledge - theoretical, practical, historical, psychologic, "was there witness" and many other - to find out what happened and how future incidents can be avoided. That would be a truly good use of the Internet.


Gunnar Englund

 

[pablo02]

skogsgurra: I think you should add a couple of others:

a) lack of transmission growth, upgrading (due to NIMBY, no financial incentives or return due to pricing or regulation or due to management's focus on the short-term bottom line)

b) deregulation and the marketing of power resulting in local generation being replaced by purchased power

c) power flowing through power lines via physics vs marketing (those islands established before no longer have the on-line generation to support the island; and overloading existing infrastructures)

d) underfrequency relaying schemes based upon older generation/transmission schemes rather than newer marketing efforts and not being updated (how many have been thoroughly reviewed and updated either from the late '70's or from Y2K? -- although the Y2K effort was very, very limited)

e) fewer engineers dedicated to relaying functions and schemes, making this another chore on remaining overworked engineering staffs (plus those who developed those schemes in the late '70's have retired or have been laid off)

f) financial constraints by either the market or upper management reducing maintenance of lines, relays, etc. from prudent schedules to ???

 

[Skogsgurra]

Thanks poblo02,

This is exactly the kind of questions that need answering. It is obvious that we europeans have a little more confidence in our energy companies than you have. Perhaps shall we learn from your incident and start looking into some of the questions you put?

NERC was born as a result of the 1965 outage. The purpose was to study the reasons for that outage in order to avoid future problems. It is clear that NERC did not do their homework. Or has limited resources. Or were simply ignored by the clerks ("accountant gnomes" as they are called here in Sweden).

That's why I think that we, in this forum, shall investigate and discuss the matter very openly. Without needing to take any pressure from governments, finance or employers.

My idea is that we shall collect points of view and questions first. And after that, discuss anything that is put forward by the investigators and see if the findings are plausible or the usual whitewash. I think that this open discussion will make those involved in the investigation more cautious and less prone to simplistic and popular explanations.

Gunnar Englund

 

[ohhn]

Pablo02:
I agree with what you added, I come from a different background so we've had some special problems, I would like to add the following comments:

1) Underfrecuency relays and load shedding: If there's not
somebody who is supervising the settings and the circuits
to be disconnected, there won't be enough load shedding
since management will always try to eliminate it or
diminish it. NERC should investigate if, at least, the load
that each company have to shed was connected to operating
relays and which were the settings; from here, it seems
that the load shedding scheme didn't work at all. We had
this problem in the Central American interconnected system.

2) Transmission System: We are a system that's begining to
be derregulated, I've made some economic calculations for
generating plants and I've read some papers on dividing the
electric industry, to be honest I'm still confused in some
aspects; but something that every expert agree while not
saying it loudly is that the transmission part of the
industry is the one with the lowest TIR, so if I were an
investor in one of the US companies, unless I have an
incentive, I wouldn't approve anybody in management who
suggested an increase in my transmission system (specially
with deregulation being done in the NY area).

So I agree with you about the incentive.

3) Islanding: There were more than 400 power plants tripped
(according to the press). I am sure that a lot of them were
steam coal power plants; as I understand it (I have not
experience with this kind of central), there is a time when
you can still connect it back to the system, if the trip
was external. If this period is ended without connecting it
back you have to let all the tuberies cool down and then
start the power plant by heating them up slowly, that's why
you need from 24 hours to 2 or 3 days to put them back in
line. So how come there was no planning for these units to
connect the load near them?...While I'm sure that, due to
the transmission grid, in some places this can not be done,
there must be places where it could be done.

Were there these kind of emergency plans? were they
implemented? or do you also have the eternal fight between
dispatchers and plant operators, where dispatchers says
that everything have to wait for them to decide???

This is only about coal steam power plants, please advice
if I'm right...

4) Islanding again: since I don't have the one line drawing
of the system, I can not decide if the lines tripping one
hour before (read the NERC preliminary disturbance report)
had an impact on the blackout, even having it I don't have
the experience on this system...

Never the less, if these trips had a big impact, NERC
should investigate why they didn't open the interconnected
system in an orderly way, since they had the time.

!WARNING¡ Please note that I'm not trying to blame anybody.
I haven't worked in the dispatch center; but there were
some times that I advised it not to disconnect load (the
one in my country) since I believed that the system, as it
was, could withstand the load increase, which is what I
believe may have happenned there...And when you do that,
sometimes you're taking a risk, since no day is equal to
another day.

 

[pablo02]

ohhn:

one of the problems with islanding and generation is the ability to isolate a generating unit either in an island of sufficient load or as an island within itself: self-generating its aux (station) power -- failure of these, it goes down dark [with plant efforts confined to protecting H2 systems, lube oil systems, hot turbine bearings, possibly water, battery supply, etc..] they then need black start-capabilities to restart, usually with a diesel-generator and or small combustion-turbine or for most, external power (I personally knew the guy who restarted NYC in '65).. (and unfortunately, everything has to work perfectly for the island concept to work) -- and then you rebuild the system by increments, hopefully bringing on hot units who are ready to start as soon as power is available..

for those plant operators not prepared for tripping off in the black or doing a black start, it is pandemonium... for the recovery to happen as quick as it did, I must give credit to some good dedicated operators...

 

[jbartos]

Suggestion: The news often mentioned that safeguarding systems did not work properly. Therefore, there was the blackout.
It is evident if there is a power plant connected to grid and it trips, the grid is supposed to be designed such a way that the loss of generation of such power plant should not trip other power plants. If this is not the case, than the design and associated safeguards are inadequate. This may be the case, if the transmissions are inadequate. E.g. one transmission has attached more generating stations and become so significant carrier, that in case of its malfunction, the grid stability is affected and causes the other plants to trip. The safeguards isolate the affected grid part from other grid parts.

 

[margaret695]

Very interesting points have been made so far.

I am not an engineer, but have worked doing electrical drawings and design... mainly for building systems for about 25 years. Did work for an electrical ultility for a couple of years in my youth, doing distribution record drawings mainly for residential areas with 12kv, 24kv and sometimes even 4kv feeders. Also did some record drawings of 110kv lines.

Any ways.... the thoughts that popped into my brain when I first heard about the power failures were... in no particular order.

How is the overload and voltage protection as well as methodology for breaking the grid into islands for the interconnected utilities coordinated? Meaning what exactly is the procedure involved... how is the paperwork and operational characteristics, timing, etc., etc. shared between the utilities... and who exactly attends the meetings and how often is it done.

I am also sort of curious if any investigation will be done in order to verify if any protection type components didn't operate as per their published data... and if the methodology of how the published data was compiled is being looked at. Also curious how the temperature of stuff in free air affects the operation. Okay.... so I have often had some issues with the published data for HID and compact fluorescent lamps.... at get the explanation that the data is based on mean numbers of stuff, etc., etc. Also am aware that the actual operation of fuses installed within a disconnect switch, or distribution board, is different then the published data for free air due to temperature considerations.

How dependant are whomever is selecting the overload and voltage protection equipment on computer programs that determine the calibration for stuff.

What is the methodology in use for verifying the computer programs that coordinate data and share data and whatever between all the utilities that are interconnected.

The points brought up about lower staffing levels, over extended staff and about how many of the engineers who designed the networks during the 70s have retired, are very very pertinent, in my opinion.

I am also curious to find out whether any of the relays, or components of load shedding systems failed due to the voltage swings, etc. It's been my experience that often the low voltage controls for some types of air conditioning equipment will start "chattering" and ummm... fry.... if a single phase situation happens. Lighting contactors that are electrically held will also do the same whatever if a phase is lost.

Another thing that I have noticed is that people with various political agendas are latching on to this power failure in order to forward their own opinions. The people who stress energy conservation.... which I do by the way... are saying that this is a wake up call for conservation. The people who want to put a lot of money in building new transmission lines and redundant type stuff are saying that this is a wake up call for investing more money into the power grids. The people who want to have more usage of alternate forms of energy, i.e., solar and wind power, etc., are putting that notion forward.

My feelings are that in the short term the existing electrical grid should be reviewed and repairs and modifications, should be done so that the existing systems will operate as designed. Also that the engineering procedures be reviewed. That energy conservation be stressed... stuff that can be implemented in a straight forward manner like replacing chillers and boilers for large and medium sized buildings, power factor correction and installing DDC controls for lighting and HVAC systems. And that more redundancy should be integrated into the existing system and at the same time alternate methods of distributing power in conjunction with the existing power grids, should be investigated, then implemented in a planned and organised manner.

Margaret

 

[SidiropoulosM]

Margaret, I couldn't understand all the points you made. But I do agree with you that many people are exploiting this blackout to advance their own political agendas. Some are already saying that we need to drill for oil in Alaska. Others are saying that we need to build more nuclear plants. And we still don't know exactly what caused the blackout. It doesn't appear, so far, to be the result of insufficient generating capacity. It could very well turn out to be human error.

Ohhn, the underfrequency load shedding is not left at the discretion of the power customer. Here in North America, utilities install the relays at their substations and the relays trip feeders that supply customer loads. Your concern on this item is unwarranted.

 

[margaret695]

SidiropoulosM: Sorry if the points that I was trying to make were unclear to you. I tend to view stuff in relation to my own experience, which is the implementation of engineering concepts. Also tend to be a paper work oriented type. Probably because it has been my experience that the sharing and coordination of information with affected parties is very very important to any engineering related activity.

I agree that we still don't know exactly what caused this blackout and probably won't for months, as all the relevant data has to be very carefully studied.

I also agree that it doesn't appear to be insufficient generating capacity and that it may very well turn out to be a combination of several human errors... or a case where not all the possible sequence of events had been fully studied before design decisions were taken.

As for the load shedding relays.... since I worked on record drawings for an electrical utility in Canada for a couple of years in the 70s, I am aware that the load shedding is done at the substations.

The point that I was trying to convey is that I am curious how the actual relays that are physically in place to do load shedding for the utilities react to an undervoltage situation... because it has been my experience that sometimes the electronic relays specified for HVAC and lighting control for buildings fail during a low voltage situation. For that reason, a lot of electrical engineers for buildings will specify mechanically held relays with 120v clearing coils in certain instances. Was just a question on my part because I am not familiar with the components used to control transmission lines.... but am used to asking lots of questions. That is how I learn stuff.

Margaret

 

[SidiropoulosM]

Margaret, here in the west we use two types of load shedding: (1) underfrequency and (2) undervoltage.

Underfrequency load shedding is by far the most common and the Western Electricity Reliability Council (WECC) has made it mandatory.

Undervoltage load shedding is not as common but can be equally effective, especially in load areas that are vulnerable to voltage collapse. I am aware of at least three undervoltage load shedding schemes in the western states. The relays are typically set to pick up at 90% voltage and will trip loads after a 2 second time delay. This should not be confused with the undervoltage protection of motors at customer sites.

 

[pennpoint]

We can say: "It happened one night in Cleveland"
What I hope comes out of this is that:
We rely too much on the computer.
We rely too much on one man alone.
That this occurance was, a fluke and could have been avoided if president Nixon had just opened up oil drilling in the Alaska Frontier (joking).
Oh Please! We have capacity to provide the required power. What is lacking is the transmission and distribution of that power. Thats what failed here.
Maybe someone was asleep at the controls, I hope they can pinpoint that and install a back-up system.
I think its the system problem rather than the the fault of one individual.
I hope we all can learn from this event.. Mainly that we are nieve about alot of issues about our power, which we take for granted.
I know it is within our capability to prevent future occurances like this. Its people like you on this form who are capable of preventing these type of occurances, I trust in you to do the right thing!

I wish you all well
pennpoint

 

[margaret695]

SidiropoulosM: Thank you for your input. I am just trying to wrap my brain around what happened. Oh and isn't there also load shedding for situations when the demand is too high for the available generating capacity? When I worked at an electrical utility during the 70s, the systems planning department assigned levels to all the 12kv and 24kv feeders... with feeders serving residential areas having the lowest priority and feeders serving essential services like hospitals, etc., having the highest priority.

And by my understanding of the graphs from the on the EnerNex Corporation site, the voltage sagged below 90% once for what appears to be almost 2 seconds and another time for between 5 and 6 seconds... with the qualification that local load reaction on the 13kv to 480v transformer may have affected the voltage measurements. So that seems to imply to me that undervoltage relays set at 90% may possibly perhaps have been implicated.

Since you have stated that undervoltage relays aren't in use in all networks... it would be interesting to find out whether they are in use in any of the grids that went down.... and of course whether undervoltage relays set at 90% are in use in any of the grids that stayed on line.

This is very interesting since the media seems to be fixating on the overload of demand whatevers with little mention of voltage... just vague comments about the grid becoming "unstable" and that people in the effected areas should upplug computers, etc., because of the possibility of voltage surges when the power is brought back on line.

Which brings me to another concern... which may, or may not be valid... there is a lot of mention of how protection for electrical grids should be computerized.... and my thoughts about that whatever... given the sensitivity of computer networks to voltage swings, noise, etc..... then in my opinion, every computer and computer network and digital type equipment involved in protecting a power grid, should have the power feeding the computers, etc., protected by voltage regulating equipment. Which ties into my perhaps naive question about whether the relays utilized in the control of power grids are electronic.... meaning digital in nature.

Margaret

 

[SidiropoulosM]

On the lighter side: will the birth rate in the affected areas show a marked increase nine months from now?