×
INTELLIGENT WORK FORUMS
FOR ENGINEERING PROFESSIONALS

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Big blackout. What happened?
40

Big blackout. What happened?

Big blackout. What happened?

(OP)
When I got words about the big outage, I immediatley went to my puter to find out what my engineering friends in the US had to say about it. But no Eng-Tips page available. Of course I can understand that. No power - no Internet.

Power was restored piece by piece and I now find Eng-Tips up and running again. My question is still valid: What happened?

Glad to see you again!

Gunnar Englund, Sweden

RE: Big blackout. What happened?

8

The causes of the blackout will trouble many of us in the days to come. We should keep in mind that it took six days to determine the cause of the 1965 New York blackout.

Let's make this a long thread with as many comments and ideas as possible. Here are some topics that, in my view, can be used as a guide:

-Cascading. The Western Electricity Coordinating Council (WECC) has struggled with the definition of cascading outages for some time now. How would you define a cascading outage? What are the causes? How is cascading modeled?

-Protection. How are transmission lines tripped for power surges? What settings are used? Interaction of distance and out-of-step relaying?

-Restoration. Why does it take so long to restore power? How does black start work?

-Islanding. Is it possible to devise control systems that will prevent cascading by breaking up load areas into healthy islands?

-Security. With the advent of microprocessor and internet based control, are we becoming vulnerable to cyber-terrorism?

This is by no means an exhaustive list. I'm sure our friends here will add many more topics as the thread develops.

RE: Big blackout. What happened?


A general question — In the affected region, is it customary to use GPS-time synchronization for fault reorders, sequence-of-events monitoring and numerical relays?  
  

RE: Big blackout. What happened?

See http://www.nerc.com/BlackoutTable.pdf for a discussion of previous major outages. The western interconnect is apparently designed to break itself into 5 islands in the event of a major disturbance. According to this report it worked during an event in 1996.

RE: Big blackout. What happened?

3
Suggestion/Comment: It appears that the blacked out interconnect is still too big since it affected 50 Million people and caused large losses. It appears that some decentralization will be under consideration.

RE: Big blackout. What happened?


Busbar, your NERC links show that major disturbances started a full two hours before the final cascading collapse. It's interesting, to say the least, that better human operator judgement did not come into play during this time to prevent the blackout.

Alehman, you are correct that the western interconnection broke into islands during the two 1996 blackout events. But is this designed into the system...and how? My own sense tells me that this is an inherent feature of the western system. Load centers are far apart and are connected to power sources and to other load centers via long lines. The ties are weaker than in the eastern system, where load density is high. Power surges will trip these weak ties creating self-sufficient islands.

RE: Big blackout. What happened?

3
(OP)
Thanks for all views and answers so far. I have not had the time to digest them all, but if you grade the probability from unlikely(0), possible(1), probable(2), likely(3), certain(4), documented(5). How would you grade these causes?

A  Overload of the complete system

B  Local overload

C  Stability problems due to line impedance

D  Stability problems due to generator regulation

E  Protection coordination

F  Human inadequacy

G  Lower "operator presence" in the physical plant

H  Computers and HMI

I  Other1

K  Other2

L  and so on

The list is far from exhaustive. SidiropoulosM has asked several questions that also should be contemplated and "baked in". Let us combine our knowledge - theoretical, practical, historical, psychologic,  "was there witness" and many other - to find out what happened and how future incidents can be avoided. That would be a truly good use of the Internet.


Gunnar Englund

RE: Big blackout. What happened?

skogsgurra:  I think you should add a couple of others:

a) lack of transmission growth, upgrading (due to NIMBY, no financial incentives or return due to pricing or regulation or due to management's focus on the short-term bottom line)

b) deregulation and the marketing of power resulting in local generation being replaced by purchased power

c) power flowing through power lines via physics vs marketing (those islands established before no longer have the on-line generation to support the island; and overloading existing infrastructures)

d) underfrequency relaying schemes based upon older generation/transmission schemes rather than newer marketing efforts and not being updated (how many have been thoroughly reviewed and updated either from the late '70's or from Y2K? -- although the Y2K effort was very, very limited)

e) fewer engineers dedicated to relaying functions and schemes, making this another chore on remaining overworked engineering staffs  (plus those who developed those schemes in the late '70's have retired or have been laid off)

f) financial constraints by either the market or upper management reducing maintenance of lines, relays, etc. from prudent schedules to ???

RE: Big blackout. What happened?

(OP)
Thanks poblo02,

This is exactly the kind of questions that need answering. It is obvious that we europeans have a little more confidence in our energy companies than you have. Perhaps shall we learn from your incident and start looking into some of the questions you put?

NERC was born as a result of the 1965 outage. The purpose was to study the reasons for that outage in order to avoid future problems. It is clear that NERC did not do their homework. Or has limited resources. Or were simply ignored by the clerks ("accountant gnomes" as they are called here in Sweden).

That's why I think that we, in this forum, shall investigate and discuss the matter very openly. Without needing to take any pressure from governments, finance or employers.

My idea is that we shall collect points of view and questions first. And after that, discuss anything that is put forward by the investigators and see if the findings are plausible or the usual whitewash. I think that this open discussion will make those involved in the investigation more cautious and less prone to simplistic and popular explanations.

Gunnar Englund

RE: Big blackout. What happened?

Pablo02:
I agree with what you added, I come from a different background so we've had some special problems, I would like to add the following comments:

1) Underfrecuency relays and load shedding: If there's not
somebody who is supervising the settings and the circuits
to be disconnected, there won't be enough load shedding
since management will always try to eliminate it or
diminish it. NERC should investigate if, at least, the load
that each company have to shed was connected to operating
relays and which were the settings; from here, it seems
that the load shedding scheme didn't work at all. We had
this problem in the Central American interconnected system.

2) Transmission System: We are a system that's begining to
be derregulated, I've made some economic calculations for
generating plants and I've read some papers on dividing the
electric industry, to be honest I'm still confused in some
aspects; but something that every expert agree while not
saying it loudly is that the transmission part of the
industry is the one with the lowest TIR, so if I were an
investor in one of the US companies, unless I have an
incentive, I wouldn't approve anybody in management who
suggested an increase in my transmission system (specially
with deregulation being done in the NY area).

So I agree with you about the incentive.

3) Islanding: There were more than 400 power plants tripped
(according to the press). I am sure that a lot of them were
steam coal power plants; as I understand it (I have not
experience with this kind of central), there is a time when
you can still connect it back to the system, if the trip
was external. If this period is ended without connecting it
back you have to let all the tuberies cool down and then
start the power plant by heating them up slowly, that's why
you need from 24 hours to 2 or 3 days to put them back in
line. So how come there was no planning for these units to
connect the load near them?...While I'm sure that, due to
the transmission grid, in some places this can not be done,
there must be places where it could be done.

Were there these kind of emergency plans? were they
implemented? or do you also have the eternal fight between
dispatchers and plant operators, where dispatchers says
that everything have to wait for them to decide???

This is only about coal steam power plants, please advice
if I'm right...

4) Islanding again: since I don't have the one line drawing
of the system, I can not decide if the lines tripping one
hour before (read the NERC preliminary disturbance report)
had an impact on the blackout, even having it I don't have
the experience on this system...

Never the less, if these trips had a big impact, NERC
should investigate why they didn't open the interconnected
system in an orderly way, since they had the time.

!WARNING¡ Please note that I'm not trying to blame anybody.
I haven't worked in the dispatch center; but there were
some times that I advised it not to disconnect load (the
one in my country) since I believed that the system, as it
was, could withstand the load increase, which is what I
believe may have happenned there...And when you do that,
sometimes you're taking a risk, since no day is equal to
another day.

RE: Big blackout. What happened?

ohhn:  

one of the problems with islanding and generation is the ability to isolate a generating unit either in an island of sufficient load or as an island within itself: self-generating its aux (station) power -- failure of these, it goes down dark [with plant efforts confined to protecting H2 systems, lube oil systems, hot turbine bearings, possibly water, battery supply, etc..] they then need black start-capabilities to restart, usually with a diesel-generator and or small combustion-turbine or for most, external power (I personally knew the guy who restarted NYC in '65)..  (and unfortunately, everything has to work perfectly for the island concept to work) -- and then you rebuild the system by increments, hopefully bringing on hot units who are ready to start as soon as power is available..  

for those plant operators not prepared for tripping off in the black or doing a black start, it is pandemonium... for the recovery to happen as quick as it did, I must give credit to some good dedicated operators...

RE: Big blackout. What happened?

Suggestion: The news often mentioned that safeguarding systems did not work properly. Therefore, there was the blackout.
It is evident if there is a power plant connected to grid and it trips, the grid is supposed to be designed such a way that the loss of generation of such power plant should not trip other power plants. If this is not the case, than the design and associated safeguards are inadequate. This may be the case, if the transmissions are inadequate. E.g. one transmission has attached more generating stations and become so significant carrier, that in case of its malfunction, the grid stability is affected and causes the other plants to trip. The safeguards isolate the affected grid part from other grid parts.

RE: Big blackout. What happened?

2
Very interesting points have been made so far.  

I am not an engineer, but have worked doing electrical drawings and design... mainly for building systems for about 25 years.  Did work for an electrical ultility for a couple of years in my youth, doing distribution record drawings mainly for residential areas with 12kv, 24kv and sometimes even 4kv feeders.  Also did some record drawings of 110kv lines.

Any ways.... the thoughts that popped into my brain when I first heard about the power failures were... in no particular order.

How is the overload and voltage protection as well as methodology for breaking the grid into islands for the interconnected utilities coordinated?  Meaning what exactly is the procedure involved... how is the paperwork and operational characteristics, timing, etc., etc. shared between the utilities... and who exactly attends the meetings and how often is it done.

I am also sort of curious if any investigation will be done in order to verify if any protection type components didn't operate as per their published data... and if the methodology of how the published data was compiled is being looked at.  Also curious how the temperature of stuff in free air affects the operation.  Okay.... so I have often had some issues with the published data for HID and compact fluorescent lamps.... at get the explanation that the data is based on mean numbers of stuff, etc., etc.  Also am aware that the actual operation of fuses installed within a disconnect switch, or distribution board, is different then the published data for free air due to temperature considerations.

How dependant are whomever is selecting the overload and voltage protection equipment on computer programs that determine the calibration for stuff.

What is the methodology in use for verifying the computer programs that coordinate data and share data and whatever between all the utilities that are interconnected.  

The points brought up about lower staffing levels, over extended staff and about how many of the engineers who designed the networks  during the 70s have retired, are very very pertinent, in my opinion.  

I am also curious to find out whether any of the relays, or components of load shedding systems failed due to the voltage swings, etc.  It's been my experience that often the low voltage controls for some types of air conditioning equipment will start "chattering" and ummm... fry.... if a single phase situation happens.  Lighting contactors that are electrically held will also do the same whatever if a phase is lost.

Another thing that I have noticed is that people with various political agendas are latching on to this power failure in order to forward their own opinions.  The people who stress energy conservation.... which I do by the way... are saying that this is a wake up call for conservation.  The people who want to put a lot of money in building new transmission lines and redundant type stuff are saying that this is a wake up call for investing more money into the power grids.  The people who want to have more usage of alternate forms of energy, i.e., solar and wind power, etc., are putting that notion forward.  

My feelings are that in the short term the existing electrical grid should be reviewed and repairs and modifications,  should be done so that the existing systems will operate as designed.  Also that the engineering procedures be reviewed.  That energy conservation be stressed... stuff that can be implemented in a straight forward manner like replacing chillers and boilers for large and medium sized buildings, power factor correction and installing DDC controls for lighting and HVAC systems. And that more redundancy should be integrated into the existing system and at the same time alternate methods of distributing power in conjunction with the existing power grids, should be investigated, then implemented in a planned and organised manner.

Margaret



RE: Big blackout. What happened?

Margaret, I couldn't understand all the points you made. But I do agree with you that many people are exploiting this blackout to advance their own political agendas. Some are already saying that we need to drill for oil in Alaska. Others are saying that we need to build more nuclear plants. And we still don't know exactly what caused the blackout. It doesn't appear, so far, to be the result of insufficient generating capacity. It could very well turn out to be human error.

Ohhn, the underfrequency load shedding is not left at the discretion of the power customer. Here in North America, utilities install the relays at their substations and the relays trip feeders that supply customer loads. Your concern on this item is unwarranted.

RE: Big blackout. What happened?

SidiropoulosM:  Sorry if the points that I was trying to make were unclear to you.  I tend to view stuff in relation to my own experience, which is the implementation of engineering concepts.  Also tend to be a paper work oriented type.  Probably because it has been my experience that the sharing and coordination of information with affected parties is very very important to any engineering related activity.

I agree that we still don't know exactly what caused this blackout and probably won't for months, as all the relevant data has to be very carefully studied.  

I also agree that it doesn't appear to be insufficient generating capacity and that it may very well turn out to be a combination of several human errors... or a case where not all the possible sequence of events had been fully studied before design decisions were taken.

As for the load shedding relays.... since I worked on record drawings for an electrical utility in Canada for a couple of years in the 70s, I am aware that the load shedding is done at the substations.  

The point that I was trying to convey is that I am curious how the actual relays that are physically in place to do load shedding for the utilities react to an undervoltage situation... because it has been my experience that sometimes the electronic relays specified for HVAC and lighting control for buildings fail during a low voltage situation.  For that reason, a lot of electrical engineers for buildings will specify mechanically held relays with 120v clearing coils in certain instances.  Was just a question on my part because I am not familiar with the components used to control transmission lines.... but am used to asking lots of questions.  That is how I learn stuff.

Margaret

RE: Big blackout. What happened?

Margaret, here in the west we use two types of load shedding: (1) underfrequency and (2) undervoltage.

Underfrequency load shedding is by far the most common and the Western Electricity Reliability Council (WECC) has made it mandatory.

Undervoltage load shedding is not as common but can be equally effective, especially in load areas that are vulnerable to voltage collapse. I am aware of at least three undervoltage load shedding schemes in the western states. The relays are typically set to pick up at 90% voltage and will trip loads after a 2 second time delay. This should not be confused with the undervoltage protection of motors at customer sites.

RE: Big blackout. What happened?

We can say: "It happened one night in Cleveland"
What I hope comes out of this is that:
We rely too much on the computer.
We rely too much on one man alone.
That this occurance was, a fluke and could have been avoided if president Nixon had just opened up oil drilling in the Alaska Frontier (joking).
Oh Please! We have capacity to provide the required power. What is lacking is the transmission and distribution of that power. Thats what failed here.
Maybe someone was asleep at the controls, I hope they can pinpoint that and install a back-up system.
I think its the system problem rather than the the fault of one individual.
I hope we all can learn from this event.. Mainly that we are nieve about alot of issues about our power,  which we take for granted.
I know it is within our capability to prevent future occurances like this. Its people like you on this form who are capable of preventing these type of occurances, I trust in you to do the right thing!  

I wish you all well
pennpoint  

RE: Big blackout. What happened?

SidiropoulosM:  Thank you for your input.  I am just trying to wrap my brain around what happened.  Oh and isn't there also load shedding for situations when the demand is too high for the available generating capacity?  When I worked at an electrical utility during the 70s, the systems planning department assigned levels to all the 12kv and 24kv feeders... with feeders serving residential areas having the lowest priority and feeders serving essential services like hospitals, etc., having the highest priority.

And by my understanding of the graphs from the on the EnerNex Corporation site, the voltage sagged below 90% once for what appears to be almost 2 seconds and another time for between 5 and 6 seconds... with the qualification that local load reaction on the 13kv to 480v transformer may have affected the voltage measurements.  So that seems to imply to me that undervoltage relays set at 90% may possibly perhaps have been implicated.

Since you have stated that undervoltage relays aren't in use in all networks... it would be interesting to find out whether they are in use in any of the grids that went down.... and of course whether undervoltage relays set at 90% are in use in any of the grids that stayed on line.

This is very interesting since the media seems to be fixating on the overload of demand whatevers with little mention of voltage... just vague comments about the grid becoming "unstable" and that people in the effected areas should upplug computers, etc., because of the possibility of voltage surges when the power is brought back on line.

Which brings me to another concern... which may, or may not be valid... there is a lot of mention of how protection for electrical grids should be computerized.... and my thoughts about that whatever... given the sensitivity of computer networks to voltage swings, noise, etc..... then in my opinion, every computer and computer network and digital type equipment involved in protecting a power grid, should have the power feeding the computers, etc., protected by voltage regulating equipment.  Which ties into my perhaps naive question about whether the relays utilized in the control of power grids are electronic.... meaning digital in nature.  

Margaret

RE: Big blackout. What happened?

On the lighter side: will the birth rate in the affected areas show a marked increase nine months from now?

RE: Big blackout. What happened?

SidiropoulosM:  Given the imperical data on birth rates that is available vis-à-vis the black outs in 1965 and 1977...... yep... there most probably will be more babies born 9 months from in the affect areas then average.  :D

Margaret

RE: Big blackout. What happened?

Margaret, you're right. Every time the demand is greater than the available generating capacity, the system frequency declines. That's when underfrequency load shedding comes into play....load is dropped in order to re-establish a balance between load and generation. As you say, the load shedding is not done on high priority loads, such as hospitals or other critical loads.

I am also interested in finding out whether undervoltage load shedding was available but I doubt it. Every reference I have seen with respect to UVLS had to do with implementations here in the west.

RE: Big blackout. What happened?

(OP)
Having a quick look at the postings so far (it is night over here - the cat needs to be attended to) I can see that we are moving into a creative chaos and that is just the way it should be. Confusion is one of the most important steps on the road to knowledge.

Pennpoint has a point (tried to avoid that, couldn't) in pointing (oops!) out that people on this forum can - and should - contribute to knowledge that can be used to avoid similar situations in the future.

We can do that by increasing the confusion by adding more and more points of view. The confusion will then be processed into valuable knowledge (the processing has already started) after having been discussed thoroughly by all of us. We have a great advantage here. We can allow ourselves to admit that confusion reigns. That is nothing that a goverment or other authority official can do.

We have a lot of technical input so far. And some human-related things too. What about economic issues, and dollar-driven practices? Tell us about overloading and protection "adjustments" or whatever that is being done out there in order to squeeze some extra pennies out of the system.

RE: Big blackout. What happened?

Skogsgurra, you bring up an important point. Transmission expansion has fallen far behing the growth in demand and generation capacity during the last 15-20 years. We have been trying to maximize the utilization of our existing transmission assets by postponing new transmission and by installing remedial action schemes (RAS) that take certain control actions for specific contingencies, to keep the system together. This has worked well so far but we can't stretch it any further. The main problem is that regulators are reluctant to grant rate increases to utilities. And utilities are therefore reluctant to invest in new facilities. We need more transmission but someone has to pay for it.

RE: Big blackout. What happened?

Wouldn't this have been avoidable if a power system analysis and simulation tool would have been in place to simulate such a scenario?

RE: Big blackout. What happened?

After going through all the views here, I feel I should add some more. I think it will be helpful to all of our friends in Eng-Tips.
I start with the contingency condition in the system. Contingency condition is the one of the conditions of study, which allow to verify what wil be the outcome if a generating station or a line or a load is rejected from the system. In such study for small system like the one in developing countries where I live, the (n-1) [read En minus One] contingency condition is studied and implemented. This means what is the consequence if one line is out or one generating station is out etc. In more developed countries the present practice is to have (n-2) contingency condition study indicating "what will be the consequences if two units lines or generations are out". This gives very high reliability to the system. But the cost is also high and the loss due to such condition should be high enough to make such level of reliability feasible. Now the question is "is it enough for the US system to have (n-2) contingency condition?" from the perspective of the views of the consumers, political people, regulators, experts etc in short all the stakeholders of Electricity Supply System. Such study can be done by DOE or NERC or in my opinion by Eng-Tips also. In my personal view, the US system specially Eastern Grid needs (n-3) contingency condition considering the size and the expectations of the consumers. I feel that perhaps because of this situation, the system collapsed, though I can not surely say this without the proper Disturbance Record Data. I hope this will be seen in few days or week time.
Another important aspect of the transmission system in the present deregulated environment is that the lines have been used to their maximum thermal limit (possibly) due to the use of FACTS (Flexible AC Transmission Service) Devices, which help in maintaining the power transmission optimally by changing the power angle of a line. So with slight increase beyond capacity means a catastrophic to the system. This is, however, a risk taking for using a line to its maximum capacity.
The present day use of distributed generation using microturbines using gas or even fuel cell and solar could prove to be very efficient way of using system and way to better and more reliable condition, as it will unburden the existing transmission system both increasing stability margin and reducing line losses. This will also help sustain the Islanded system as each Island will have enough generation to maintain the system.
Another Important aspect of the system interconnection is to connect them asynchronously. This will help maintain the system even if some Ac lines were out.
  

RE: Big blackout. What happened?

I think you should think again about anwering these types of questions. There is a lot of people out there that can use this information for bad intentions. Loose lips sinks ships.

RE: Big blackout. What happened?

Dear all,

interesting question dictated by the present emotions

One important question I always had, and that upset lot of people generally, is as follow:

"had the proper Testing/Pre-commissioning/Commissioning activities carried out?"

Testing: is to verify ON SITE that the single equipment/apparatuses has not being damaged by transport/storage/installation, and they are still perfectly functioning.

Pre-Commissioning: is to verify that every single System within the overall plan is properly working and reacting as expected.

Commissioning: Is to verify that the Overall Plan is properly working as designed.

In my quite long work experience I verified that often, due to budget restriction and Management skills inadequacy, the Project Management Team is reducing the attention right at this extremely important and delicate stage: Commissioning.

Most of the Management is convinced that Factory tests shall not be verified on site, that relays shall be only set to the designed values and that cables shall not be tested "because they're right, I can see it...!".

This mismanagement is often supported by the employment of under-qualified personnel (this is just for "yes-sir" purposes), adding problem to problem.

A distribution system that could not stand an overload (if it was it) or a localised short circuit, lightning strike, outage of one power plant, etc, is just a system born lame and destined to give troubles. Where the Experts were during the construction and Commissioning of this overall system? Or were they been "overhauled" by politicians in needs of a quickie...?

So far, I noticed a good job from everybody in this forum, even if sometime the truth is quite painful and obvious...

At the end, some operator will be the final responsible, the ones, of course, that could be sacrified without affecting the prestige of their bosses...Welcome to the real world of hypocrisy! But the problem will still there...

Cheers to all.

Giuseppe

RE: Big blackout. What happened?

As a humble jobbing sparky much of what you are all discussing is a bit over my head, but it's interesting non-the-less.
When I was at college my lecturer recounted a tale of when he was "the man in charge" at a power station. He said that on one night another power station had shut down and demand kept rising. He and his colleague were watching the dial showing the output frequency, as the needle dipped towards the big red danger zone.
He said that they really really didn't want to have to press the "special red button" to isolate the power station. He explained that in doing so you relied on a lot of air breakers and emergency steam vents to operate or you ended up doing horrible damage to the turbines. At that point there was no-one in the country still able to fabricate a whole new shaft and they would have to get one from Japan at a frightening cost and the station would be out of action for a long time.
He explained that that put you under a lot of pressure NOT to press the button.
Fortunately as he and his colleague debated whether the needle of the dial was on the line or below it, demand started to fall and the frequency picked up.

Now, I suspect that things are a little different now (his tale must have been from 30 years ago) but you can imagine people being in a position where they really don't want to have to make the call....

"I love deadlines. I love the whooshing noise they make as they go past." Douglas Adams

RE: Big blackout. What happened?

In my opinion,the effect of this outage could have been reduced or minimised had power system operators acted promptly to initiate manual load shedding or islanding schemes.Some where some body has goofed up for sure.

RE: Big blackout. What happened?

My 2 cents worth....

After all is said and done, I believe the investigation will show a chain of events and that there will not be 1 single cause of the outage.

From an industry point of view, I believe all the attention to the system will be a positive for the future. I believe utilities will be able to get rate hikes to pay for transmission upgrades/new lines. I also believe that this will spark a public debate over deregulation. In the end, I believe deregulation will be used as a tool to spark reinvestment in the transmission system (note that the rates have to accomodate this) and regional ISO's will be the tool to regulate reliability standards.

In the end, everyone's rates will go up slightly, but that's the price you have to pay for reliable power, in my humble opinion!

RE: Big blackout. What happened?

I haven't seen it mentioned in the posts up to now, but what is the mix of generation types in the affected area? I know there are about six nuclear plants, but what are the rest?

Combustion turbines tend to react badly to dropping frequency because their compressor efficiency falls away so quickly. A drop in frequency of about 3.5% would cause the turbines at this site to dump load by disconnecting from the grid, as an example. If that takes place at a few sites, a cascade will occur unless the transmission system can shed load before the remaining generating plants dump load to protect themselves.

I don't know how a nuclear or hydro plant responds - I guess they are a lot less sensitive.

RE: Big blackout. What happened?

2
I had a rude awakening when I read the following in the IEEE PES PRSC report on Wide Area Protection and Emergency Control (talking about a WSCC disturbance on 10 August 1996)-
"The conditions leading to this incident built up over a period of 1 1/2 hours before the disturbance started.  During this preliminary period three 500 kV lines in Washington and Oregon tripped out.  Since these lines were not heavily  loaded at the time, it was not recognized that the transmission system strength was being dangerously undermined with respect to its ability to withstand another contingency."
  
The discussion goes on, with a lot of other startling similarities to the present case.  Seems that lessons weren't learned or the lesson wasn't adequately passed on.

RE: Big blackout. What happened?

Peterb, good find. It will be interesting to compare the two events once the details of the recent one are better known.

RE: Big blackout. What happened?

7
Based on what I've seen from preliminary disturbance reports (http://www.nerc.com/pub_doc/PreliminaryDisturbanceReport.pdf) and heard (from system operators), and from my knowledge of the Eastern Interconnect, my gut feeling is that there was likely a voltage collapse on the transmission grid, caused by large imports of power and triggered by the loss of critical generation and transmission facilities.

It will be interesting to find out the root cause of all of this, not to mention the fallout in industry.  I hope that FERC, DOE, Congress, and others do not step in and make rash decisions and policies as a result of this.

That being said, I've gotta get on my for a second:

Government has given industry more than enough already, with open access and the development of power markets (as good as those things are) contributing as much to this as the physical outages that triggered the event (through the pre-event state of system operation).

I firmly believe that reliability policies are best left to those in industry who know their systems best.  Contrary to what you've heard on the news, I believe that NERC is doing its job, and that utilities as a whole are doing everything they can (within the frameworks they are constrained) to prevent these types of things from occurring.

Anyway, that's my two cents on the issue.  As I said, it will be interesting to find out root causes.

RE: Big blackout. What happened?

This type of thing happens because the system is at some point constructed for n-2 or n-1 conditions, and then consumption growth causes peak loading which exceeds the limits for n-1 operation (ie, less than n, no operation). The growth is allowed to occur faster than additional necessary infrastructure can be funded and built.

Regarding islanding, I think you will find that it is illegal for operators to disconnect from the grid. They must stay connected and ride it down through automatic shedding with everyone else. Without this, each operator could indepently decide to protect his own equipment as a priority and the system would collapse quicker and MUCH more often.

Basically, infrastructure growth must keep pace with load growth. If it doesn't, this type of thing will happen and we will get a lot of band-aids until it happens again.

RE: Big blackout. What happened?

Here is my 2-cents worth. I heard today that the "System" was only running at 75 % of capacity when the outage occured.
 I heard also today that our Electric Rate is going up( I live in Ohio, now) becuase they are NOW going to invest in new equipment.
 Why havent they built a new power plant in my area in almost 20 years ? It sorta snuck out when someone mention drilling for oil in Alaska, and the response was OH we cant do that. Why NOT ?
 Why cant we build new nuclear facilities with safeguards, that they all have now, The Perry Plant in Eastlake , East of Cleveland shut itself down no problem.
 As far as I am concerned Wind power is just as the name implies, a bunch of hot air.

 Thank You for allowing me MY 2-Cents worth.
Wm.Colt

RE: Big blackout. What happened?

I'm in a large industrial on the fringes of the outage, but in it none the less.  Internally my system islanded very quickly to protect our assets (a good 15 minutes before the lights went out), but as indicated by GHz this is a detriment to the overall system but I come first.  (I'm using underfrequency loadshedding to seperate myself from the utility and control internal load on generation, and it works very well, though relatively frequently at least once a year)  The small utility we're embeded in is a "slave to the master regulator (goverment agency), and although their system could easily island, they cannot take independant action.  As a result we blacked out killing all local generation (except our own internal) and the only saving grace was a maint man in a remote station who managed to isolate one gen to station services, had it not been for that the system would not have had the ability to restart.  (No black start generation in area)

What is better - island and keep some lights on or try and ride and fall hard.  From my point of view power will come on a lot quicker from an island versus a desert.

A point of interst, I only ever hear my utility talk on N-1 contingencies, what is the practice out there?

RE: Big blackout. What happened?

(OP)
This is getting more and more interesting.

I have spent many hours reading your postings and searching the net for additional information. And I am getting confused. But, behind the smoke, I can feel that it all is about finance having had a little too much to say and not having listened to the technical guys, those that have the necessary knowledge and the necessary insight. But that is what the world looks like today. Not at all unique with the electric. We probably have to live with this state of affairs.

My thinking is that we need very realistic models if we want to communicate better with the "accounting gnomes". Models that are updated in real time and that knows about every generator, every protection setting, every tap changer, every switch, every capacitor bank, everything. Having that model we can follow the development of problems and by testing "what if" also find out when critical situations are close.

By showing the result in easy-to-understand graphics I think that even the top management can understand when a dangerous situation is developing and take/make the right actions/decisions. It is far better to do it from insight than out of necessity, like now.

In the process industry, such a model is called an "observer". It runs parallel to the real process and much knowledge can be gained from such systems. A corresponding system for electric power transmission systems will of course be a huge system and needs a lot of real time data transmission, which has to be reliable. But still, I think that it could be done.

Do you know about any systems like this? Systems that cover large distribution networks and that are 100 percent correct and truly real time?

Would such a system be of any help?


Gunnar Englund

RE: Big blackout. What happened?

One of the concerns mentioned by many was how did it cascade so fast -- overheard an ex-tech who worked on relays associated with these issues saying that maybe we've gotten too fast (using new digital/microproceesor based relays that work down to less than a cycle where as the old electro-mechanical relays took 3-6 cycles to operate..) maybe some truth to it?

We had a major blackout in the metro-Denver area long time ago when a major transmission line went down -- our station was the only one that survived and we trying like mad to keep our units from tripping -- our "island" covered about 10 blocks of downtown Denver(a Friday afternoon just before quitting time) what a nightmare...  (as I recall the coal unit went down, the gas units saved the day)

RE: Big blackout. What happened?

Gordonl, the practice here in the west is to study all N-1 contingencies and all credible N-2 contingencies. N-2 contingencies are regarded to be credible if there is a common mode of outage. For example, two lines on the same tower. Two lines on the same right of way. Two lines sharing a breaker at the substation bus, etc. My own experience is that certain non-credible contingencies have a greater probability than some credible ones. For example, two lines can go out together simply because they terminate at the same substation, because of relay misoperation. This type of outage does happen and yet it is considered non-credible. This type of dichotomy (credible vs. non-credible) will be resolved when we move toward probabilistic planning, as there is now an effort under way to move in that direction. The credibility of contingencies will be determined by the probabilities of occurrence. Even a N-3 contingency will have to be considered if it has a sufficiently high probability.

RE: Big blackout. What happened?

2
Pablo02,  I think the opposite. I would think some of the problem is that many relay schemes are electro-mechanical and not solid state.  The electro-mechanical does not allow the same selectivity as electronic types.  
With the limited settings on electro-mechanical relays, and the difficulty in delaying the trip (or locking out) of an electro-mechanical relay due to other considerations that might be available with PLC programming of an electronic relay, may have also contributed.

RE: Big blackout. What happened?

I've found by far the most common problem with protective relaying (E-M or digital) is incorrect installation or incorrect settings. However fast and accurate, numeric relays offer plenty of opportunities for configuration errors. This takes us back to guiseppe's post about commissioning (with which I agree). My experience with commissioning in the industrial world closely parallel's his. I know some utility companies have written procedures for certain types of systems, but are there any standards or recommended practices for commisioning T&D systems?

RE: Big blackout. What happened?

skogsgurra:  Hydro Quebec has real time simulators for power grids.... that they happily sell... because they like making money.  This stuff is not my area of expertise... but the various programs seem quite interesting and I have talked with people who have visited Hydro Quebec's lab and were quite interested.  Okay... so Quebec had some issues with ice a few years back.... but their simulation products may possibly be applicable.
 
Here is the link:

http://www.hydroquebec.com/transenergie-tech/en/index.html

Margaret

RE: Big blackout. What happened?

(OP)
Thanks margarete695,

Yes, these simulators are fine for simulation. But my vision is a little bit different. I think of a system that runs parallel to the real system and that gets all information (including protection settings, switch positions et cetera) in real time from the real system and where "warning conditions" and "catastroph limits" are built in.

The purpose of the system is not to simulate in order to study different configurations and load situations, but rather to give a clear picture of the stability margins in the actual system and to give an early warning if the margins for safe operation gets too narrow.

"Any simulator can do that" I can hear.
Can they, really? And, if that is so, have they been implemented on a large (i.e. 50 - 100 Mpeople) scale?

I can see two important uses for such a parallel system.
The first use is obvious; give the operators a reliable tool that shows witout mercy what the outcome of different actions will be.
The second use is to be able to communicate the stability (or fragility) of a system to the non-technical individuals that decide about investments. A map of the eastern US and Canada showing how cities lose power within seconds is more convincing than any report, written by technicians - for technicians.

RE: Big blackout. What happened?

I found it interesting that the enernex website shows a very definite over-frequency event... under-frequency schemes work nicely for under-frequency, but over-freq strikes me as being a whole other ballgame.

But it strikes me that it is just another of those unfortunate incidents that will happen sometime somewhere - chance will always find a sequence of events you couldn't ever have imagined!  Not that I'm saying we should give up trying, but we should recognise that "solving" an engineering problem just moves the problem set to something we are more comfortable with.

Bung
Life is non-linear...

RE: Big blackout. What happened?

(OP)
Bung,

The recordings that I have seen have a very crude time scale. The one I am looking at now (from enernex.com) is from 2003-08-14 09:00:00:00 and ends at 2003-08-14 21:00:00:00 with 2 hours per division. Each division is about 20 mm wide, so we have 120 minutes on 20 mm, or 6 minutes for each mm. The whole tripping probably took less than a couple of minutes so we shall not expect to see any underfrequency recordings there. The temporal resolution is not high enough.

The overfrequency that goes up to 60.25 Hz is a natural reaction to losing a lot of load. The frequency regulator takes frequency back to 60.05 Hz in about ten minutes and then, after 60 minutes back to slightly above nominal frequency.
If you look at the frequency at ten o'clock you will see that it falls rapidly below 60.0 Hz and that is a sure sign for increasing load (it could also bee that one or more generators switched off at that instance, leaving the others to pull the waggon).
Frequency stays just on, or slightly below, 60.0 Hz most of the morning, but just after noon that day, it falls to a low 59.96 Hz and stays there until before two o'clock, when it goes up rather fast, which I think is when a large part of the grid was disconnected to protect from further damages.
The frequency goes up to just below 60.0 Hz, but falls below 60.00 again due to high load. It struggles to keep up delivery, but cannot, and just after 16:00 there is a large disconnect which makes the frequency rise, giving the overfrequency event that you have noted. It is not a fault cause, but rather a consequence of the fault.

I cannot guarantee that my interpretation is correct, but I feel that (having seen lots of similar recordings from events on a smaller scale) it is logical and physically feasible.

Comments invited

Gunnar Englund

RE: Big blackout. What happened?

as  far  as  I  remember,  there  was  an  article  about  
the  affect  of  sun  on  the  earth.  Although  it  is  
rarely,  the  sun  might  increase  its  magnetic  field  
on  the  world  and  this  might  cause  phenomenal  and  
strange  trips  on  electrical  transmission  lines  and  
power  plants  too....this  is  what  i  guess....

regards  to  everyone...

RE: Big blackout. What happened?

Hi All.

I'm not electrical so I don't follow all of the above talk.

Most discussion is about searching for the root cause.

Two things that have been learnt in recent years from official investigations into complex industrial accidents:-

1. There is usually more than one cause - so don't allow anyone to skew the talk towards one, single cause. Parties who push forward a particular explanation may have vested interests.

2. There will be contributing factors - not causes, as such, but factors that existed that assisted the development of the accident

While it is natural to look for the starting point of this incident, if I was a New Yorker I would want to know why MY power failed. The answer to that lies in the LOCAL power station or distribution station, not in Ohio. Why did New York's power fail? That must be a relatively easy question to answer, and perhap more useful.

Cheers,
John.

RE: Big blackout. What happened?

fundam --

LOL!  That's the catch-all you usually hear from the relaying techs when they can't track down the root cause of a relaying event.  "Damn sunspots..."  I have a hard time believing this had anything to do with the outage, especially given that NERC monitors solar magnetic disturbances and no warnings were issued prior to the event.

all --

I think it's safe to say that yes, there were probably some relaying problems that contributed to this event.  However, I don't think you can blame the whole thing on that alone.  As I alluded to in my last post, I believe that a large root cause of this is simply the way in which power systems are operated today given the framework in which industry is given to work.  It's common for there to be thousands of MW of transfers occurring on the system simultaneously -- occurring on a system that was designed to transmit local generation to local load.  Over-reliance on external generation resources logically leads to problems when transmission leading to those sources trips.

I know I'm oversimplifying things, but you can't separate the operational issues from the technical problems.  When you operate a power system in an operationally insecure manner, you're asking for trouble.  The big question about this whole event is this:  were system operators and reliability coordinators aware of the potential for a cascading outage, and were they doing everything possible to return the system to a state in which this was not a threat?  My bet is that they were not fully aware of the implications of the next outage, simply due to the fact that there was a lot going on, but that they were doing everything they could to return the outaged facilities to service prior to the cascading event.  Why do I think they weren't aware of the implications of the next outage in real-time?  Anyone with experience with EMS/SCADA systems knows that it's a very difficult task (not to mention very expensive) to get a state estimator up and running, let alone running quickly enough to provide accurate information about stability issues in a timely manner.  Yes, we can analyze these issues from a planning perspective using established models rather quickly (you can get good information, but it won't be 100% accurate), but to take real-time information and use it to analyze operations is a much more difficult task.  I don't know the stats on the issue, but I'm not aware of many companies that have state estimators running that can provide this information.

Let me emphasize -- I don't believe this to be an event of negligence.  I do believe that this event was a result of a number of improbable events, some operational, and some technical.

Again, just my opinions.

RE: Big blackout. What happened?

(OP)
jstickley,

Yes it is a much more difficult task to run an observer (or state estimator) than to run a simplified model. I guess hundred or thousand times more complex than the model.

But, if you can learn from such a tool. And really demonstrate to the guys with the money that things have to be changed before the next event occurs. Perhaps even use the observer as a fast decision-maker that takes the correct actions quicker than the proverbial pig can wink his eye? Wouldn't that be worth some effort?

Building such a system will be expensive, yes. But probably not more expensive than having the nation more or less dead for days. And certainly not even one percent of the cost of the total generation/distribution system.

Of course, there will be practical and technical problems. One such problem is that electro-mechanical protections usually can't be read by computers. But the solution is easy; have them read and reported by humans. After all, you do not change those settings very often.

The more I think about such a system, the more I like it. What do you say?

Gunnar Englund

RE: Big blackout. What happened?

Here is a detailed timeline of the blackout:

- 2 p.m. FirstEnergy's Eastlake Unit 5, a 680-megawatt coal
generation plant in Eastlake, Ohio, trips off. A giant puff
of ash from the plant rains down on neighbors. On a hot
summer afternoon, "that wasn't a unique event in and of
itself," says Ralph DiNicola, spokesman for Akron, Ohio-
based FirstEnergy. "We had some transmission lines out of
service and the Eastlake system tripped out of service, but
we didn't have any outages related to those events."

- 3:06 p.m. FirstEnergy's Chamberlin-Harding power
transmission line, a 345-kilovolt power line in
northeastern Ohio, trips. The company hasn't reported a
cause, but the outage put extra strain on FirstEnergy's
Hanna-Juniper line, the next to go dark.

- 3:32 p.m. Extra power coursing through FirstEnergy's
Hanna-Juniper 345-kilovolt line heats the wires, causing
them to sag into a tree and trip.

- 3:41 p.m. An overload on First Energy's Star-South Canton
345-kilovolt line trips a breaker at the Star switching
station, where FirstEnergy's grid interconnects with a
neighboring grid owned by the American Electric Power Co.
AEP's Star station is also in northeastern Ohio.

- 3:46 p.m. AEP's 345-kilovolt Tidd-Canton Control
transmission line also trips where it interconnects with
FirstEnergy's grid, at AEP's connection station in Canton,
Ohio.

- 4:06 p.m. FirstEnergy's Sammis-Star 345-kilovolt line,
also in northeast Ohio, trips, then reconnects.

- 4:08 p.m. Utilities in Canada and the eastern United
States see wild power swings. "It was a hopscotch event,
not a big cascading domino effect," says Sean O'Leary,
chief executive of Genscape, a company that monitors
electric transmissions.

- 4:09 p.m. The already lowered voltage coursing to
customers of Cleveland Public Power, inside the city of
Cleveland, plummets to zero. "It was like taking a light
switch and turning it off," says Jim Majer, commissioner of
Cleveland Public Power. "It was like a heart attack. It
went straight down from 300 megawatts to zero."

- 4:10 p.m. The Campbell No. 3 coal-fired power plant near
Grand Haven, Mich., trips off.

- 4:10 p.m. A 345-kilovolt line known as Hampton-Thetford,
in Michigan, trips.

- 4:10 p.m. A 345-kilovolt line known as Oneida-Majestic,
also in Michigan, trips.

- 4:11 p.m. Orion Avon Lake Unit 9, a coal-fired power
plant in Avon Lake, Ohio, trips.

- 4:11 p.m. A transmission line running along the Lake Erie
shore to the Davis-Besse nuclear plant near Toledo, Ohio,
trips.

- 4:11 p.m. A transmission line in northwest Ohio
connecting Midway, Lemoyne and Foster substations trips.

- 4:11 p.m. The Perry Unit 1 nuclear reactor in Perry,
Ohio, shuts down automatically after losing power.

- 4:11 p.m. The FitzPatrick nuclear reactor in Oswego,
N.Y., shuts down automatically after losing power.

- 4:12 p.m. The Bruce Nuclear station in Ontario, Canada,
shuts down automatically after losing power.

- 4:12 p.m. Rochester Gas & Electric's Ginna nuclear plant
near Rochester, N.Y., shuts down automatically after losing
power.

- 4:12 p.m. Nine Mile Point nuclear reactor near Oswego,
N.Y., shuts down automatically after losing power.

- 4:15 p.m. FirstEnergy's Sammis-Star 345-kilovolt line, in
northeast Ohio, trips and reconnects a second time.

- 4:16 p.m. Oyster Creek nuclear plant in Forked River,
N.J., shuts down automatically because of power
fluctuations on the grid.

- 4:17 p.m. The Enrico Fermi Nuclear plant near Detroit
shuts down automatically after losing power.

- 4:17-4:21 p.m. Numerous power transmission lines in
Michigan trip.

- 4:25 p.m. Indian Point nuclear power plants 2 and 3 in
Buchanan, N.Y., shut down automatically after losing power.

RE: Big blackout. What happened?

Lets see, 1)loss of a generation unit, 2)lines overloaded and tripped, 3)and out-of-step conditions that lead to a snowball effect.
Very typical for a blackout.

The last one that happened in my country, was a bit different: Earth fault in a major EHV line, islanding (one big island with load/generation= aprox. 3 and unsuficient outage), and loss of synchronism.
An announcement said that a stork (the bird usually nest on the top of the EHV towers) was responsible for the blackout, but the islanding operation were based on a study with 20 years and the outages were not appropriated.

Who was responsible after all? The (poor) bird...

Best regards,
Morcon

RE: Big blackout. What happened?

(OP)
Beautyful, SidiropoulusM!

And now, the question? Was the system close to overload? Was it an n-2, n-1 or just plain n situation? I think the latter.

RE: Big blackout. What happened?

skogsgurra --

I'm in favor of state estimators -- I wish every utility had one.  They are great tools to provide information to system operators.  However, do you have any experience with them?  The information they provide takes TIME to obtain -- the state estimators I've seen take several minutes to perform complete single contingency analysis for even medium sized systems.

I can't see how what you're talking about is even technically feasible given currently available technology and software -- you're talking about a state estimator that not only gives power flows, but also a system that performs complete stability analysis, fault analysis and predictive relay tripping analysis in real-time.  I would imagine that even if you could put that kind of simulator together, it would take it well over an hour to produce any kind of reasonable results, by which time the information is outdated anyway.  That's not even addressing the extreme difficulty in developing such a system and keeping it up-to-date.

Don't get me wrong, what you're talking about is a fine idea.  However, it's not just a matter of time and money getting such a thing implemented.  I don't want the uninformed observer of this thread to think "man, why aren't utilities doing these things?" without the understanding that they're quite difficult (if not impossible) to implement, often with an uncertain level of usefulness.

The bottom line to me is this:  no matter how extensively you monitor your system, the system will fail if you have enough things go wrong.  Our job as utility engineers is to be sure that this risk is reasonably mitigated.  The question is, what's reasonable?  Can you really plan your system with enough redundancy to withstand 3, 4, or more contingencies?  Can you have a plan to mitigate EVERY possible event?

RE: Big blackout. What happened?

Analyzed/report by EPRI PEAC monitoring team:
Please keep in mind that there is no conclusive explaination of this 2003 power outage yet at this point.  However, here is what was reported by EPRI PEAC monitoring team:
"The waveforms indicate a phenomena that we have called “fast voltage collapse”.
This can happen during periods of heavy load, especially when there is a dominance of motor load (e.g. air conditioning and industrial load).
Recovery from the voltage sag during these conditions can be very slow.  The motors draw increased current due to the continuing low voltage and the voltage around the system collapses due to this increased demand following the sag.
Generators struggle with this increased load.
Motors will eventually stall and trip.  Load goes off and voltage can go very high as a result.
Within 3 minutes more than 20 generators in NE trip.  This is probably caused by frequency variations from the generation/load mismatch."

RE: Big blackout. What happened?


In an earlier post above I expressed my surprise that human intervention did not come into play for two hours before the collapse, while lines and generators were tripping all over the place. There's a report now that there was a breakdown in communications between system operators.

Visit: http://www.foxnews.com/story/0,2933,95095,00.html

Moral #1: no need for high-tech systems if our low tech fails us.

Moral #2: need for automatic wide area controls to reduce reliance on the human factor.

RE: Big blackout. What happened?

(OP)
jstickley,

I disagree that such a system couldn't be built. I guess that you need computing power perhaps ten times what is used for detailed weather forecasts. It will cost, yes. But it can surely be done.

I also think (I admit that I do not know) the amount of input data will be comparable, perhaps less since much data is static (rated capacity, protection settings, power line and transformer data and many more do not change dynamically).

As I said before, there are at least two uses for such a system; (1) guidance for the operators and (2) a way of showing the guys with the money what the reality looks like. There might also be a third use for such a system; to influence the operation of the grid to avoid walking to close to the edge (bad picture, I hope that you understand).

As you can see in previous postings, there were a couple of hours when the operators could have saved the situation and I think that a observer could have done something to that. The "big dip" that took place in one or two minutes happened when the system was doomed. Nothing could help then.

But, I do agree that a PC based system or even an ordinary mainframe computer can't do this. I also agree that many (un)realistic proposals often turn into unrealistic implementations, but my more than thirty years in steel and paper mill automation has taught me that some observers do a great job.

RE: Big blackout. What happened?

skogsgurra --

Do you have any idea the enormity of the system you're proposing?  You're looking at needing millions of data points in real-time if you want to monitor every breaker, switch, and line flow in the Eastern Interconnect.  I stand by my original viewpoint -- this is probably impossible at this point.  Maybe this is not the case for the system you live with in Sweden.

All --

I really think that any talk about "if operators would have just done this, this wouldn't have happened" or "if we had this system to monitor the grid, this wouldn't have happened" is foolishness.  Has anyone here besides myself been involved with system operations during a crisis?  Operators get inundated with information, especially when there are a number of problems occurring simultaneously.  EMS/SCADA systems end up constantly alarming.  Scheduling systems have changes that must be implemented.  The phone  rings off the hook.  Supervisors ask questions.  It can be a very chaotic situation, given the demands placed on operators in today's industry.

I guess what I'm saying is this, if you must place blame, place it on the adequacy (or inadequacy) of the system design and the way in which the industry expects it to be operated.  It's safe to say that the two are quite different -- the system was designed to deliver local generation to local load, but we now operate it to transfer thousands of MW of electricity.  To expect it to perform adequately under such high stresses while experiencing multiple outages is unrealistic.

RE: Big blackout. What happened?

Anyone care to speculate on the possibility that all this has something to do with automatic versus manual control of phase angle regulators scattered throughout the system?

RE: Big blackout. What happened?

37pw56gy, if the phase shifter automatic controls are not disabled following a major contingency, it is possible that the phase shifter controls could aggravate the situation.

I'm not betting the farm on this one yet, but you've made a good point.

RE: Big blackout. What happened?

Due to the importance of this thread I have been printing it as experts are expressing their point of view day after day. But today I found this page quite widely expanded so I cannot print it on a letter size paper. Please tell me some of you if it is possible to narrow it to a letter size width and how can I do this!
Thanks a lot.

RE: Big blackout. What happened?

Piterpol, at the top of this page you'll find a little window called "e-mail it". Click on it and send the thread to your own email. You can then print the email and it will print fine.

RE: Big blackout. What happened?

Suggestion: Normal generator protection devices, which can trip on the grid unusual conditions, are:
1. Undervoltage. This can cause the Generator Automatic Voltage Regulator (AVR) to be out of its voltage regulating region
2. Over/under Frequency Protection Relay. This relay protects the generator from damage if the relay setpoint, upper or lower, is passed.
3. Some generators can trip on "no load", i.e. sudden loss of 100% load. Others can withstand it. It just depends on the genset design.
4. Periodically Cycling Load. That implies some oscillations in the grid, which can potentially cause electromechanical resonance and subsequent generator trip due to oscillations.
5. Etc.

RE: Big blackout. What happened?


Jbartos, great info. Since most of the generators tripped were nuclear, can you point out which of these protections apply to nuclear generators?

Also: are there any other protections that are specific to nuclear generators?

RE: Big blackout. What happened?

Suggestion to the previous posting. The nuclear power plant generators are relatively large in MVA or MW, e.g. over 300MW and up to about 1200 MW. They are usually sensitively protected since they are expensive. Therefore, they trip first. Thanks goodness that they have tripped. Now, we have power again from undamaged gensets. Incidentally, some nuclear power plants offered tours to the public in the past. One might have seen the nuclear power plant turbo generator very closely, e.g. on the roof of the turbine building (Salem, NJ).

RE: Big blackout. What happened?

skogsgurra:  I can envision that the system that you are thinking about would be a very useful tool.  But the thought of taking all the as-built drawings for the existing transmission lines, as well as all the revisions and modification, length of lines, types of cables, protection devices, doing surveys, etc., etc, etc, and inputting said data accurately into a program sort of boggles my mind.

But I am wondering if the notion of having it part of the specifications for new transmission lines, that the contractor building said transmission lines... or modifying said transmission lines... would have to submit as part of his contract accurate data base lists... prepared in a strictly regulated manner.... tied to autocad drawings.... of all the material and equipment used.  That way the data could be compiled slowly... and hopefully accurately... over a number of years.  Okay... maybe decades.  

Any ways... if the data was compiled by those actually working on the construction and maintenance of the transmission lines... that would decrease the labour costs for the data input of the system that you are proposing.  Especially for new transmission lines, as the lists for cost estimates possibly could be tweaked into an appropriate format.

I do agree that methodologies for presenting technical information to non technical bean counter types is very very important.  

Maybe some one could write a movie script.  Hey.... a lot of non technical types got the safety related message behind "the China Syndrome."

Or possibly we could get the video game people involved.

Please note that I am only partially joking.  :)

Margaret

RE: Big blackout. What happened?

fundam:  I will admit that I checked out the records for sun spot and solar wind activity a couple of days back.  :D

Didn't look like anything unusual was happening on the 14th.

Also, by my understanding, solar activity is more likely to affect transmission lines in the far north... like the ones from James Bay, etc.... and I haven't heard of any problems on those lines.... and my lights didn't go out.  Also didn't hear of any problems with the communication satellites, which seem to be more sensitive to sun spot activity then transmission lines.

Margaret

RE: Big blackout. What happened?

have been sitting here pondering the subjects of power factor and harmonics, etc.  

The types of loads in buildings has really changed over the last few decades.  We got lots and lots of computers.  Offices used to run about 2w/sqft and now they often run at about 4w/sqft, or more.... because of all the computers... with their switching power supplies.  Then there is all those electronic ballasts out there for fluorescent lights and HID lights, along with the usual induction type loads like motors.  And motors have variable speed drives often nowadays.  And electric heating is often controlled by triacs, etc., etc., etc.

Any ways.... where I am located, the electrical utility bills on either the demand, or 90% of the kva, so as long as the power factor is above 90%, whomever is paying the utility bills does care... and even if the power factor is running just under 90%, my clients will ask for the pay back period of installing capacitor banks before actually installing capacity banks.

Makes me wonder if the power factor for the over all grid perhaps has creeped upwards during the last decade.  Also makes me wonder about the levels of harmonics.  As from what I have seen, the voltage data available for this event is rms.  

Hmmmmmmm...... of course I could be way way off base.... but does make me wonder.

Margaret

RE: Big blackout. What happened?

Okay... so I am typing impaired....

Meant to say .... so as long as the power factor is above 90%, whomever is paying the utility bills does NOT care.

Margaret

RE: Big blackout. What happened?

Going to sleep now... meant to say that the power factor could be creeping downwards.  :D

Margaret

RE: Big blackout. What happened?

There are many other layers of generator protection that can be applied at the generator terminals.  These include Overvoltage, Phase current/voltage imbalance, Phase Reversal and High Negative Sequence Current.  In addition, there are control systems (fuel, turbine and generator) that are less obvious but in this scenario may prove to be equally important to understanding what happened.  

The generator has an Automatic Voltage Regulator whose primary function is to control the field current so that the terminal voltage is held to a relatively constant value, but there are usually other functions that it provides including Line Drop Compensation, Reactive (VAR) current control/limiting, Field Over/undercurrent and System stability control.
 
Note that all of these are control functions that operate relatively independently of the protective relaying functions even though they all use the same control variables (current and voltage) for inputs.  

It is also important to consider the Turbine Control System (Governor)which manages the raw mechanical power, speed control (normal, over, under) and stability of the turbine. It is the interaction of these diverse subsystems that have an integrated impact on the overall functioning of the entire grid.

RE: Big blackout. What happened?

piterpol --

A bit of advice -- take the "words of wisdom" you find on this site with much caution.  This site, as wonderful and generally informative as it is, has no screening process to ensure the "experts" who post here really know what they're talking about.

In this thread alone I've seen a lot of ideas that are pretty far out there in terms of the reality of day-to-day power systems operation.

all --

It was stated that that most of the generators that tripped in this event were nuclear.  This is not the case.  Hundreds of units tripped -- the problem was not specific as to the type of plant it affected.

Also, note that often the reason generators trip is not due to the generator relaying designed to protect against faults, but rather interal plant loads that are necessary to keep the plant running.  We've had several instances of voltage fluctuations on the grid causing plant auxiliary loads to trip (variable speed drives in particular seemed to be particularly sensitive).

RE: Big blackout. What happened?


Jstickley, if someone said that "most generators that tripped were nuclear" he probably meant "many" instead of "most". There are many contributors here for whom English is not their first language. For some of them the words "many" and "most" may be interchangeable. In any event, it's not a big deal.

What I find amazing is that most contributors here will show a great deal of tolerance and will go out of their way to understand a question, even if written in bad English. I think this attitude enriches the forum and this is the type of attitude we want.

RE: Big blackout. What happened?

SidiropoulosM --

Don't take this personally, but clarity IS important.  As I noted in a previous post, many of those here reading this thread are not informed about the industry, and my clarification was intended to prevent confusion amongst those readers rather than to nitpick those whose first language may not be English.

To answer your question regarding generator protection, I would strongly suspect that the relaying schemes that protect nuclear generators are similar to those that protect other generation sources.  I can't imagine that the source makes much difference in the type of relaying required to protect the generator from electrical faults (jbartos's point is excellent, but I think it rings true with all types of units -- larger units will likely have more in-depth protection schemes).

The difference I see between varying sources of electricity (nuclear, coal, combined cycle natural gas, simple cycle natural gas, wind, solar, etc.) that could affect tripping characteristics is in the plant itself -- the motors, pumps, and equipment that make up the plant and the varying processes by which the sources convert prime movers to electricity.  Due to differences in the equipment used, plant tripping characteristics likely vary among different plants.  I would suspect that nuclear plants have many redundant safety systems, and these systems may be more or less tolerant of system dynamics (I can't say which, as I don't have direct experience with any nuclear plants).

RE: Big blackout. What happened?

(OP)
jstickley,

I will make myself more clear. The NERC has been active for almost forty years. Their main task is to find means to avoid big outages like the one we are talking about.

In my opinion, they have not been sucessful. The two hours before the total collapse could have been used in a much more efficient way. Even a modest observer - a lot simpler than the one I envisage - would probably have helped.

What I want to say is not that a complex system should be built right away, but that one should consider and think about such a system during the next ten or twenty years. And, frankly, I cannot understand what you have against that.

This is my last posting with regards to the observer/estimator. There are a lot of other things to discuss. Let's do just that!

Gunnar Englund

RE: Big blackout. What happened?

You guys are really getting me going today....

skogsgurra --

First off, I have no problem whatsoever with your idea -- it's just that on the scale mentioned, implied, or maybe just what I interpreted, it's not feasible.  Where did you get the idea I'm against such a thing?  I've stated that I believe that every utility should have a state estimator that can perform security analysis.

Regarding NERC, are you blaming them for the outage?  Last time I checked, NERC does not have system operators sitting there monitoring the system.  NERC's purpose is to ensure the reliability of the system, true, but it does so through establishing reliability standards, monitoring compliance with those standards, facilitating education of industry personnel, coordinating assessments of the interconnected system, etc.  NERC does not have direct authority over any individual utility regarding day-to-day system operations.  Here's a link with information about NERC:  http://www.nerc.com/about/.

I believe that as more information becomes available, you'll find two things -- the system was planned according to industry-accepted reliability criteria, and that system operators did everything they could to prevent this outage from happening.

If the system is found to be planned inadequately, that's one thing.  However, if it's planned according to established and accepted criteria, you've got to answer the following:  is our current criteria adequate, and if not, what is?

If the cascade is found to be due to the outages that occurred from 14:06 to 15:06 CST (this if from NERC's preliminary report, http://www.nerc.com/pub_doc/PreliminaryDisturbanceReport.pdf), I believe that investigations will show that operators were doing everything they could to get those lines back in service to prevent further problems.  The fact is that when outages occur, often times it takes time to get them back in service.  Even if the line is physically intact, relays lock out, and sometimes have to be manually reset by sending people to the stations.  If it turns out there was something that could have been done aside from putting the lines back in service, don't forget that hindsight is always 20/20.  I guess the tough question regarding system operation is this:  were operators taking whatever actions were reasonable and prudent to prevent this from occuring?  My gut feeling is that this is the case.

Anyway, I've rambled on enough.  I believe the bottom line in this whole event to be this:  STUFF HAPPENS (I know, the phrase is different, but profanity is not acceptable here).  Sometimes, in spite of all things, we just can't prevent problems from happening.

RE: Big blackout. What happened?

I have been reading all the comments in this forum. I
 find it extremely unbelievable that when the same 345kv line tripped twice , it was reclosed immediately without analysing the cause of the fault.
Having two hours after the first trip "with good communication and instructions from a regulating command this blackout could have been avoided.
Islanding should have worked.

RE: Big blackout. What happened?

I toil for the building services sector and I have never worked for any utility. But back in the 80's we were doing studies on the steady stability of power systems and we were able to look at the '65 and '77 outages as voltage collapse in the steady state sense.

Most of the discussions above have so far focused on the dynamic state of the power failure as a result of frequency, voltage relaying etc failures.

If I remember correctly, we were able to show in our study that dynamic collapse occurs after the steady state stability is compromised. The compromise by overburdening of the system can be in the form of loading the system over its network capacity, generation capacity, network component failure among others.

Even FirstEnergy refuses to be blamed for the power failure stating that voltage fluctuations in the western part of the grid were noted on Thursday morning. Large swings in power flow in the 1000MW magnitude were also noted in some transmission lines.

My question to the power utility sector or NERC people is whether steady state stability is being used to measure the reliability of systems. One of the tools that can be used is to assess the singularity of the Jacobian without going through the cumbersome state estimators/ load flow analysis. Could it be that the NE grid is overburdened that there are actually two solutions to the power flow which are 1000s of MW apart? And a little perturbation will cause the system to settle from one solution to the other solution.

Just my guess.

RE: Big blackout. What happened?

I can only relay a story I heard from a nephew of mine, who, as a one of a team of firemen, was called upon to rescue people trapped in an elevator.  They had to burrow thru a wall to gain access.  Upon entry into the darkened elevator, and because of his chivalrous concern for the women inside, he asked, "Any pregnant ladies in here?"  A man's voice from inside replied, "Hell, we haven't even become acquainted yet!"

RE: Big blackout. What happened?

Lukedric, the Western Electricity Coordinating Council (WECC) mandates reactive margin studies, in addition to power flow and stability. The objective of reactive margin studies is to determine how far the system is from voltage collapse. This is done by computing the Q-V curve for critical busses and for all critical contingencies. The Q-V curve is a plot of reactive power injected into a bus vs. the resulting bus voltage. It has a U-shape and the bottom of the U is typically below the x-axis. The two points of intersection with the x-axis are the two possible solutions of the system. If there is reactive deficiency at a certain bus, the two points will be close together. In such a system, the operating point can move from one point to the other, for small changes of operating voltage. This is voltage instability which may lead to system collapse.

The reactive margin method has been advocated by the WECC following the two 1996 western blackouts and, to my knowledge, it has not been used in the eastern system yet.

RE: Big blackout. What happened?

Way back was mentioned "illegal to disconnect from grid to salvage a local island".  Makes sense at the micro level, e.g. individual generators, but ridiculous at the regional level.  There surely should have been automatic isolation at the interties from Ontario to Michigan, and New York to Ontario.  If Michigan goes black surely Ontario and New York should be capable / allowed to "island" themselves?  My quick glance at a line diagram indicated what I guessed to be thyristor load managers at all major points except Niagra.  Shouldn't that be the first (immediate) step?  Place thyristor connections at Niagra, then REQUIRE systems to AUTOMATICALLY (under a set of NERC-determined conditions) back these down first before exporting one area's problems into another area?

At least might stop NY Mayor from going on CNN and doing another "blame it on Canada" again.  Too bad, 50 million CNN viewers get that information, then 50 people follow a thread like this to find out a bit more.

RE: Big blackout. What happened?

The problem with islanding is that the island load must be sufficient and STABLE enough for a generating unit to remain on-line regardless of fuel type -- as for nuclear units, in addition to the load, there are additional (NRC) requirements for operating the units -- which if not met, they must shutdown... I have elected not to go into these requirements, but I suspect they were also a factor..

RE: Big blackout. What happened?

Pablo02, if islanding is designed into the system, it must be combined with automatic load shedding. A computer-based special control system (SPS)will determine the load that must be shed to match the available local generation, resulting in a self-sufficient and stable island. Load areas that lack local generation will have to be blacked out until the system recovers. This should motivate utilities to add local generation at load centers, reducing their reliance on incoming transmission lines.

RE: Big blackout. What happened?


SidiropoulosM, I think it is unfortunate that an authority will only mandate reactive power margins after the collapse of the power system. Let us hope that NERC and N-E council(?) will follow and include reactive power margins to designate their system as stable.

My utility, which also shutdown last week, routinely depresses its system voltage in times of high demand. I think this is one of the optimization techniques used by the deregulation experts. This measure eats up the reactive power margins of the network with less capacitive power generated and more reactive power taken by the transmission lines due to increased current flows.

RE: Big blackout. What happened?

Sidi: The system is set up with areas to serve as islands (and what areas of load) and then there are sub-islands, and even sub-sub-islands depending upon what the relay scheme is -- selecting what load is out there is general -- and it will change somewhat hour by hour -- and the market now dictates what unit load is or even if a unit is running... there is little monetary motivation to add local generation unless it beats the market... (and the regulators approve)

yes there are load shedding schemes involved, but these, to the best of my knowledge are given areas (usually determined by location of substation breakers)with a semi-known load and not specific MW (although there are areas where the load is somewhat constant) -- and these load areas change all the time (we call it urban sprawl)..

and (I have been in a disturbance that resulted in a black-out) just because you are shedding load and trying to or actually islanding, the significance of the disturbance may or may not let you control the outcome (big islands are easier to handle than the smaller ones) -- in the case of PJM interchange, they were far enough away and the disturbance had't reached the proportions of closer in, they were able to open the appropriate breakers in time to isolate themselves... and help stop the blackout at their borders...

the answers seem easy, but in some cases, you're looking at a moving target and the best scheme is a shotgun approach that will probably not protect in all cases... just how many $$ do we spend?  I think there are many short comings to the current schemes that will be addressed; there are a few that may be swept under the rug, and those with political agendas will still have a voice in the outcome..

RE: Big blackout. What happened?

Lukedric, any criticism on WECC is unjustified. Voltage instability was not a well understood phenomenon until recently. If anything, the WECC deserves credit for learning from the 1996 blackouts and for pioneering the reactive margin methodology, which was unknown at the time. It will be interesting to see if voltage instability was the cause of the recent blackout anf if the WECC method will be adopted by the eastern system.

Pablo02, I don't know what you mean by "semi-known loads". We have sufficient metering to know what the loads are at any time of the day or night. Load shedding schemes trip distribution or subtransmission lines that feed load substations. The proposed special protection scheme would know what the loads are from metered quantities. It's not as complicated as you think.

RE: Big blackout. What happened?

A brief exerpt from NERC's 2003 Summer Assesment Report (dated May 2003) in regards to Eastern and Central Area Reliability Coordination Agreement (ECAR) which includes MI, IN, OH, WV and KY, on the subject of transmission -

"There is a continuing need for the reliability coordinators, transmission planners, and operators to communicate and coordinate their actions to preserve the continued reliability of the ECAR system. It is anticipated that the ECAR system could become constrained as a result of unit unavailability and/or economic transactions that have historically resulted in large unanticipated power flows within and through ECAR.If these conditions occur again this summer, local operating procedures, as well as the NERC Transmission Loading Relief procedure (TLR) will need to be invoked in order to maintain transmission system security. As long as trnasmission limitations are identified and available operating procedures are implemented when required, no cascading events are anticipated."

It goes on to discuss some improvement projects that are planned or underway in the ECAR region and advises that until the upgrades are completed, reliability of the system will depend on operating procedures.

It would be interesting to know if any TLR's were in effect at the time.

RE: Big blackout. What happened?

I just did an internet search for information on the reactive margin methodology that SidiropoulosM mentioned.  Found this techical paper that relates low inertia compressor motors for air conditioning, heat pumps and refrigeration, electronic power supplies, higher transmission and increasing use of capacitor banks, to problems with voltage stability.

http://www.transmission.bpa.gov/orgs/opi/Power_Stability/PwrElectPanelAmSC.pdf

Me thinks that perhaps fasting tripping on some types of loads should be considered.

Margaret

RE: Big blackout. What happened?

alehman --

Yes, there were TLR's in effect at the time of the cascading outage.  See:
http://www.nerc.com/~filez/Logs/index.html

Keep in mind that there were also several transmission outages that occurred within a short time frame which triggered the event.  TLR helps to prevent overloading of facilities, and typically is implemented for known areas of concern (facilities which have been observed to overload either through real-time operation or through studies, or flows / outages which can cause voltage or stability concerns), although it can be implemented for new facilities as needed (through temporary flowgates).  I believe this event to be far beyond anything which was studied or anticipated.  I don't believe this event could have been prevented easily, certainly not without massive preemptive shedding of load.

All --

A major lesson to learn from this event is that less-probable contingency studies must be performed, and linked to relay setting analysis.  Although this event appears to be more related to voltage stability than to overloading, the lesson is still applicable -- how does relaying react to problems on the system, especially those due to unlikely, but plausible, outages (parallel lines or transformers, bus outages, etc.)?  It's quite possible that some facilities have relay settings (such as backup 3-phase time overcurrent relays or zone 3 distance relays with settings that approach load levels) that will trip the facility when it's overloaded, especially if the overload is large or if it persists for some time.  Such tripping, if not coordinated, can simply transfer the problem to other facilities, which could also then trip.

RE: Big blackout. What happened?

Jstickley, you make a distinction between voltage instability and overloading. We should keep in mind that voltage instability is a direct outcome of heavy loading. Transmission lines produce reactive power when loaded below their surge impedance loading (SIL). They absorb reactive power when loaded above their SIL. The heavier the loading is, the more reactive they absorb and the greater the reactive deficiency...and the greater the voltage sag along system busses. There is a loading limit which, if exceeded, will push the system into voltage instability, due to a huge reactive deficiency. For high voltage lines this limit is usually well below the thermal limit of the lines.

RE: Big blackout. What happened?

SidiropoulousM --

My concern about clarity earlier is coming back to bite me.  Should have reread my post.

I certainly did not mean to imply that voltage instability and loading issues are unrelated -- they most definitely are.  However, in general, here in the Eastern Interconnection, a good portion of transmission lines are capable of exceeding their thermal limit without much concern about voltage stability (true for lower voltages primarily, up to about 345 kV -- higher voltage facilities are a different matter).

What I intended to convey through my post was that sometimes it's better to overload a facility than to trip it out of service.  Tripping heavily loaded lines can possibly cause overloads on other facilities and start a cascading outage (due to relaying or voltage collapse, either could be triggered in this manner).  I guess the bottom line is this -- when lines are heavily loaded, that's when they're needed most, and when you least want them to trip.

What I'm getting at (this could be a completely new thread, relaying purpose / philosophy) is that preventing overloads is not the primary purpose of relaying -- that's an operational issue, one that must always be a priority.  Protective relaying is intended to prevent damage to equipment due to faults.  Of course, with anything, there are exceptions (special protective systems are one example).

RE: Big blackout. What happened?

Apparently, PJM has an automatic islanding scheme that was instrumental in keeping PJM alive during the blackout. For an interesting description of control room activity just before the blackout, visit:

http://www.energycentral.com/sections/news/nw_article.cfm?id=4081982

Jstickley, I agree that tripping a heavily loaded line is not a good idea...it will likely overload other facilities and may lead to cascading. We don't do that here in the west. We just cut schedules. Non-firm (interruptible) loads are cut first. Higher priority loads may be also cut, if necessary.

On your last paragraph, I agree that the purpose of protective relaying is to protect equipment against faults. We usually make a distinction between protection systems and the so-called "special protection systems", which here in the west we call remedial action schemes (RAS). We have several RAS schemes in place. Their purpose is usually to take specific action for specific contingencies in order to prevent instability or collapse. I like to think of them as "system protection" as opposed to "equipment protection".

RE: Big blackout. What happened?

So you guys have got me interested.  Investigating SIL, I found several. One is at http://www.ece.nmsu.edu/~sranade/Ee535l5new.PDF

From my "interpretation" of this, it appears that due to a variety of means of developing impedance in AC power lines, a line's capacity and therefore the cost of transmission change significantly above about 250 miles (at certain sufficiently high volatges).

Does this shed any light (for me) on the recent e.g. Enron scandal(s)?  Were they trying to avoid a SIL based penalty charge for a long distance transmission (e.g. Texas to Cal) by selling blocks to "theoretical" intermediate customers at 250 mile intervals who then re-sold to the next "theoretical" intermediate customer 250 miles further on, until the block of power arrived at Cal?

RE: Big blackout. What happened?

lengould --

This is off-topic, but I'll answer.  Perhaps forum admins can move this to another thread.

SIL is totally unrelated to the Enron scandal.  Enron was basically manipulating the market to "create" congestion (and thus increase the value of their energy trades).  In essence, they took advantage of loop-holes in market rules that allowed them to artificially inflate prices to benefit themself.  I'm sure there are reports available on the web to further describe their actions (a simple google search such as http://www.google.com/search?hl=en&ie=UTF-8&oe=UTF-8&q=enron+market+manipulation brings up a lot of links, although you still have to sort through the junk).

Although I haven't read the paper you posted, the cost that you're reading about is surely in regard to cost of building transmission.  Basically, as the length of a transmission line increases, so does the cost of building the line, the cost of the losses absorbed by the line, and the cost of supplying reactive power (via capacitor banks, etc.) to support voltage on the line.  At some point (as I recall, about 250-300 miles according to most studies/textbooks), the economics dictate that it's better to build high-voltage DC rather than high-voltage AC (DC lines are cheaper to build as compared to AC lines due to materials required, but require expensive converter stations to interface with the grid that bump up the cost quite a bit).

As far as I know, there is no transmission tariff out there that addresses SIL in any way -- typically tariff rates are dictated by the revenue requirements of the grid (determined by value of facilities, costs of operation and maintenance, etc.).

RE: Big blackout. What happened?

Interesting story I just ran across on CNN.  Looks as if the 345 kV outages identified in NERC's preliminary report caused massive overloads of other transmission lines in the system, leading to cascading outages of the facilities.

http://www.cnn.com/2003/US/Midwest/08/22/blackout.investigation.ap/index.html

Of course, NERC's investigation is just beginning (I understand massive data requests have been made to utilities in and surrounding the area of the blackout).  It will be interesting to see what specific event triggered the cascade and what could have been done to isolate the problem.

RE: Big blackout. What happened?

What about the situation with Hydro Quebec, which is a major energy supplier to North Eastern US areas including New York. They seemed to continue without outage, whereas the adjacent province of Ontario was without power. Major cities such as Montreal were unaffected, whereas everyone in the same region of North America was down. I read a statement that they were deliberately out of phase with the rest of the region. What does that mean?

RE: Big blackout. What happened?

Modula2:  I checked out the Hydro Quebec web site and found a lot of information about their transmission to other areas, but haven't found any thing about that deliberately out of phase whatever... as of yet.

Here is a link to their path diagram for metered interchanges:

http://www.transenergie.com/oasis/info/ilot/guide_en.pdf

I will keep on looking to see if I can find some more information.

Margaret

RE: Big blackout. What happened?

Almost certainly Hydro Quebec uses the thyristor DC - AC converters where its transmission lines interconnect with others to limit any effects of customer systems on it's own system.  Which may also be why it is not a member of the regional group, which could impose overriding requirements on this(?).

Which is what my previous post was suggesting, each large (state / province sized) area should implement similar load control systems at their borders to enable islanding in the next event.  And, pardon me but I tend to think Ontario with 30 Gw generation should be able to operate as an island for at least a little time.  Surely they're no all induction generators, and controlled emergency load shedding for peaks should be able to handle. No?

RE: Big blackout. What happened?

lengould:  I just opened some of the individual diagrams for Hydro Quebecs schematic.  

You are correct, they are using DC to AC convertors, etc.  

I knew that that was how the lines from up North operate... because of the very long distances involved... and now have found out that that is what Hydro Quebec does for selling power to other networks.

Margaret

RE: Big blackout. What happened?

Suggestion: Assuming that the generators will be sensitively protected and trip on a grid condition that is potentially harmful to them, then it is necessary to concentrate on the grid and transmission line to minimize or eliminate outages.
For example, consider 500kV Transmission Tine (TL) that can carry about 1000A. This gives kVA=sqrt3 x 500kV x 1000A = 866,025kVA
If this type of TL malfunction, the grid will be somewhat affected. Now, if by some coincidence more transmission of this size trip, the grid is in trouble. To minimize or eliminate such trouble, the following is needed:
1. Transmission Line Loading. The large transmission lines will either have to have a redundant transmission line on standby or carry a small percentage of MVA such that, if it happens to malfunction and another similar one at the same time (approximately), there will be a loss of power flow tolerable by the grid.
2. Transmission Redundancy. As mentioned in 1. the powerful transmissions need redundancy to be able to switch to another one in case of malfunction in the transmission or in the switchyard.
3. Transmission Topology. It is known from the computer networks that there are Ethernet, Star, Token Ring, etc. topologies. Each one has some advantages and disadvantages.
Therefore, the transmission line topologies should be built as required. E.g. the switchyard should have a topology corresponding to its importance in the grid, e.g. dual ring bus, if needs to be (Why do computer networks or fiber optics backbone have dual buses, e.g. dual token ring?).

Certainly, the transmission lines importance, as evidenced by the blackout, does not lend itself for any kind of monetary speculations, "milking", "Wall Street Greed", etc.
All applicable and proven designs, engineering, and scientific theories have a plenty of room in the electrical power transmission and distribution industry.

RE: Big blackout. What happened?

lengould:  I just spent a while searching on the internet to see if I could find information about Ontario's generating capacity.... with the result that I will freely admit that I am confused.  Some web sites put forward the notion that Ontario has enough generating capacity... while others comment on how many of the generating facilities are out of date and that Pickering has apparently had 4 reactors down for years, etc., etc.  

As for the "deregulation" situation in Ontario... it seems that the costs per kwh are now above what the consumers are paying and that the Ontario government is making up the difference with tax payers money.  Sigh.

From what I have read it seems to me that every different electrical utility, or whatever... whether government or privately owned, in North America seems to have it's own individual attitudes towards how money is allocated for engineering tasks and maintenance and/or upgrading, as well as operation of their networks. Hydro Quebec is government owned... but makes money by selling power to other utilities... hence it seems logical that they have invested a certain amount of money in the physical equipment and transmission lines, etc., that are utilized to sell power.

But as was very well documented a few years back....  Hydro Quebec was....  and may perhaps still be.... some what short of redundant feeders in the area south of Montreal.  Okay.... so I know people who were without power for over a month in 98.... during the winter.

In my opinion, this whatever ties in with jbartos mentioned about "milking" and "Wall Street"... and I will include also "Bay Street" greed.  And please note that I have the feeling that even government owned utilities may be affected by greed when it comes to certain types of decisions.  Sigh.

I think that there are many possible engineering and technical type solutions out there.... to hopefully avoid similar large black outs in the future.  But the whole operative whatever is who exactly pays for the improvements.  In my opinion, it probably will fall on the consumers.  Sigh.

Margaret

RE: Big blackout. What happened?

Margaret695 - Easy to see why much confusion re. Ont. generating capy.  e.g. OPG site states they own 24,278 MW of generation in Ontario, but the detailed list includes only 4 of Ont's Nukes. (I know at least the 8 at Bruce (4 working, 2? being re-commissioned, 2 dedicated to heavy water production now mothballed) are owned by Trans Canada Pipelines, bought from British company last year who bought them from Ont Hydro about 2000).  TCPL also owns all the big Nat. Gas Turbos like the ones at Pearson Airport, which I think I remember the Rolls Royce engineers at the building I used to work in telling me were 100+ MW each.  Not sure who owns the remaining Nukes.  If OPG only 4, who owns Pickering 8? or Darlington 4?

http://www.hydroone.com/en/about/quick_facts/
Hydro One - "Transmission Serves Ontario Generators: 174 (includes 78 generating stations owned by Ontario Power Generation)."

I've seen their trading website, listing all the gens in Ont. at about 30,000+ MW.  (Fairly evenly divided 1/3 Hydro, 1/3 Nuke, 1/3 Coal+Gas+Wood Waste from Northern Pulp mills).  Website seems to be shut down now, least I couldn't find it.  Also states "All-time System Peak: 25,629 MW".  Balance of 5,000+ must be held in reserve / offline for maintenance etc.

Re. Why Ont. now short of power, my impression is, short term problem due to a safety system which dumps gadolinium into the reactor core of the nukes at Darlington under certain conditions, which apparently happened.  Takes about a week for the gadolinium to (decay/get filtered out).  Also about (some number between 6 to 12) workable reactors were mothballed in the 1990's, I think largely for political reasons while the NDP govt. was in.  Eves PC govt. has been trying to get (some of) them back on line (4 at Darlington), but is taking a lot longer than estimated.  Was stupid to shut them down with no alternate strategy, but so goes politics.  Hillarious to hear current NDP opposition now on TV criticising PC's for not arranging enough power.

Ah well.  Voters have such short memories.

RE: Big blackout. What happened?

Ontario is a net importer during peak times, which they attempt to hide by giving average import/export values, with of course the vast majority of exports occuring off peak.  On any given week imports of 2000MW are common, which comes almost mostly from the ties in the east to New York.  The govt agency Ontario Power Generation owns all nukes except Darlington, the PC govt has been pushing privatization of generation and has sold Darlington and approx 400MW of Hydro and will likely keep going if buyers appear.  The transmission system which is almost totally owned by another govt agency Hydro One was under threat of privatization but the PC govt recently backed off.  With the nukes coming back online (which is required for long term planning) we should be self sufficient, but with rising demand and next to zero development (public or private) we seem to be in trouble even with the nukes online.

It boggles my mind how the privatization can be viewed as positive, I don't see how improved efficiency in these organizations will out weigh the profits taken, not to mention reliability.  With deregulation here power prices increased, and the increased profits to the govt agency OPG are being used to offset increased costs, but the private gens stick it in their pockets.  So where does the tax payer gain with more privatization?  It seems the govt is purely interested in the one time booster shot gained by the asset disposition.  I think I'm crossing the line to rant, so end here.

RE: Big blackout. What happened?


For some new info on the blackout visit:

http://www.energycentral.com/sections/news/nw_article.cfm?id=4087012

From this article it seems that a large generator in Ohio was lost, which should have prompted operators to cut load. They chose to rely on interies instead, which soon overloaded and began separating the systems. Which, in turn, caused other interties to overload, causing more separation etc.

RE: Big blackout. What happened?

As I could understand from the postings, the episode started with the inadequate transmission capacity. There was a very interesting article in IEEE, Spectrum in July 2001 highlighting the defeciency in transmission system. I reproduce the abstract here under.


Energy woes [US National Energy Policy]
Sweet, W.   Bretz, E.A.   
This paper appears in: Spectrum, IEEE

Publication Date: Jul 2001
On page(s): 48-53
Volume: 38,   Issue: 7
ISSN: 0018-9235
References Cited: 0
CODEN: IEESAM
INSPEC Accession Number: 6988066



--------------------------------------------------------------------------------
Abstract:
The long-awaited and much-discounted National Energy Policy report, issued by a task force headed by Vice president Dick Cheney, was not the one-dimensional document critics and adversaries of the Bush administration eagerly awaited. Much of the criticism since release of the report would seem, in fact, to reflect more what people expected to read than what is actually in the report. This paper excerpts the report. In addition, for a contrasting view, the paper excerpts a report prepared by researchers at several national laboratories which presents quite a different outlook on the role fossil fuels need play in the country's energy future. In the report, the very real crisis in electricity has been bundled with concerns about increased prices for gasoline and home heating oil-increases that may prove quite transitory-to sell the public hard on an all-out effort at new production, with environmental protection given much less emphasis. What the administration seeks from the public, at bottom, is much more production of fossil fuels and streamlined or less onerous procedures for approving new plants and infrastructure

You may visit the site:
http://ieeexplore.ieee.org/iel5/6/20169/00931883.pdf?isNumber=20169&prod=JNL&arnumber=931883&arSt=48&ared=53&arAuthor=Sweet%2C+W.%3B+Bretz%2C+E.A.%3B

The article states:
Energy used more economically but Electricity infrastructure deteriorates. Consumers are spending less of their household income on energy now then 1973 energy crunch.But transmission owners have spent less on infrastructure despite growing demand.

It further states:
Since 1989, electricity sales to consumers have increased by 2.1 percent annually, yet transmission capacity has increased increased by 0.8 percent annually.

Transmission projects always get lower investment priorities than the generation projects.

One of the respondent has rightly suggested islanding of the local networks. There was a collapse of large part of national grid last year in India. Some utilities, who had installed islanding relays could survive the unwanted trippings. They also helped to revive the system by feeding necessary power.

RE: Big blackout. What happened?

gordonl:

You stated OPG owns all nukes except Darlington.  If true then they must be leasing etc. the Bruce facility to the (British Energy --> TCPL) etc? group who are operating.  Seems more likely Ont. Govt. would be leasing not OPG.

http://www.brucepower.com/brucepower/

"Duncan Hawthorne is President and Chief Executive Officer of Bruce Power. Prior to the recent change in ownership of Bruce Power, Duncan also served as British Energy's Executive Director - North America. He remains a member of the British Energy Board of Directors, "

further -
"Bruce Power employs more than 3,000 highly-skilled employees with experience in the safe and reliable operation of nuclear generating stations. With safety as its first priority, Bruce Power generates enough clean electricity to supply approximately 15 per cent of Ontario's power needs."

Agreed Ont. is net importer at peaks, but IMHO for stupid reasons.  According to following, 2 of the 4 BruceA units ars back "summer 2003"?  Doesn't say now or later.  These guys (politicians) should ALWAYS keep Ont. in net export condition, no reason not to with all the "spare" nukes about.

"Plans for the restart of Bruce A Units 3 and 4 are now in full swing, with both units expected to be online by summer 2003, subject to a thorough environmental assessment and receipt of all necessary regulatory approvals. The restart of two of the Bruce A units, together with performance improvements to Bruce B will deliver another 2000 megawatts of clean electricity into the Ontario market"

RE: Big blackout. What happened?

WOW.  Interesting perspective is right!  Just read that IEEE Spectrum article, and can't help but wonder if the author hasn't penned a lot of words he'll have to eat later.  There's an awful lot of finger-pointing and sweeping generalities if you ask me.  Although most of the issues identified are very much real, the conclusions and overall negative tone of the article are quite bothersome.  I'll admit, I believe the electric power industry has some problems, but to basically trash everyone involved from system operators to elected legislators is a little too much.

I don't know about anyone else here, but the allegation that cuts deepest to me is the assertion that the "best students are not going into power engineering".  That one gets personal, huh?  Nice to know that, in general, power engineers apparently aren't much brighter than trained monkeys, especially since the industry is so simple to understand and its engineering tasks so straightforward.

(BTW, if you're not catching the sarcasm here, maybe he's right!)

RE: Big blackout. What happened?

lengould,

Yes, for whatever reason Darlington was on my mind, but indeed it is Bruce which was sold not Darlington.

RE: Big blackout. What happened?


jstickley:

I agree with you totally on the inappropriatness of such comments as follows in the Spectrum article.

<quote says Roytelman, echoing a viewpoint that is ubiquitous among the leading U.S. experts on the grid system, "the best students are not going into power engineering." >

What I don't get (and I suspect is a mere reflection of the quality of the authors), they spend nearly the entire article explaining why all the problems of the grid system are financial and political, then close with the kick at engineers.  They havn't indicated a SINGLE REASON why better engineering might resolve the problem, even though several posters here have, in ways far more acceptable.

Is the lack of new grid capacity caused by lack of engineers?  How long might it take to assemble a team of engineers who could "great circle" the continent with 10 or so MVA of 1MV DC if they didn't have to pay from their own pockets and fight local wars at every property line?

Very poor reporting job. "leading U.S. experts on the grid"? Ha.  He forgot to insert financial there.

RE: Big blackout. What happened?


Sometimes we shoot the messenger for the bad message. Ilya Roytelman, a distinguished researcher with many contributions in the field, said that "the best students are not going into power engineering". Now, what's wrong with that statement? If it's not totally and absolutely correct, it's partially correct, it has an element of truth. Certainly in Roytelman's experience, as he is an academic. And, I can say, in my own experience also. When I graduated in 1974, the brightest students went into digital electronics and Bio-med. In any event, if there is a problem it will not be solved by those who are offended but by those who can face the truth and deal with it.

On the other hand, many bright people have gone into the various fields of electric power and the quality of technical research in our field is the best testamemt to that. Many "average" students have developed into formidable experts when faced by the enormous complexity of the electricity supply systems.

Now, on the subject of the Spectrum paper that I posted above. I really don't know why some have come down on it so hard. Is it because it says that:

"...Long before power had been restored to businesses and residences from New York City to Cleveland, Detroit, and Toronto, politicians and commentators on both sides of the border were pointing fingers..."

"...For more than five years, NERC has sought and failed to get legislative authority to make its rules mandatory..."

"...as growth in demand for electricity has outstripped additions to transmission capacity by a factor of two, the grid itself has come to be ever more thinly stretched..."

..."We all knew something like this was coming along," a leader in a 1999 DOE study of the transmission grid told IEEE Spectrum, on condition of anonymity. "We were all just waiting for the big one."...

"...The temptation, at first blush, may be to put the blame squarely on FirstEnergy, the Ohio utility whose operators and managers seem to have been sleeping at the switch. Could this be a mere case of one poorly run company bringing down an entire system—of one rotten apple corrupting the whole barrel?..."

..."Reporters for The Wall Street Journal, The New York Times, and ABC-TV's "Nightline" news analysis show promptly uncovered a pattern of chronic difficulties FirstEnergy had had with regulators. These included penalties for violation of health, safety, and environmental rules, a requirement by auditors that it restate its profits over the last few years, and the cost of replacing electricity from the shut-down Davis-Besse plant, which ran into hundreds of millions of dollars. The company also is saddled with billions of dollars in debt, much of it associated with its acquisition in 2001 of the utility GPU Inc. (Morristown, N.J.), the former owner and operator of the Three Mile Island nuclear plant that suffered a near meltdown in 1979..."

"...Even if the tripped lines and FirstEnergy’s failure to detect and report them prove to be the exclusive initiating cause of the 2003 blackout, it still will be necessary to account for the failure of local and neighboring control authorities to prevent the cascading outages that ensued. As events unfolded, observers were struck at how unevenly various regulating organizations performed..."

"...Long before most states had even begun to set up ISOs, however, it was becoming apparent to federal regulators that the supposedly independent organizations were held hostage by local utilities and other special interests. What’s more, ISOs lacked adequate authority to get transmission lines built and transmission services properly priced..."

"...Six months ago, NERC singled out the Middle West as the one part of the country at risk of a devastating grid breakdown, and put local utilities and regulators that they had to be on high alert this summer. Evidently the message did not get through. Why is that?..."

..."The market forces that caused this," the report went on to say, "pervade all of North America. Similar effects should be expected in other regions as well," including the eastern interconnection, where the same kind of migration of technical support from utilities to emergent ISOs, with attendant loss of memory, was apparent....

"...William Hogan of Harvard University’s John F. Kennedy School of Government (Cambridge, Mass.) noted that the Senate Commerce Committee had recently voted to bar FERC from implementing RTO market designs based on the PJM Interconnect model. But, he wrote hopefully, "the blackout should change the game."...

I can go on and on but I'll end up reprinting the whole article. Open your eyes and read it once again (or more than once if necessary).









RE: Big blackout. What happened?

Sid:  I can't find a single instance in your selections which would indicate <poor quality engineering> might be a cause or even a contributer to the problem.  Some might point to some operaters asleep at a console, but engineering?  It's a tribute to the engineering that the system can survive at all given the management problems described.  The <last kick> at engineers was unjustified.


"the supposedly independent organizations were held hostage by local utilities and other special interests. WhatÂ’s more, ISOs lacked adequate authority to get transmission lines built and transmission services properly priced..."

..."The market forces that caused this,"

RE: Big blackout. What happened?

SidiropoulosM --

I certainly was not shooting the messenger.  No personal attack was made (or intended) upon Ilya, nor upon the author.  My comments were simply expressing my reaction to the article -- I felt that although it made some good points and summarized some of the main issues, it had an overall tone of negativity which detracted from it's factual content.  I also felt that the implied assertion that the blackout was somehow related to a lack of bright students studying the field of power engineering was ridiculous -- while it's true that power engineering programs suffer from a lack of prestige and do have smaller enrollments than other electrical engineering disciplines, I find it difficult to believe that they do not produce quality engineers who are capable of planning and operating a reliable power system.

I stand by my comments (which I'll be happy to elaborate upon more, if need be), and am sincerely sorry if you did not catch my poor attempt at humor.

RE: Big blackout. What happened?

jstickley:  Superb posting.  Clear and (sounds) authoritave.  The 9th frame carries heavy condemnation of Ontario IMO.  ".. stays interconnected and tries to support both Michigan and Ohio for 2 minutes."  Clearly an operating error or absence of forethought in the operating rules.  Could be called cause of problem for all of Ontario and New York.  Definitely needs fixing.

RE: Big blackout. What happened?

All --

Although the information presented in the ITC analysis seems very clear, the root causes of this event are still under investigation.  I would caution everyone to be patient and let NERC (and others) run the course of their investigations, and try to refrain from jumping to conclusions.

The one thing that bothers me most about this entire situation is the amount of finger-pointing that is going on, most all of it based upon very preliminary information.  So many different organizations and individuals are trying to use this event as fuel for their own agendas, and almost all of them are doing so without firm information regarding root causes of the event to back their stance.  Let's all stand firm, and try to refrain from playing the blame game (note, I too am guilty of this).  Answers will be given in due time; if appropriate, blame can then be placed upon those who deserve it.

RE: Big blackout. What happened?

Suggestion: The following links from above are reproduced:
http://www.itctransco.com/documents/B_8-14-2003_ITC_Analysis.pdf
http://www.spectrum.ieee.org/WEBONLY/special/aug03/black03.html
to focus on their contents addressing the weaknesses in the power distribution and generation hardware. It is clear that the hardware can be designed with redundancy, and with safe loading that the grid would be "selfprotected" or "immune" to more than one fault in a specific location. As the bottom link indicates, a series of adverse events in one narrow location triggered the "grid dominos" to fall. This is the actual vulnerability of the system, which has to be remedied to avoid other blackouts. The built-in robustness in the power transmission and distribution will be a good investment for the future considering that the oil reserves depletion is inevitable in the future time that has already been  estimated / projected.

RE: Big blackout. What happened?

jstickley:  Agreed.  Sorry for stating any conclusion, just running off with an "ass"umption.  If makes you feel any better, I'm not involved in industry and have no influence whatever, just one more interested resident of area affected.

RE: Big blackout. What happened?

In the reference http://pro.energycentral.com/professional/news/power/news_article.cfm?id=4087012
there are these two paragraphs at the end:

Even before last week's blackout, federal regulators were keeping close tabs on the way power is managed in the Midwest.

Since May, the Federal Energy Regulatory Commission has been on site monitoring the Midwest Independent Transmission System Operator, which began operating less than two years ago and oversees the Ohio utilities that suffered line failures.

Do any of you attach any significance to the fact that FERC people may have been on site?

RE: Big blackout. What happened?

Suggestion: It appears that this last blackout will gradually become forgotten just like big New York City blackouts in 1965 and 1977. Visit
http://blackout.gmu.edu/main.html
for more info

RE: Big blackout. What happened?

Thats pretty depressing.  Curious.  No such thing as web in '66 or '77.  Where were these surveys getting their input?  Whatever it was, doesn't appear to be available now.

RE: Big blackout. What happened?

I have a client with a facility on Long Island NY. His building is part of a load curtailment program. When the utility company foresees a load problem (or any problem) on the grid which would ease by reducing load, they contact this building owner to start their standby emergency generators, and have them remove themselves from the grid until notified.
This summer, surprisingly, they had not been contacted as they have been several times per summer in the past, to remove themselves from the grid ....... until 12:30PM on the day of the blackout (4 hours before NYC went down). Somehow, LIPA knew to activate their load curtailment program, 1 1/2 to 2 hours before any major items are documented in Ohio.
  

RE: Big blackout. What happened?

RonShap, this is interesting. It probably indicates a low voltage situation on the LIPA system. Which means heavy loading of transmission lines rather than a generation deficiency. This may explain why the Ohio events caused the cascading which led to the blackout.

It seems odd that after three weeks we still have to speculate. I wonder how long it will take before we know the root cause.

RE: Big blackout. What happened?


Regarding the threat of a terrorist attack on the grid:

"Government scientific advisers and officials painted a grim picture Thursday of the consequences of a terror attack on the nation's power grid, saying that any outage that lasted longer than a couple of days would reduce urban centers to chaos and collapse the economy.

"With power out beyond a day or two, both food and water supplies would soon fail. Transportation systems would be at a standstill ... natural gas pressure would decline and some would lose gas altogether -- not good in the winter time ... Communications would be spotty or non-existent. ... All in all, our cities would......"

visit:  http://www.washtimes.com/upi-breaking/20030904-070049-7423r.htm

RE: Big blackout. What happened?

At least I agree with this statement.

quote <a successful assault is "a lot easier than we wish it were.">   Seems they do a lot of wishing.

Quite humerous to read about the PhD student causing the stir.  Genius?  Sure it wouldn't require Einstein to map utility or fiber grids, just a lot of unimaginative effort.  They give PhD's for that sort of thesis?  Course it may have included a lot more that referred to in article.

I think they were 'way alarmist in article, though everyone cited and the reporter himself had a clear interest.  Something like Y2K computer problem, only ones talking were those with something to gain, including journalists.  Of course way out of proportion.

Was just talking with my son about the 1 km asteroid just spotted with 1000 to 1 odds of hitting earth.  Were speculating on result.  He'd heard that if it hits, will throw enough smoke to darken sun for 9 mos, enough to cause near total crop losses worldwide for at least 1 year.  Would be 10 yrs warning, but we agreed that, with reasonable absence of panic very few would need to starve.  Reality different though, I suppose.

I can't agree with doom scenario described, however.  In 1999 Montreal area was hit with power loss due to ice storm in middle of winter.  Most people were without power for about 2 weeks.  Large city, subways etc.  Much hardship but very little real injury.  People very soon figured out how to survive the event, and in doing so proved, I think, they could have done it much longer.

But I may be wrong (again).

RE: Big blackout. What happened?

Comment: A couple of years ago, a major Diesel genset manufacturer was giving a presentation of its genset products. It was mentioned that 6kW to 10kW gensets were being in high demand by home (house) owners. Apparently, there is some distrust about the electricity supply availability no matter whether the electricity grid is reliable or not. The apartment buildings, except a few, do not have back up gensets, in general; therefore, it is much harder to survive the blackouts in apartment buildings.

RE: Big blackout. What happened?

Suggestion: Visit
http://www.nautilus.org/energy/grid/papers/bickel.pdf
for:
Figure 4 above shows the desired performance of
a grid in which a 1300 MWe NPP suddenly trips.
In the short term (first 20 – 30 seconds) only the
availability of interchange power prevents the
grid from collapsing. Over the longer term, the
ability to dispatch hydroelectric power recovers
the grid frequency to the nominal 50 Hz.

RE: Big blackout. What happened?

As a newbie here, I went right away to http://www.nautilus.org/energy/grid/papers/bickel.pdf
and there is a rather interesting point to be found about nuclear plant standards.

It appears that U.S. plants  are  predicated on the availability of reliable shared power from the grid, precisely because it is a big grid full of interconnects. The  author puts this in contrast with, say, Puerto Rico, where the  grid is not as stable. However, the author later says, "overall the trend of U.S. [nuclear power plant] trips  has decrease significantly over the ast decade and reliability of onsite power systems has increased. However, as a result of electric utility industry restructuring and the lack of addition of significant new generating plants, capacity margins are less. This brought with it the potential for larger cascading grid events."

BTW, the author glaringly erred by talking about the "1966" big Blackout on his first page, when it was actually 1965.

RE: Big blackout. What happened?

Star to jbartos for informative article.  Bet it can get pretty hairy in the control room of a pair of reactors on  a hot August afternoon when the whole state trips out.  All eyes on the temp. indicators.  Startup the hamsterwheel pumps.  Well no, but I stayed in a Days Inn last night.

I still don't quite see why every reactor isn't required to have 250% over-capacity in its condenser installation (capable of sinking total Mwt of the reactor), capable of operating continuously as long as  operators want.  In a trip scenario, down the turbine / generator and bypass the steam directly to a condenser designed for the purpose.  Add a small standby steam turbine / gen. in that circuit which is capable of servicing plant site [ x 2+], leave the reactor running until enough information available to decide if necessary to shut it down.  Sure fast-start diesel generators / pumps or eductors could keep such a condenser within sufficient limits to avoid damage to the reactor in all scenarios.

Might at least have saved Darlington from poisoning its reactors, with the week+ delay in restart.  I mean, they've got a whole great lake right outside the door.  Dont mess up the lake by design, guys, but if it's that or a meltdown, by all means, boil a cup.

Course a bit more complicated with CANDU's than the others since need to also run the moderator cooling pumps etc.  Still, should be possible to set up an 8 reactor site with enough redundancy to service all that reliably, even when completely isolated from the grid.

But then what do I know.

RE: Big blackout. What happened?

For a chuckle, try this site.  Everyone onto bandwagon.

http://www.evworld.com/databases/storybuilder.cfm?storyid=567

quote< Lovins predicted the Big Blackout was inevitable twenty years ago in his book, Brittle Power. And he believes that until the system is redesigned to be more decentralized, it will continue to be susceptible to failure, both accidental and deliberate>

No argument with much of what Lovins has said over years, and theoretically most of publications make sense.  Love the design of the Hypercar!  Wish they could get it built (for a price I could afford).  At least he declares his shareholdings in the concepts promoted up front.

Anyway.

RE: Big blackout. What happened?

So, where do we go from here?

1. Can a repeat be avoided if the distributiion system has installed upgraded technology? Can FACTS  systems reduce the chances of another blackout?

2. Newer units in developing countries are designed to allow island operation with quick reload back to former loads , by use of turbine bypass systems and upgraded controls. This could have kept the large coal fired and mukes operating ( likely require fast runbacks to 70% - load) , but maybe some areas would have recovered within minutes.

3. UPgraded solid state components at substations  has allowed reducing the design margins in transmission systems. Has the reduced margin contributed to the failure?

4. It is my understanding that most transmission lines were licensed to carry a specified nominal max load, but that this is routinely exceeded. Is this practice of "winking" at the max permitted loads part of the "human errors" that represent a systemic error ?

RE: Big blackout. What happened?

The article on the blackout by Thomas Casten at energypulse definitely strikes a note with me.

http://www.energypulse.net/centers/article/article_display.cfm?a_id=457

you may need to register (free) if not already

He presents a case for distributed generation in an interesting manner, but IMHO ignores issues of linesman safety, right of access for less priveleged, long-term fuel sources, etc. etc.  These well-known hurdles should have been mentioned.

RE: Big blackout. What happened?

davefitz --

Let me give your questions a shot.

1.  No, and maybe.  The distribution systems had nothing to do with the blackout -- bulk transmission problems triggered the event.  Doesn't matter how great your distribution system is if you can't get the power to it.  FACTS devices MAY have been helpful, but hard to tell without full details of root causes of the event.

2.  Maybe.  Restoration of islanded systems is a tricky procedure.  I would bet that even if some generation were rapidly restored, it would have still taken a couple days to completely reconnect the system.

3.  What design margins are being reduced due to solid-state components?  Guess I need more clarification before tackling that one.

4.  To my knowledge, lines are not licensed to carry certain amounts of load at all.  They may be designed to certain criteria (sag, tension, clearances, etc.), but the rating of the line is dynamic in nature due to the variety of factors that influence its current-carrying capability (say that five times real fast), assuming of course that the line is thermally limited.  Yes, we do tend to push the envelope more now on conductor ratings, but the system is also stressed in ways never anticipated when designed, and system operators and engineers are forced to find ways to transmit more energy over the same facilities.  That being said, I do not believe any self-respecting engineer or operator turns a blind eye toward facility ratings.  This operational necessity is a symptom of a larger problem -- the difficulty in expanding the system to meet the extreme demands of the electricity market (which has been addressed in many ways in this thread already).

Now for a little rant:

I don't know about others here, but I'm REALLY getting tired of reading articles and interviews where people are using this whole event as fuel for their own agendas.  I'd really like to just see the investigation run its course, then have conclusions made at that point (and blame placed, if appropriate).  No offense intended to those that have placed links to such.

Anyway, let me reiterate a point I've made before:  no power system is 100% reliable.  The question to be answered is this:  is the system planned and operated in accordance with established reliability criteria?  If so, the question becomes:  is the reliability criteria we have good enough?

RE: Big blackout. What happened?

Came accross this Sept 5 interview of FirstEnergy CEO.  Guy's good, you come out of the read feeling sorry for them.  Provides very few real facts except "only 40% load on tripped line", "long record of reliable operation" of lines involved.

Definitely a friendly treatment by (Ohio based) reporter.  Don't know.

http://www.ohio.com/mld/beaconjournal/6704181.htm

RE: Big blackout. What happened?

Also this from Sept 5, 4:57

http://asia.reuters.com/newsArticle.jhtml?type=topNews&storyID=3392866


quote<A transcript of telephone calls between FirstEnergy and the Midwest power grid operator showed chaos and confusion in FirstEnergy's control room in the hours before the blackout.

"We have no clue. Our computer is giving us fits, too. We don't even know the status of some of the stuff around us," an operator at Akron-based FirstEnergy said in the transcript.>


Then further:

quote<Another member of the task force said it could take up to a year to complete a report on the cause of the outage and recommendations to prevent another one.>

They won't talk until a year?  Something smells bad to me.

RE: Big blackout. What happened?

Is this report of interest to the current discussion? [I think it might be, but I am just getting up to speed here.]

http://www.selinc.com/techpprs/6141.pdf

It is about a Mexican blackout in February 2002 that was monitored and documented and evidently included effects that were novel.

RE: Big blackout. What happened?

lengould --

Something may "smell bad" to you, but I'd imagine their estimate on time frame for the report is about right.  Power systems are quite complex, and it will take a lot of time just to sort through the MASSIVE amounts of data that have been requested for the investigation (I can't find the letter online, but trust me, it's a LOT of data), let alone determine every detail that contributed to the event.

Granted, it may be somewhat clear what the root cause was just from initial, high-level data (time frames of line outages and generator outages, for example), but without digging into the details, we'll not truly understand everything that contributed to the event.  If we don't get a full understanding of all of those contributing factors, it would be easy for knee-jerk reactions to dictate industry change, possibly causing bad policies and/or legislation to be enacted.

RE: Big blackout. What happened?


lengould, it does smell a bit. One year is far too long. There's no reason whatsoever why data collection, complete analysis and report should take more than three months. One year is ridiculous.

spitfire, the report you've posted is quite relevant and touches many of the problems that arise from running the system close to stability limits.

RE: Big blackout. What happened?

2
Still I am curious to know what happened

The following are few expressions from the release transcripts of telephone communications provided by the Midwest Independent Transmission System Operator, including exchanges between MISO grid monitors and FirstEnergy, during the afternoon of Aug. 14.

1- "We have no clue," ..an engineer at the First Energy Corp.
2- "Something strange is happening," an MISO technician, Don Hunter, told the Ohio utility, not sure what was amiss.
3- "I've got to find my calculator,"….. he said, trying to get a handle on the power fluctuations.  ( I am curious what could be done with a calculator during this event)
4- "We've got something going on,"…..for the next 20 minutes there was confusion over what lines were out and what the implication might be for the power grid.
5- "We have no idea what happened," ….Snickey said. "We have no clue. Our computer is giving us fits too. We don't even know the status of some of the stuff (power fluctuations) around us."
6- "I called you guys like 10 minutes ago, and I thought you were figuring out what was going on there,"
7- "Well, we're trying to," Snickey replied.
8- "Our computer is not happy. It's not cooperating either."
9- "I can't get a big picture of what's going on," Hunter fretted.
10 - "We won't jump to conclusions,"… Energy Secretary Spencer Abraham said.

RE: Big blackout. What happened?

star to cuky2000:  item 10, good bit of humour.

RE: Big blackout. What happened?

Just studying NERC projections for all regions on North America 2000 to 2010.  Interesting.

http://pro.energycentral.com/professional/industrydata/loadgrowth/default.cfm

  MAIN, which is the area whicn blacked out, claims to have existing 64,170 MW capability 2001, and 74,101 summer demand forecast for 2010, an increase of 9,931 MW predicted to come mostly as dual-fuel turbines (4,677) and combined cycle Nat. Gas. (768) plus out-of-region purchases (2,814).

  SERC (southeast except florida) by comparison is huge, 156,058 2001; 192,306 2010; increase 36,250 all gas turbine or combined cycle.

  WSCC (most of US and Canada west coast) is also huge, at 130,892 2000; 197,741  2010; increase 66,849 again all (95%) gas turbine or combined cycle gas.

from WSCC "Adequate supplies of natural gas are expected to remain available during the next ten years. However, as demonstrated by California’s experience in 2000, natural gas-fired generation may be subject to uncertainties during periods of unstable natural gas prices."

  SERC is planning to add 2,600 miles of transmission in that time, mostly 230KV (%not avail), and WSCC is planning to add 3,000 miles of transmission 230KV or above. (+4.4%) MAIN data not available, though I have seen it elsewhere, and if I remember, the percentage was smaller.  

  Other than the fuel choices for expansions, these analyses looked rational and done by knowlegeable persons.  We should remember that the types of generation planned as addition don't tax transmission the same as big hydro, or even central thermal and nuclear.

  Point is, I fail to see where US$100 billion might be spent usefully on transmission.  Figure 10,000 km of 2 Gw lines in 300Km pieces = US$10.8 billion would likely add some significant stability if reasonably placed. I still have a lot of difficulty with that politician's 100 billion figure, and am worried about the interests involved.  How did he come up with that number?  10,000 km circles continent 2 or 3 times, 2 Gw is a measurable percentage of entire MAIN generation forecast for 2010.

ref "ABB - for DC xmission + stations  2 Gw 300Km in $US = $325,000,000"

  I am also worried about plans to expand all future generation (e.g. 95%), with Natural Gas fired generation with no debate on how rational that is.

BP's world energy forecasts indicate N Gas proved reserves in trillion cuM.  
              1982        1992       2002       2003
USA           5.78       4.73          5.02     5.19
CANADA        2.75,       2.71,      1.69,      1.70
MEXICO        2.15       2.01          0.84     0.25
TOTAL N Amer 10.67       9.45          7.55     7.15

Consumption in billion cuM.
              1982        1992       2002       2003
USA           514.0       583.4      667.5       na
CANADA        53.7         71.7       80.7       na
MEXICO        29.0         28.9       42.1       na
TOTAL N Amer  596.7       684.0      790.3       na
 
Easy to see that as of 2002, proved reserves only come to 9.0 years consumption.  I know, proved reserves aren't all there is, but still i've seen realistic estimates of serious shortages by 2030.
That trend is going the wrong direction for my taste, given winters in canada.

I know about LNG ships, huge reserves in Russia but still, shouldn't new generation use a different fuel? Even oil, so much cheaper to transport by ship.  300 yr coal reserves.  I'd vote for rationally installed nd operated nukes.  Sure wind isn't going to do it, just a subsidy grab all it is so far.  Oceans?  Not even tested yet, wait 'till enviros get a look at it.  Solar so far costs more energy to manufacture than the cells ever generate.

Also seen several items make me suspicious about accuracy of "proved reserve" figures.

"In the late 1980s, OPEC countries added as much as 300 Gb to their reported “proved” reserves although only about 10 Gb were added from new discoveries." www.hubertpeak.com

This should probably be different thread, but it's late.

RE: Big blackout. What happened?

For anyone still following, this was from Reuters Sept 4, published on Energy Central Professional.


<quote>
"U.S. infrastructure needs seen at $1.6 trlllion"

"The American Society of Civil Engineers said the total five-year cost of the work needed on items from roads to drinking water systems and schools has surged to $1.6 trillion from the $1.3 trillion it estimated two years ago.

"The lights go out on Broadway. What can happen if America fails to invest in its infrastructure? Anything," the group warned in a statement.

Electricity industry experts blamed aging infrastructure for last month's cascading blackout in cities across the Midwest and Northeast United States and the Canadian province of Ontario
<quote>

Also carries this

<quote>
Public power companies served notice Monday that they are unlikely to sign off on proposals that would allow the operators of the electrical-grid system to increase their transmission charges to help finance infrastructure improvements.

The rate that the grids charge now will be adequate if the money is properly reinvested into the infrastructure, officials said.

The Federal Energy Regulatory Commission "allows a 12 percent rate of return on transmission charges, and that is quite adequate," said Alan Richardson, president of the American Public Power Association
<quote>


Also carries a discussion of heated debates among engineers at this weeks IEEE conference.

By Dan Piller, Fort Worth Star-Telegram, Texas -- Sept. 9
<quote>
The wrangling over who -- and what -- is responsible for the blackout dominated discussions among electrical engineers at the annual IEEE Transmission and Distribution conference at the Dallas Convention Center
<quote>


Guess i'm not the only one.

RE: Big blackout. What happened?

from http://pro.energycentral.com/professional/news/power/news_article.cfm?id=4122003

article Sept. 8 (Utility Spotlight)

<quote>Joe F. Colvin, president & CEO of the Nuclear Energy Institute, pressed in his response to the Committee for a "diverse portfolio of fuels and technologies" to provide flexibility and the ability to "balance economic and environmental considerations."

"Looking forward, we believe that lack of investment in our nation's critical energy and electric power infrastructure is a major problem," he added. "We are not investing enough in new base-load coal and nuclear plants and we are not investing enough in new electricity transmission."

He pointed to an EEI assessment showing that about 183,000 Mw of electric generating capacity in the U.S. is 30- to 40 years old and approximately 104,000 Mw is 40- to 50 years old. "That represents 35 percent of our 800,000 Mw of installed capacity, and is clear evidence that we are underinvesting - relying too much on old, less efficient, more polluting generating capacity and not investing in new, more efficient and cleaner facilities." In the area of electricity transmission, he added, investment has fallen by $115 million per year for the last 25 years, and transmission investment in 1999 was less than one-half of the level 20 years earlier "despite dramatic increases in the volumes of electricity being moved to market."

He cited another EEI study showing that simply maintaining transmission adequacy at is current levels would require a capital investment of $56 billion by 2010, equal to the book value of the existing transmission system.
<quote>

Somehow I get nervous with numbers like $56 billion.  Is that justified?

RE: Big blackout. What happened?

I stumbled across a paper by B.A. Carreras of Oak Ridge National Labs that was presented in January 2003 at an IEEE conference in Hawaii (International Conference on System Science).

http://www.computer.org/proceedings/hicss/1874/track2/187420065babs.htm

It speaks to probabilities of large blackouts due to efforts at mitigation.  I.e., "apparently sensible efforts to reduce the risk of smaller blackouts can sometimes increase the risk of larger blackouts."

RE: Big blackout. What happened?


It seems authors of the IEEE-ICSS paper might have since felt serendipitous and sickened at the same time, given recent events.
  

RE: Big blackout. What happened?

Hi all,

I have enjoyed reading all the opinions and theories but unfortunately most of them miss the mark.

It is quite clear that the Midwest ISO did not have a properly working information system during the event.  This lead to no response by the ISO to mitigate the outage.  Why there was a computer system problem needs to be investigated and why there wasn't enough redundancy built into the system is also an issue that needs some investigation.  Having said that, I believe that the black out could have been mitigated if the Midwest ISO operators had good information to work with.  Why other neighboring ISOs did not respond when they had working computer systems is another issue.  They may be more resposible for the event being as far reaching as it was.  Lets not forget the many ISOs that did respond correctly to the system instablilty and isolated themselves as they should have.

Another point is that was never mentioned is the fact that every day the grip operates correctly, isolating problems all over the country so that events like this are not a common occurance.  Unfortunately when the grid operates as designed, its not front page news. I don't think we need to panic over this blackout but there are many improvements that do need to be made, and I'm sure we will still have another blackout sometime in the future.  Just like cars that will always breakdown so will the electrical system.  We can only minimize the problem, we will never eliminate it no matter what we do. A perfect system by imperfect beings is impossible.

Lastly, even though others mentioned this, I found it remarkable that the grid was restored so quickly, and to my knowledge without any major injuries to the people responsible for putting it back together.  Cudos to those responsible for a job well done. (also not on the front page)

I thought I would end on a positive note.  Thanks for the opportunity to participate.

RE: Big blackout. What happened?

etronics,

You say that "I have enjoyed reading all the opinions and theories but unfortunately most of them miss the mark".

I have to disagree with you. This has been a superb thread ant the opinions expressed have covered a wide spectrum of possible causes and possible solutions.

..."why there wasn't enough redundancy built into the system is also an issue that needs some investigation"...

This is well known, that the investment in transmission infrastructure has not kept pace with the load growth. However, redundancy costs billions of dollars. The resulting increase in reliability may not be perceptible enough to consumers to justify their increased rates, which may double or triple. Furthermore, even with total redundancy you'd never be sure that another blackout will never happen. Human errors, acts of God, sabotage and other catastrophic events can occur and take out entire substations with cascading consequences.

"Why other neighboring ISOs did not respond when they had working computer systems is another issue".

We don't know that other ISO's had sufficient information early enough to take timely action. They would if this were a frequency problem, as frequency would be the same everywhere in the system. But it seems that this was a voltage problem, which can be highly localized until the cascading takes place, when it's too late to do anything.

I agree with your last 2-3 paragraphs.

RE: Big blackout. What happened?

To those interested, I'd like to see some of you more technical types pick up again on the technical discussions, particularly re. Gunnar Englund - state machine monitor back on Aug. 18.  I think it's not my place, Gunnar, but if you wanted you should start a new thread clearly marked "experts only" to persue further how a (relatively) small amount of money spent on [upgraded / improved / completely new] [concept controls systems and-or strategies / management decision tools / financial models / generation addition strategies] might enable the existing infrastructure to service requirements with fairly modest (e.g. as was in plans prior to blackout) additions to transmission capacity.  I think you might consider starting several threads, e.g.

1) Realtime digital monitor/modeler/predictive analyser for large transmission grid.
2) Neural net / trainable software / expert systems  predictive problem analyser (as operator assistant), how far ahead could it predict and with what reliability?  
3) Standardized grid changes analysis (using historical data from 1?), analyses proposed load, gen, line and switch adds / deletes, recommends relaying / settings, communications delay times, islanding schemes etc.

I'd sure follow something like that closely, and promise not to post there.

Should thread "Big Blackout. What Happened II" also be started?  Takes a long time to load this one now.

RE: Big blackout. What happened?

I agree with lengould, it is time to start some new sub-threads. With all due respect to the previous posters, we need to maintain a focus on the original problem statement of "...What Happened?"  As a power engineer, I'm less concerned with the 'coulda, woulda, shoulda' and more interested in gathering and analyzing the data that will help determine the actual sequence of events.  I'm certain that during the process of discovery we will all get a finer appreciation of how electricity is generated, transmitted, distributed, and controlled.  Armed with this fresh knowledge we will be in a much better position to influence the future.

RE: Big blackout. What happened?

This thread now jumps to the Big Blqackout-What Happended II thread at:

Thread238-70891

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members!


Resources