×
INTELLIGENT WORK FORUMS
FOR ENGINEERING PROFESSIONALS

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

Emergency Stop button not working
10

Emergency Stop button not working

Emergency Stop button not working

(OP)
There's this saw mill where the safety is built around gates, light barriers and quite a few red mushroom style emergency buttons that are connected via several AS-interface buses, around ten of them.

We were asked to go and have a look at the mill. There were a multitude of reasons for that. One of the main reasons was that there were a lot of nuisance stops. We traced the stops to the AS-i buses and the interference level present on them (PWM drives in hundreds) with motor cables running parallel to the yellow bus cable. The yellow cable was in separate and well grounded trays, but the interference was still quite high. After connecting the GND terminal of the Masters to ground (an oversight from the panel builder and commissioner), the interference level was a lot lower and the bus worked a lot better.

Then, we noticed that there were Emergency Stops that didn't work at all. They had been crossed out in the PLC program, but left for anyone to see and press in the plant. Someone had given up and just left the saw mill with the Emergency Stops in non-working order. There wasn't even a sign or a note about the non-functioning devices.

After this long introduction, the question is short and simple: On a scale running from Stupid via Careless and Irresponsible to Criminal - where would you put such a behaviour?

The second question is also simple: What about OSHA and legislation? What about laws in EU? I have never seen anything like this before and welcome any view on the matter, be it emotional, factual or whatever.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working


I don't understand how you can override some stop buttons and not all in a PLC, normally they are wired in series to one input, but since they are crossed out in program, the manufacturer designed them to work and most likely some careless Tech under pressure for production jury rigged the machine. Unethical, criminal, and careless for sure.

In the US by OSHA rules I believe, Employees are required to report safety issues and can refuse to work under unsafe conditions, report the condition to OSHA for immediate investigation and correction.

RE: Emergency Stop button not working

(OP)
No, they are not at all wired in series. Have a look at Jokab Pluto. Every button has a unique address in the PLC program and this is (SW)SIL stuff. So, the behaviour is really very remarkable. The buyer wanted maximum safety and he got none at all. At least in some parts of the mill.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working

It's stupid, until someone is injured or killed, at which time it becomes criminal.

My very first engineering job was investigating a project that went horribly bad and resulted in lawsuits. The project was for an OEM of cardboard compactors, my new employer had (prior to my joining) sold a control panel to them, but with no wiring, they just mounted the components because the OEM wanted to save money by wiring them himself. He somehow managed to wire the power leads feeding the auger hydraulic pump from the wrong side (line side) of the main disconnect. So true to Murphy's Law, a grocery store employee was tossing boxes into the hopper to be compacted, saw something he wanted, reached in to rescue it and got his sleeve caught in the auger. It was moving slowly so his screams for help got someone there on time, who pulled the disconnect to kill power. Unfortunately because of the wiring error, the hydraulic pump was NOT stopped and it tore the guys arm off. Stupid cannot do justice to the tragedy as it related to the guy who willfully disregarded simple safety protocols, the compactor OEM eventually was jailed for reckless endangerment. Stupid turned out to be my employer selling a "control panel" with their name and UL label on it without having done the actual wiring design or fabrication. The lawyers went after the deepest pockets.

"Will work for (the memory of) salami"

RE: Emergency Stop button not working

Somewhere at the reckless and dangerous end of the spectrum. The designer and the commissioning guys should have some awkward questions to answer, but not half as awkward as those that the facility operator faces if someone is hurt or killed.

I wouldn't be happy with a comms-based ESD system, SIL or otherwise. Call me old fashioned.

RE: Emergency Stop button not working

Asi Estop, my take on Comms based estop networks is that basically for them to work they have to be separate network from the controls. The people I talked with said you can do it with the controls network but you have to do a bunch of checks to ensure the sil level. I came to conclusion that creating an estop comms network was the only way to ensure it worked correctly. This went for the AB type Ethernet or devicenet estop type networks. I know profinet was sold this way too but again don't believe the reps when they say you can have both control devices and estop sil devices on same network.

RE: Emergency Stop button not working

skogs,

What was the response of the controller to loss of comms from an E-Stop? I would expect that if a device was polled by the controller and it failed to respond within the expected time then the controller should fail safe, i.e. initiate a trip. Was this the source of the nuisance tripping?

Does the network architecture allow you to use concentrators or hubs to gather a number of E-Stops in physical proximity to each other and then connect the hubs back to the controller by optical fibre? That might give you a chance of controlling the interference by breaking the network into smaller galvanically isolated elements. I've done this using other network types when I have had very bad interference problems but I know next to nothing about AS-i bus. I'm dimly aware that it might be a Siemens product.

RE: Emergency Stop button not working

I suspect Scotty is right and so the 'culprit' had to disable those various E-stops to get the plant mostly moving. Still mega-stupid.

Keith Cress
kcress - http://www.flaminsystems.com

RE: Emergency Stop button not working

(OP)
I simplified the situation a bit. The colleague that does this part of the investigation would probably express himself more correctly. The Jokab/ABB system is a very good one and it is built with an architecture very similar to what Scotty describes when saying "concentrators or hubs to gather a number of E-Stops in physical proximity to each other and then connect the hubs back to the controller by optical fibre". The problem here is that someone didn't realise that a non-functioning Emergency Stop is a violation of the safety thinking that is prevalent in the mill and that is expressed in the Machinery Directive 2006/42/EC and the Control systems safety standards EN ISO 13849-1 and EN 62061.

One could see it as an excusable "slight oversight", but I cannot do that; one slight oversight, two oversights, gross ignorance... The problem is known from nuclear power plants and oil fields - with known consequences. There is no place to draw the line other than below zero oversight.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working

I would classify this a b... dangerous and pull the fuses until it was fixed!!
Commonly, programmers today have never been in the real world and seen what happens when things go wrong.
Recently had an argument with a bunch of programmers who maintained that fail-safe meant that the motors kept running if the comms was lost. Reason, if the motor happened to be in a tunnel and there was a fire that burnt through the comms cable, then it may be important that the fans kept running, so that was fail safe!!
Would not accept that there were fire mode specs that were separate to fail safe operation expectations.
Gunnar, I would get someone to check that the motor cables were a) screened and b) the screens are correctly clamp terminated at each end. This usually sorts these types of problems.
Best regards,
Mark.

Mark Empson
Advanced Motor Control Ltd

RE: Emergency Stop button not working

Beyond the fact that what you describe runs somewhere between careless and criminal, I would also think the plant has a poor safety culture if they were operating without the E-stop buttons operational. Did this plant spend a whole bunch of money to implement a fancy new safety system but not be bothered to spend the time to create the proper safety procedures and ensuring their employees understand that safety is a top priority? Part of the safety procedures or safety culture should have been testing the E-Stop and other safety systems on a regular basis. The best safety systems are no good if they're not embraced by the employees.

RE: Emergency Stop button not working

I feel your pain Gunnar. If someone is injured or killed, then someone may go to jail, but probably not until the damage is done.
I am presently on a large project which will soon be going online. There are about 1000 electricians on site. The LOTO (Lock Out/ Tag Out) procedures are rigorous. Everything is documented. There is a paper trail to prove conclusively that a circuit is safely locked out before work starts.
Unfortunately there is no procedure in place to verify that the CORRECT circuit has been locked out.
I have been suggesting that point to point testing be mandatory as part of the LOTO plan.
Management doesn't feel that point to point testing is required.
I have seen several LOTO violations.Some have been serious violations.
Just one example. There have been others.
A crew was about to work on a panel and went through the LOTO procedure correctly. They then went to the panel and fortunately did a test before touch. The panel was hot! 600 Volts. The panel had been energized by a temporary feed.
Discussion at a safety meeting.
Me;
"We were very lucky. Suppose that the temporary feed had been switched off when the crew did the test before touch test. They would have started work and the current may have been turned on at any time."
The voice of hubris;
"Well any time that you don't feel safe, you have the right to decline the work." (Ya right, that would be a good career move in this culture!)
"I am not worried about myself. I am worried about the younger, less experienced workers. And Sir, I am worried about your liability position should someone be injured due to a flawed LOTO system .
Please consider, had that circuit been switched off and the men started work, and then the power came on and someone was killed. You may be in more liability than you care to think about."
End of meeting.
If I was higher up the food chain that would have been my last day on that site!
The head of our companies safety division on this site supported me. He has been transferred to another site.
I'll probably be laid off eventually if I don't shut up.
It won't be the first time.
The last time it happened, I was exonerated by the grievance procedure and at least one high placed supervisor lost his job.
I was luckier than Cass that time.

Bill
--------------------
"Why not the best?"
Jimmy Carter

RE: Emergency Stop button not working

It's counterintuitive, but you CANNOT verify a 'typical, real-world" power panel by tracing cables.

You LOTO process is NOT safer by requiring that cables and wires be physically traced. The ONLY way to verify the panel is de-energized is to open the panel wearing the proper PPE using the proper. Then, once within the panel, you MUST use prudent safeguards and procedures to verify by probe and voltmeter that the leads themselves are dead.

It is physically IMPOSSIBLE to trace cables from a panel back to the trays and back to their assumed power source.

RE: Emergency Stop button not working

Hey Bill,

You wrote: "I have been suggesting that point to point testing be mandatory as part of the LOTO plan. Management doesn't feel that point to point testing is required."

The term 'point to point testing' I have always applied to field annunciation testing [except we call it end-to-end testing], where each alarm point is tested one at a time to confirm its functionality, something that our company's management also seems to no longer be willing to commit the resources to, incidentally...

But racookpe1978 speaks of "tracing cables"...

What exactly do you understand 'point to point testing' to mean?

CR

"As iron sharpens iron, so one person sharpens another." [Proverbs 27:17, NIV]

RE: Emergency Stop button not working

racookpe1978;
I agree with you on the futility of following cables. Antone who suggests it as a safety measure is displaying a complete and dangerous lack of field experience.
Sometimes a cables is "walked" as a last resort to try to locate which field device the cable feeds. Any one who has done this in a large plant knows how difficult and misleading this method is, not to say time consuming. After following a cable for a couple of hundred feet to a field device, a continuity check often fails. Somewhere you have lost the cable on a corner or in a bundle and have been following the wrong cable.
By Point to Point testing I mean a continuity test from end to end of the circuit. Two workers with communication. On the command of one worker, the second worker at the other end of the circuit makes and breaks continuity, usually three times. Some conversation is encouraged to make the intervals between make and break and repeat more random.
This is the basic test, there are refinements to deal with special cases.

Just checking for voltage is a must before the point to point test, but does not prove that the correct circuit is locked out.
I worked in one plant where the prints were not dependable. There were hundreds of electric heat trace circuits, all on thermostatic control. It was fairly common for workers to turn off the circuit indicated by the prints and then find that the field junction box was still energized. Bad.
Sometimes the field junction box was dead, but in a short time the thermostat would cycle on and the junction box that may have been assumed to be dead was now hot. WORSE.
If you have voltage, you can prove the circuit by switching the breaker or control on and off several times. Still two men with radios and a meter.
I have been involved in a few trouble shooting sessions assisting in tracing out circuits in order to do a safe and proven safe lockout when the prints were in error.

Bill
--------------------
"Why not the best?"
Jimmy Carter

RE: Emergency Stop button not working

I maybe late in chiming-in but don't you people use ground clamps prior to working a LOTO-approved task?
Just my two cents.

RE: Emergency Stop button not working

(OP)
Very seldom seen it done in practice. And never by mechanical guys that are doing adjustments and repair.

Safety is not only for electricians. And, if you have a high powered DOL motor - there will be enough fireworks to cause injuries when the clamp is applied.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working

Quote (Skogsgura)

Very seldom seen it done in practice. And never by mechanical guys that are doing adjustments and repair.
IIRC, mechanical guys use blocking and releasing of stored energy as part of the LOTO in the same manner electricians use grounding and chaining to compliment the installation of blinds, restraining barriers, etc. Bluntly, a fool-proof LOTO could help a lot.

RE: Emergency Stop button not working

My issue is the lack of verification that the correct switch has been locked out.

Re Chaining;
Chaining is commonly used here for higher voltages. Generally above 4160 Volts. When there is a possibility of a circuit being energized from more than one source grounding jumpers or "Ground chains" are used for added protection.
At 480 volts and 600 Volts ground chains are never used.

Bill
--------------------
"Why not the best?"
Jimmy Carter

RE: Emergency Stop button not working

Safety and Software do not and can not ever go together.

No amount of standards and documentation purporting Safety Integrity Level can change that.

This is a perfect example. Because all it took to violate the SIL was for the on-site commissioning engineer or a plant engineer to hack some of the code out.

Where is that documented in the SIL certification process?

RE: Emergency Stop button not working

MintJulep

I totally disagree with that: "Safety and Software do not and can not ever go together."

When that engineer is done with all the estop software I have seen, its all locked down, and if someone should go in there and change it usually the system will fault out, if they do not have the key.

So, any estop can be bypassed either hardwired or software, its up to that engineer to commission it correctly, leave it in a state where it operates to that SIL level, and train the people left behind, maintenance, on how to maintain it.

RE: Emergency Stop button not working

(OP)
IRL, CD. IRL - that seems to be kind of an impossible dream.

Of course, hard-wired circuits are even easier to by-pass or override. But that is usually also easier to detect and restore.

The problem in this case is that there's no "safety culture" at all. OK, there is one - you will be reported if you are seen without hard hat and hi-vis vest. But that is all "safety culture" there is.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working

2
That's not a safety culture, that's safety theater.

RE: Emergency Stop button not working

Ha-ha, that comment is noted for future use. smile

RE: Emergency Stop button not working

(OP)
Agree. Absolutely!

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working

I spent a short time working for a contractor some years ago where the field workers often repeated the phrase:
"Safety as a weapon!"

Bill
--------------------
"Why not the best?"
Jimmy Carter

RE: Emergency Stop button not working

Quote (MintJulep)

This is a perfect example. Because all it took to violate the SIL was for the on-site commissioning engineer or a plant engineer to hack some of the code out.

Really? All it takes to bypass a hard-wired E-stop is a screwdriver and a few seconds of time.

RE: Emergency Stop button not working

(OP)
Not quite so - there's seldom a SIL level associated with a hard-wired emergency stop that can bypassed that easily.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

RE: Emergency Stop button not working

I had a talk with the head of safety for our company a few days ago. Corporate hubris, corporate compliance, (don't rock the boat), attack the messenger, denial. Felt like making water into a very strong wind.
Today we just had another "near miss" with a live cable that should have been de-energized.

Bill
--------------------
"Why not the best?"
Jimmy Carter

RE: Emergency Stop button not working

(OP)
I finished the report about one week ago and mailed it to our customer (the company that built the line).

The report had a section on how to best handle safety, saying that the Safety Officer should initialize a thorough test of all interlocks and emergency stops, that there should be a detailed protocol that the Safety Officer should sign and that this protocol should be available to the workforce (be put on the company bill-board).

Thursday last week, I got a phone call from our customer. He thanked for work done and told me that the complete report (including lots of other problems with bus interference, motor insulation, bearings etcetera) had been sent to the mill and, it seems, is now available for all (operators, maintenance, bean counters, management) to read.

That is very satisfying and I think that "we take safety seriously" is a little more than just a saying in this company. They just needed the heads-up. The outcome surprises me, but I am glad it worked.

Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members!


Resources