Emergency Stop button not working
Emergency Stop button not working
(OP)
There's this saw mill where the safety is built around gates, light barriers and quite a few red mushroom style emergency buttons that are connected via several AS-interface buses, around ten of them.
We were asked to go and have a look at the mill. There were a multitude of reasons for that. One of the main reasons was that there were a lot of nuisance stops. We traced the stops to the AS-i buses and the interference level present on them (PWM drives in hundreds) with motor cables running parallel to the yellow bus cable. The yellow cable was in separate and well grounded trays, but the interference was still quite high. After connecting the GND terminal of the Masters to ground (an oversight from the panel builder and commissioner), the interference level was a lot lower and the bus worked a lot better.
Then, we noticed that there were Emergency Stops that didn't work at all. They had been crossed out in the PLC program, but left for anyone to see and press in the plant. Someone had given up and just left the saw mill with the Emergency Stops in non-working order. There wasn't even a sign or a note about the non-functioning devices.
After this long introduction, the question is short and simple: On a scale running from Stupid via Careless and Irresponsible to Criminal - where would you put such a behaviour?
The second question is also simple: What about OSHA and legislation? What about laws in EU? I have never seen anything like this before and welcome any view on the matter, be it emotional, factual or whatever.
We were asked to go and have a look at the mill. There were a multitude of reasons for that. One of the main reasons was that there were a lot of nuisance stops. We traced the stops to the AS-i buses and the interference level present on them (PWM drives in hundreds) with motor cables running parallel to the yellow bus cable. The yellow cable was in separate and well grounded trays, but the interference was still quite high. After connecting the GND terminal of the Masters to ground (an oversight from the panel builder and commissioner), the interference level was a lot lower and the bus worked a lot better.
Then, we noticed that there were Emergency Stops that didn't work at all. They had been crossed out in the PLC program, but left for anyone to see and press in the plant. Someone had given up and just left the saw mill with the Emergency Stops in non-working order. There wasn't even a sign or a note about the non-functioning devices.
After this long introduction, the question is short and simple: On a scale running from Stupid via Careless and Irresponsible to Criminal - where would you put such a behaviour?
The second question is also simple: What about OSHA and legislation? What about laws in EU? I have never seen anything like this before and welcome any view on the matter, be it emotional, factual or whatever.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.





RE: Emergency Stop button not working
I don't understand how you can override some stop buttons and not all in a PLC, normally they are wired in series to one input, but since they are crossed out in program, the manufacturer designed them to work and most likely some careless Tech under pressure for production jury rigged the machine. Unethical, criminal, and careless for sure.
In the US by OSHA rules I believe, Employees are required to report safety issues and can refuse to work under unsafe conditions, report the condition to OSHA for immediate investigation and correction.
RE: Emergency Stop button not working
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Emergency Stop button not working
My very first engineering job was investigating a project that went horribly bad and resulted in lawsuits. The project was for an OEM of cardboard compactors, my new employer had (prior to my joining) sold a control panel to them, but with no wiring, they just mounted the components because the OEM wanted to save money by wiring them himself. He somehow managed to wire the power leads feeding the auger hydraulic pump from the wrong side (line side) of the main disconnect. So true to Murphy's Law, a grocery store employee was tossing boxes into the hopper to be compacted, saw something he wanted, reached in to rescue it and got his sleeve caught in the auger. It was moving slowly so his screams for help got someone there on time, who pulled the disconnect to kill power. Unfortunately because of the wiring error, the hydraulic pump was NOT stopped and it tore the guys arm off. Stupid cannot do justice to the tragedy as it related to the guy who willfully disregarded simple safety protocols, the compactor OEM eventually was jailed for reckless endangerment. Stupid turned out to be my employer selling a "control panel" with their name and UL label on it without having done the actual wiring design or fabrication. The lawyers went after the deepest pockets.
"Will work for (the memory of) salami"
RE: Emergency Stop button not working
I wouldn't be happy with a comms-based ESD system, SIL or otherwise. Call me old fashioned.
RE: Emergency Stop button not working
RE: Emergency Stop button not working
What was the response of the controller to loss of comms from an E-Stop? I would expect that if a device was polled by the controller and it failed to respond within the expected time then the controller should fail safe, i.e. initiate a trip. Was this the source of the nuisance tripping?
Does the network architecture allow you to use concentrators or hubs to gather a number of E-Stops in physical proximity to each other and then connect the hubs back to the controller by optical fibre? That might give you a chance of controlling the interference by breaking the network into smaller galvanically isolated elements. I've done this using other network types when I have had very bad interference problems but I know next to nothing about AS-i bus. I'm dimly aware that it might be a Siemens product.
RE: Emergency Stop button not working
Keith Cress
kcress - http://www.flaminsystems.com
RE: Emergency Stop button not working
One could see it as an excusable "slight oversight", but I cannot do that; one slight oversight, two oversights, gross ignorance... The problem is known from nuclear power plants and oil fields - with known consequences. There is no place to draw the line other than below zero oversight.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Emergency Stop button not working
Commonly, programmers today have never been in the real world and seen what happens when things go wrong.
Recently had an argument with a bunch of programmers who maintained that fail-safe meant that the motors kept running if the comms was lost. Reason, if the motor happened to be in a tunnel and there was a fire that burnt through the comms cable, then it may be important that the fans kept running, so that was fail safe!!
Would not accept that there were fire mode specs that were separate to fail safe operation expectations.
Gunnar, I would get someone to check that the motor cables were a) screened and b) the screens are correctly clamp terminated at each end. This usually sorts these types of problems.
Best regards,
Mark.
Mark Empson
Advanced Motor Control Ltd
RE: Emergency Stop button not working
RE: Emergency Stop button not working
I am presently on a large project which will soon be going online. There are about 1000 electricians on site. The LOTO (Lock Out/ Tag Out) procedures are rigorous. Everything is documented. There is a paper trail to prove conclusively that a circuit is safely locked out before work starts.
Unfortunately there is no procedure in place to verify that the CORRECT circuit has been locked out.
I have been suggesting that point to point testing be mandatory as part of the LOTO plan.
Management doesn't feel that point to point testing is required.
I have seen several LOTO violations.Some have been serious violations.
Just one example. There have been others.
A crew was about to work on a panel and went through the LOTO procedure correctly. They then went to the panel and fortunately did a test before touch. The panel was hot! 600 Volts. The panel had been energized by a temporary feed.
Discussion at a safety meeting.
Me;
"We were very lucky. Suppose that the temporary feed had been switched off when the crew did the test before touch test. They would have started work and the current may have been turned on at any time."
The voice of hubris;
"Well any time that you don't feel safe, you have the right to decline the work." (Ya right, that would be a good career move in this culture!)
"I am not worried about myself. I am worried about the younger, less experienced workers. And Sir, I am worried about your liability position should someone be injured due to a flawed LOTO system .
Please consider, had that circuit been switched off and the men started work, and then the power came on and someone was killed. You may be in more liability than you care to think about."
End of meeting.
If I was higher up the food chain that would have been my last day on that site!
The head of our companies safety division on this site supported me. He has been transferred to another site.
I'll probably be laid off eventually if I don't shut up.
It won't be the first time.
The last time it happened, I was exonerated by the grievance procedure and at least one high placed supervisor lost his job.
I was luckier than Cass that time.
Bill
--------------------
"Why not the best?"
Jimmy Carter
RE: Emergency Stop button not working
You LOTO process is NOT safer by requiring that cables and wires be physically traced. The ONLY way to verify the panel is de-energized is to open the panel wearing the proper PPE using the proper. Then, once within the panel, you MUST use prudent safeguards and procedures to verify by probe and voltmeter that the leads themselves are dead.
It is physically IMPOSSIBLE to trace cables from a panel back to the trays and back to their assumed power source.
RE: Emergency Stop button not working
You wrote: "I have been suggesting that point to point testing be mandatory as part of the LOTO plan. Management doesn't feel that point to point testing is required."
The term 'point to point testing' I have always applied to field annunciation testing [except we call it end-to-end testing], where each alarm point is tested one at a time to confirm its functionality, something that our company's management also seems to no longer be willing to commit the resources to, incidentally...
But racookpe1978 speaks of "tracing cables"...
What exactly do you understand 'point to point testing' to mean?
CR
"As iron sharpens iron, so one person sharpens another." [Proverbs 27:17, NIV]
RE: Emergency Stop button not working
I agree with you on the futility of following cables. Antone who suggests it as a safety measure is displaying a complete and dangerous lack of field experience.
Sometimes a cables is "walked" as a last resort to try to locate which field device the cable feeds. Any one who has done this in a large plant knows how difficult and misleading this method is, not to say time consuming. After following a cable for a couple of hundred feet to a field device, a continuity check often fails. Somewhere you have lost the cable on a corner or in a bundle and have been following the wrong cable.
By Point to Point testing I mean a continuity test from end to end of the circuit. Two workers with communication. On the command of one worker, the second worker at the other end of the circuit makes and breaks continuity, usually three times. Some conversation is encouraged to make the intervals between make and break and repeat more random.
This is the basic test, there are refinements to deal with special cases.
Just checking for voltage is a must before the point to point test, but does not prove that the correct circuit is locked out.
I worked in one plant where the prints were not dependable. There were hundreds of electric heat trace circuits, all on thermostatic control. It was fairly common for workers to turn off the circuit indicated by the prints and then find that the field junction box was still energized. Bad.
Sometimes the field junction box was dead, but in a short time the thermostat would cycle on and the junction box that may have been assumed to be dead was now hot. WORSE.
If you have voltage, you can prove the circuit by switching the breaker or control on and off several times. Still two men with radios and a meter.
I have been involved in a few trouble shooting sessions assisting in tracing out circuits in order to do a safe and proven safe lockout when the prints were in error.
Bill
--------------------
"Why not the best?"
Jimmy Carter
RE: Emergency Stop button not working
Just my two cents.
RE: Emergency Stop button not working
Safety is not only for electricians. And, if you have a high powered DOL motor - there will be enough fireworks to cause injuries when the clamp is applied.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Emergency Stop button not working
RE: Emergency Stop button not working
Re Chaining;
Chaining is commonly used here for higher voltages. Generally above 4160 Volts. When there is a possibility of a circuit being energized from more than one source grounding jumpers or "Ground chains" are used for added protection.
At 480 volts and 600 Volts ground chains are never used.
Bill
--------------------
"Why not the best?"
Jimmy Carter
RE: Emergency Stop button not working
No amount of standards and documentation purporting Safety Integrity Level can change that.
This is a perfect example. Because all it took to violate the SIL was for the on-site commissioning engineer or a plant engineer to hack some of the code out.
Where is that documented in the SIL certification process?
RE: Emergency Stop button not working
I totally disagree with that: "Safety and Software do not and can not ever go together."
When that engineer is done with all the estop software I have seen, its all locked down, and if someone should go in there and change it usually the system will fault out, if they do not have the key.
So, any estop can be bypassed either hardwired or software, its up to that engineer to commission it correctly, leave it in a state where it operates to that SIL level, and train the people left behind, maintenance, on how to maintain it.
RE: Emergency Stop button not working
Of course, hard-wired circuits are even easier to by-pass or override. But that is usually also easier to detect and restore.
The problem in this case is that there's no "safety culture" at all. OK, there is one - you will be reported if you are seen without hard hat and hi-vis vest. But that is all "safety culture" there is.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Emergency Stop button not working
RE: Emergency Stop button not working
RE: Emergency Stop button not working
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Emergency Stop button not working
"Safety as a weapon!"
Bill
--------------------
"Why not the best?"
Jimmy Carter
RE: Emergency Stop button not working
Really? All it takes to bypass a hard-wired E-stop is a screwdriver and a few seconds of time.
RE: Emergency Stop button not working
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Emergency Stop button not working
Today we just had another "near miss" with a live cable that should have been de-energized.
Bill
--------------------
"Why not the best?"
Jimmy Carter
RE: Emergency Stop button not working
The report had a section on how to best handle safety, saying that the Safety Officer should initialize a thorough test of all interlocks and emergency stops, that there should be a detailed protocol that the Safety Officer should sign and that this protocol should be available to the workforce (be put on the company bill-board).
Thursday last week, I got a phone call from our customer. He thanked for work done and told me that the complete report (including lots of other problems with bus interference, motor insulation, bearings etcetera) had been sent to the mill and, it seems, is now available for all (operators, maintenance, bean counters, management) to read.
That is very satisfying and I think that "we take safety seriously" is a little more than just a saying in this company. They just needed the heads-up. The outcome surprises me, but I am glad it worked.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.