In all industries, it is paramount that the causes of equipment failure are fully understood so that future failures can be eliminated. In doing so, it’s also important that investigations proceed beyond simply identifying the cause as being due to human error.
Catastrophic failure
While walking around a mine process plant, I noticed that a cyclone feed pump discharge hose was looking old and sun-damaged and appeared about to fail. At this plant, the cyclone feed pumps were absolutely critical to production, with no stand-by pump available. Failure of one of these pumps (or hoses) would lead to large production losses.
As a consultant, I recommended to the maintenance team that they prepare a spare hose, position the crane and make sure the tools required to change the hose were readily available and in a useable condition. However, no-one really listened, or wanted to listen, and I wasn’t able to persuade them to follow my advice. The planners and reliability team explained that the hose was not due for changeout for a further 6 months.
That night, the very same cyclone feed pump discharge hose blew. Slurry covered everything within range and filled a hydraulic valve power unit and controls with slurry. The repair took 18 hours to complete. The loss of the hose also meant that production had to be cut while the hose was replaced.
I was then tasked with carrying out a root cause analysis investigation into the reason for the failure. I felt that, while no-one was saying it, no-one really wanted to hear the reason for the hose rupture. I was instructed to carry out a “5 whys” root cause analysis.
Conducting a “5 whys” root cause analysis
I went to the hose and inspected it. It appeared that the hose had failed through wear (as would usually be expected). However, upon further investigation, the hose appeared to have peeled from the bottom, almost as though the glue binding the hose layers together had given way. This had caused the slurry to come into contact with the outer skin (the part of the hose not designed to handle wear) and subsequently rupture.
I took photographs, discussed the failure with my peers, then decided to go and visit the warehouse. At the warehouse, I found a spare cyclone feed pump discharge hose sitting outside in the weather. One end of the hose was wrapped in plastic, giving it some protection from the elements. However, the other end was unprotected and had turned into a chalky, inflexible substance, as far away from black, flexible rubber as you could get.
I wrote my root cause analysis report, working through the 5-why’s steps and came to the conclusion that the hose failure was due to incorrect storage. Given that I had used this manufacturer’s hose previously on multiple other sites, I suspected it wasn’t a manufacturing fault as I had never seen similar delamination before. I gave the 5-why’s report to the manager and received a reprimand for writing a “nonsense” root cause analysis report, because according to the manager, the failure was due to manufacturing error from the original equipment manufacturer.
Once the 18-hour unplanned downtime had been completed and the plant was up and running again, I decided to dig further into the cause of the hose failure.
Consulting the original equipment manufacturer
Firstly, I went into the computerised maintenance management system (CMMS) and found that the last hose had been delivered from the manufacturer 7 years previously. I queried the hose manufacturer and discovered that no new hoses had been ordered for 7 years, so we were definitely running on old stock. I requested that the hose manufacturer come to site and inspect the hose, which they agreed to.
I took the hose manufacturer’s representatives to the area where the blown hoses were put. They looked at the failed hoses kept there, all of which had failed through wear. They saw the hose in question and, like me, noted that the hose had de-laminated.
I then took the manufacturer’s representatives to the warehouse, where they saw the chalky, inflexible degrading rubber on the end of the hose. Immediately, they pointed to this and confirmed my suspicions that the premature failure of the hose was caused by the way the hose was being stored.
I re-visited my original Root Cause Analysis investigation, adding in the visit by the hose manufacturer and their comments. This time the manager accepted the report without question.
What can we learn from this?
So what, I hear you ask, is the point of the above story? During my time at the mine, I witnessed “5 whys” being carried out and accepted as “3 whys”, clearly not reaching the final root cause. There seemed to be something not quite right in the way root cause analysis was being carried out. And if the root cause analysis had not been carried out properly, the root cause for the hose failure was unlikely to have been found.
- The hose had burst, the same as any other worn-out hose.
- People were quick to point at the most obvious failure (the one that suited their purpose the most).
- People were quick to trust the reliability team and their assertion that the hose wasn’t due to be changed out for 6 months.
- No-one actually went to the hose and saw the internal delamination.
- No-one went to the warehouse to see how the hoses were stored.
The likelihood is that the spare hose would have been fitted with an expectation that it would reach its normal life span before needing replacement. No new hoses would have been ordered or be available in the warehouse to cover for the inevitable infant mortality it would suffer due to the sun/ weather damage. The financial impact for this would have been huge.
A lesson for the future
It is imperative that a root cause analysis investigation is carried out to its maximum depth, whether it takes the form of a “5-whys”, “failure mode and effect analysis” or a “fishbone” diagram. Stopping at human error, or at the obvious component failure level, is unlikely to provide learnings which can be used to eliminate future failures of the same type.
If you would like help to improve the quality of your root cause analysis investigations, we provide in-person and online training in best practice root cause analysis. We can also support you as consultants if you prefer – see the service pages below for case studies and more information.