What Can Go Wrong, Will Go Wrong: Quality Case Studies
Quality is not simply a theoretical idea about how to cut costs and make better products and services. Quality is about anticipating the consequences of errors, carelessness, and inefficiency, and putting processes in place to ensure they don’t occur. For most organizations, errors can cost time and money, which can have a significant impact on the financial bottom line and brand reputation. In some situations, the consequences of error can be catastrophic, resulting in loss of life, environmental damage, and extensive financial liability.
The purpose here is to:
- provide some examples of how human tragedy and catastrophic financial damage can be the result of Quality failures, and
- provide examples of how rigorous responses to Quality failures can mitigate damage and prevent future occurrences.
Readers learning about the fundamental principles of Quality and the impact they can have on an organization will benefit from these case studies as they learn the value of paying attention to the details of their processes in their Quality Management projects.
On April 20, 2010, the Deepwater Horizon, an oil rig leased to BP Exploration & Production from Transocean for extracting oil from the Macondo well 50 miles off the coast of Louisiana, suffered a series of explosions caused by the uncontrolled flow and ignition of oil from the well onto the rig platform during exploratory drilling. Explosions, and the resulting inextinguishable fire, continued to rock the Deepwater Horizon for 36 hours, killing 11 workers and injuring 17 others before the rig sank on April 22 in waters approximately 5,000 feet deep. The Macondo well continued to vent oil for 87 days before it was capped. The result was the discharge of almost 5 million barrels of oil into the fragile ecosystem of the Gulf of Mexico, making this the largest marine oil spill in the history of the petroleum industry and an unprecedented environmental disaster.
The engineering assessment of the failures that led to the explosion is complex.i During the exploratory drilling, workers pumped cement down the well to create a casing that would, in conjunction with a blowout preventer (BOP), prevent the pressure from the well from forcing the oil back up the drill and onto the platform. According to the Deepwater Horizon Accident Investigation Report, a series of cascading equipment failures allowed a blowout that forced a mixture of oil and gas onto the oil platform and into the engine rooms, which was the likely point of ignition.ii
Numerous post-incident documents report that many of the failures that led to the disaster were based on shortcomings in Quality planning. In particular, the BP Macondo well team neglected to follow the best practice of performing Quality assurance and risk assessment of the design and testing of the cement casing, the integrity of which failed to prevent the initial escape of oil and gas that led to the blowout.iii Subsequent investigative journalism and testimony paints a picture in which the Culture of Quality was subordinated to cost-cutting and procedural shortcuts to try and get the drilling project, which was five weeks behind schedule, back on track. These shortcuts included replacing a mixture of heavy mud, which was pumped into the well to reduce pressure, with a lighter, cheaper, and less effective mixture of seawater because of concerns about cost overruns, which failed to provide the necessary protection against blowouts.iv Post-disaster interviews with survivors indicate that workers were concerned with BP and Transocean officials’ diminishing concern for Culture of Quality, but feared reprisals if they discussed or attempted to address it.v
The comparison of the Cost of Good Quality (COGQ) to the total damage of the disaster makes for sobering reading. Testing the cement used in the casing would have taken 10 hours of time and cost $128,000. Neglecting that minimal Cost of Prevention resulted in the loss of 11 lives, $11 billion in financial damage to BP, inestimable environmental damage to the Gulf of Mexico, and long-term association of the BP brand with a tragic disaster of monumental significance.
The Executive Summary of the Deepwater Horizon Accident Investigation Report summarizes well the vital importance of both a systemic Quality Management program and a Culture of Quality in an organization, as well as the terrible human and environmental consequences of neglecting this responsibility:
“Through a series of rig audit findings and maintenance reports, the investigation team found indications of potential weaknesses in the testing regime and maintenance management system for the BOP. The team did not identify any single action or inaction that caused this accident. Rather, a complex and interlinked series of mechanical failures, human judgements, engineering design, operational implementation and team interfaces came together to allow the initiation and escalation of the accident. Multiple companies, work teams and circumstances were involved over time.”vi
In early 2014, the International Council on Clean Transportation began working with researchers at West Virginia University’s Center for Alternative Fuels, Engines, and Emissions (CAFEE) to follow up on reported discrepancies in the emissions of Volkswagen diesel vehicles. The California Air Resources Board (CARB) had previously subjected the vehicles to rigorous emissions testing in the laboratory, and the vehicles had all passed with no indication of any problems. The CAFEE researchers did their emissions test in the field and produced some very different results.vii
The researchers discovered that when operating in the real world, the vehicles produced emissions that fell far outside the limits allowed for diesel vehicles to be certified in the United States. After more testing, the researchers discovered a sophisticated software application that used environmental data from the vehicle, such as the absence of movement from the steering wheel, to determine when the vehicle was being tested in a laboratory and when it was operating in the real world. If the software determined the vehicle was undergoing testing, it would turn on the emission control to pass the test. If the software determined the vehicle was operating in the field, it would turn off the emission control, which would increase emissions beyond acceptable levels but would also greatly improve fuel economy and vehicle performance.
After initial attempts at denial, Volkswagen eventually admitted to having knowingly created and installed this software application as a way of intentionally deceiving the testing mechanisms of regulators around the world between 2009 and 2015, affecting about 11 million vehicles.viii
As a consequence of this deception, Volkswagen announced plans to spend $18.32 billion on fixing the issue.ix In January 2017, Volkswagen pleaded guilty to criminal charges and agreed to pay a fine of $2.8 billion. Additional criminal charges are pending against seven former employees, several of whom are executives and two of whom were directly responsible for Quality.
Dieselgate is an example not of the failure of a dysfunctional system but of willful manipulation of that system and a violation of the principles of a Culture of Quality. By all accounts, Volkswagen had a sterling reputation for Quality throughout the organization, and yet it has been reported that several people at the management level had been aware of the deception for several years.x The Dieselgate scandal is far from over and will continue to provide a warning about the vulnerability of the Culture of Quality and responsibility to contrary priorities of profit and personal gain.
NASA Challenger Disaster
On January 28, 1986, the Space Shuttle Challenger was destroyed 73 seconds after lifting off from Cape Canaveral, Florida. Seven crew members died, a $3 billion-dollar orbital vehicle was lost, and NASA’s Space Shuttle program was suspended for 32 months.
The official cause of the disaster was the failure of an O-ring to prevent hot gases from leaking through the joint in the solid rocket motor during launch.xi The Rogers Commission –the body tasked with investigating the disaster – found that the O-ring design had been a point of concern for several years prior to the disaster, but that any concerns had been either poorly communicated or ignored in favor of maintaining project delivery on-time and on-budget.xii
In addition to the faulty initial design of the O-rings, the Commission determined that the unusually cold temperatures at the time of the launch (conditions in which none of the dependent systems on the Space Shuttle had ever been tested) meant that the rubber O-rings became inflexible and allowed the flow of gas to escape and ignite, a failure demonstrated by committee member Richard Feynman on live television during the inquiry. NASA had observed O-rings behaving in unusual and unanticipated ways during previous flights but had made the decision that as long as there was no cataclysmic failure of the equipment, this was an acceptable deviation, a phenomenon referred to as “normalization of deviance.”
Feynman produced an appendix to the final report in which he wrote: “It appears that there are enormous differences of opinion as to the probability of a failure with loss of vehicle and of human life. The estimates range from roughly 1 in 100 to 1 in 100,000. The higher figures come from working engineers, and the very low figures from management. What are the causes and consequences of this lack of agreement?”xiii According to post-disaster analysis, NASA’s management culture in the mid-1980s was strongly biased against the methods of risk assessment that would have highlighted the likelihood of a disaster.xiv
The Challenger disaster is a failure of NASA’s overall Quality Management System (QMS), particularly the Culture of Quality.xv The fact that the design flaw was a known defect but was incorrectly categorized as an acceptable risk, combined with a management structure replete with communications flaws that allowed managers to bypass Quality Management procedures, meant that NASA’s QMS was ill-equipped to prevent or manage a disaster of that scale.
Several of these QMS failures were cited as having a direct impact on the destruction of the Space Shuttle Columbia on February 1, 2003, the definitive cause of which was the impact of a piece of dislodged foam on the left wing of the vehicle during launch. This impact created a breach in the thermal protection system of the wing which, during reentry, allowed superheated air to enter the panels, which subsequently led to the destruction of the vehicle. The result in this case was the loss of seven crew, the destruction of the Space Shuttle Columbia, and the dismantling of the entire Space Shuttle program.
The reportxvi from the Columbia Accident Investigation Board cited poor risk-assessment, lack of managerial interest in promoting safety and Quality, overly simple presentationxvii of complex information required for decision-makingxviii, and normalization of deviance as significant contributing factors, demonstrating that even such a cataclysmic event as the Challenger disaster is sometimes not enough to demonstrate the importance of a QMS to organizations with deeply entrenched process failures.
Canadian Listeriosis Outbreak
In 2008, Canadians in several provinces reported illnesses that were eventually diagnosed as listeriosis, a type of food poisoning related to the listeria monocytogenes bacterium. A total of 57 people became seriously ill and 22 people died. The source of the bacteria was eventually traced to the Bartor Road Maple Leaf Foods plant in Toronto, Canada. An independent investigator determined that the outbreak was a result of contamination of the slicing machines in the plant and not the cooking or preparation process.xix Maple Leaf Foods engaged in an extensive decontamination of the plant in early September of that year. The administrative costs of the recall for Maple Leaf Foods were approximately $20 million. In December 2008, Maple Leaf Foods settled several class action lawsuits for $27 million.
The investigation revealed that Maple Leaf Foods had an exemplary record for safety, with a proactive approach to meeting the compliance regulations put forth by the Federal Government of Canada. The Bartor Road plant undertook immediate corrective actions with any positive tests for listeria contamination. However, the care with which each corrective action was taken gave employees a false sense of security that the issue had been addressed. Employees were collecting data on each incident of contamination, but no one was given the responsibility of conducting a trend analysis to determine if there existed any patterns that would lead to an underlying cause of the contaminations, and the data was never passed on to the office of the Chief Executive Officer.
Investigators determined that two meat slicing machines, despite having had all surfaces thoroughly cleaned, had meat residue lodged deeply inside the internal workings, which allowed listeria to grow unchecked over a long period of time and provided the “ground zero” source of the contamination. In this case, the day-to-day attention the employees paid to the health and safety procedures was not enough to diagnose a contamination that would only have been revealed by long-term trend analysis of the incidents as a series.
Two other Quality factors played significant roles in this event. First, Maple Leaf Foods had created a low-sodium version of its products to meet the request of its larger hospital and long-term care facility customers. Reducing sodium increases the risk of listeria growth, but Maple Leaf Foods did not adjust its procedures to mitigate against this increased risk. Second, Maple Leaf Foods did not disclose the presence of listeria to inspectors at the Canadian Food Inspection Agency prior to the contamination outbreak. Although there was no requirement for Maple Leaf Foods to volunteer this information, it meant that they missed another opportunity to spot a pattern and engage in trend analysis of the previous outbreaks.
This event provides an example of the way in which Quality must become a mindset that permeates every level of an organization, from the floor worker to the Chief Executive Officer. Floor workers cannot be expected to engage in long-term trend analysis, so the exercise must become an essential process stage at a managerial or executive level. Quality is both a big-picture and an on-the-ground way of life, and even a superlative Quality program, such as that of Maple Leaf Foods, is prone to failure without this type of thinking.
Merck Vioxx Recall
In 1999, pharmaceutical giant Merck introduced Vioxx, an anti-inflammatory drug used for treating osteoarthritis. In total, doctors wrote more than 107 million prescriptions for Vioxx between 1999 to 2007. Studies sponsored by Merck as early as 1997, prior to the release of Vioxx, had noted possible negative impacts to the cardiovascular system in test subjects. Despite this knowledge, Merck excluded any investigation into cardiovascular risk in the methodologies of subsequent studies leading up to its application to the U.S. Federal Drug Administration (FDA) in 1998.
In 1999, Merck conducted a large study into Vioxx to expand the parameters of its FDA approvals by demonstrating its efficacy in reducing gastrointestinal side effects compared to other rheumatoid arthritis treatments. This study, known as VIGOR (Vioxx Gastrointestinal Outcomes Research) examined more than 8,000 patients, but had no defined parameters for collecting data on increased cardiovascular risk, despite sufficient evidence from the contrary study to warrant them. When VIGOR was published in the New England Journal of Medicine, any results that pointed to this risk were obscured in order to highlight the positive gastrointestinal benefits. An additional study comparing Vioxx to other treatments (the ADVANTAGE study) was published in Annals of Internal Medicine.
The journal later learned that the study had been conducted primarily for marketing purposes, contained errors relating to cardiovascular risk, and was written without sufficient accreditation by Merck. On September 23, 2004, a clinical trial involving Vioxx revealed an increased risk of heart attack. Merck publicly announced the withdrawal of Vioxx from the market on September 30, 2004. In 2007, Merck agreed to pay $4.85 billion to settle more than 27,000 lawsuits related to Vioxx, in addition to its legal costs of $1.2 billion.xxi
The Vioxx recall is a Culture of Quality failure that extends beyond the boundaries of the Merck organization. Merck’s failure to live up to its own Quality standards was compounded by the failures of medical journals, government agencies, and other industry safeguards to detect the biased information Merck presented to them. There was an interdependence among each of the organizations involved in the larger pharmaceutical culture that encouraged the proliferation of bias and prevented objective scrutiny of evidence, even though such scrutiny was exactly what those organizations, such as academic journals, were supposed to provide. This pan-organizational Culture of Quality must remain collaborative to ensure that it doesn’t simply become an echo-chamber that allows Quality failure to become expected, acceptable, and systemic.
Takata Airbag Recall
In 2013, automakers began recalling models containing airbags made by Takata. As of January 2018, the recall applies to more than 37 million vehicles from 19 different automakers, with the airbags having been cited as the cause of at least 15 deaths.xxii Three Takata executives have been charged with fabricating test data in an attempt to mask the airbag’s defects. Takata has received a fine, so far, of $1 billion and declared bankruptcy in June 2017.
The Takata airbags used a propellant that was susceptible to the effects of long-term exposure to humidity. Prolonged exposure to these conditions meant that the propellant broke down more quickly and would burn too quickly when triggered during a vehicle impact. This would create too much pressure for the airbag inflator, and the inflator could explode and shower the inside of the vehicles with metal shrapnel.
The Takata recall is a Culture of Quality failure in which the manufacturer allegedly deceived automakers regarding the safety of its airbags.xxiii However, since Takata was part of a larger supply chain, automobile manufacturers, in conjunction with the National Highway Traffic Safety Administration (NHTSA) effectively assessed the risk to all vehicles and initiated a staggered recall prioritizing older vehicles in geographic regions with high humidity. Automakers are currently replacing the airbags in all affected vehicles at no cost to the owners as part of a coordinated effort to mitigate the impact of one of the largest and most complex recalls in U.S. history.xxiv
Samsung Battery Recall
In September 2016, Samsung suspended sales of its Galaxy Note 7 smartphone after reports of units catching fire and even exploding. After recalling all the affected units, Samsung determined the batteries, supplied by a third-party vendor, were the cause of fire. Samsung outfitted all units with batteries from a different supplier, only to discover that these, too, continued to catch fire. After a second recall, Samsung found the replacement batteries were the cause of the most recent fires. Samsung recalled all units and stopped production on the Galaxy Note 7, at an estimated cost of US$5 billion.xxv
The failure of the batteries provides a textbook example of the vital importance of supply chain Quality Management within every organization, since suppliers are not a separate entity, but are, in fact, an extension of a manufacturer’s business. The first battery had a design flaw in which the outside casing did not account for thermal expansion effects and, as a result, was not flexible enough to allow the battery to expand and contract during normal use. This oversight led to short-circuits and fires. The replacement battery did not have a design flaw, but in its rush to fill Samsung’s order for 10 million units, the supplier’s poor Quality Management System allowed sloppy workmanship to introduce new flaws into the battery.
Quality needs to live in every part of an organization, from the C-suite to the shop floor and the field. Quality also spans the entire industry of which each organization is a part. However, “Quality” can’t simply become a mantra that organizations speak about in loose terms and assume will become a philosophy that subconsciously permeates each worker and magically produces results. Quality must be praxis, not lip-service to vague ideals, especially when the consequences of getting it wrong can be so high.
Click here to download a pdf of this article.
Intelex Technologies is a Toronto, Canada-based provider of Environmental, Health & Safety, and Quality (EHSQ) Management and workflow software for organizations of all sizes. The company is a leader in software-as-a-service solutions and serves customers from across a wide range of industries, located around the world. The Intelex platform is a mobile solution and provides integrated tools for front-line EHSQ professionals. We can be found at www.intelex.com.
- Deepwater Horizon Study Group, Final Report on the Investigation of the Macondo Well Blowout (March 1, 2011), 49-73, accessed March 9, 2018, http://ccrm.berkeley.edu/pdfs_papers/bea_pdfs/dhsgfinalreport-march2011-tag.pdf.
- BP, Deepwater Horizon Accident Investigation Report (September 8, 2010), accessed March 9, 2018, https://www.bp.com/ content/dam/bp/pdf/sustainability/issue-reports/Deepwater_Horizon_Accident_Investigation_Report.pdf.
- Deepwater Horizon Accident Investigation Report, pp. 33-36.
- Scott Bronstein and Wayne Drash, “Rig survivors: BP ordered shortcut on day of blast,” CNN, June 9, 2010, accessed March 8, 2018, http://www.cnn.com/2010/US/06/08/oil.rig.warning.signs/index.html. v. Ian Urbina, “Workers on Doomed Rig Voiced Concern About Safety,” The New York Times, July 21, 2010, accessed March 8, 2018, http://www.nytimes.com/2010/07/22/us/22transocean.html. vi. Deepwater Horizon Accident Investigation Report, p. 11.
- Andrew Wendler, “How Volkswagen Got Busted for Skirting EPA Diesel Emissions Standards,” Car and Driver, September 21, 2015, accessed March 8, 2018, https://blog.caranddriver.com/how-volkswagen-got-busted-for-gamingepa-diesel-emissions-standards/.
- Jack Ewing, “Volkswagen Says 11 Million Cars Worldwide Are Affected in Diesel Deception,” The New York Times, accessed March 8, 2018, https://www.nytimes.com/2015/09/23/business/international/volkswagen-diesel-car-scandal. html.
- William Boston, “Bad News? What Bad News? Volkswagen Bullish Despite Emissions Costs,” The Wall Street Journal, April 28, 2016, accessed March 8, 2018, https://www.wsj.com/articles/volkswagen-says-diesel-car-buy-backs-to-costalmost-9-billion-1461831943.
- Theo Leggett, “VW papers shed light on emissions scandal,” BBC, January 12, 2017, accessed March 8, 2018, http:// www.bbc.com/news/business-38603723.
- Report of the Presidential Commission on the Space Shuttle Challenger Accident, Chapter IV (June 6, 1986), accessed March 9, 2018, https://history.nasa.gov/rogersrep/v1ch4.htm.
- Report of the Presidential Commission on the Space Shuttle Challenger Accident, Chapter VI (June 6, 1986), accessed March 9, 2018, https://history.nasa.gov/rogersrep/v1ch6.htm.
- Report of the Presidential Commission on the Space Shuttle Challenger Accident, Volume 2, Appendix F (June 6, 1986), accessed March 9, 2018, https://history.nasa.gov/rogersrep/v2appf.htm.
- Trudy E. Bell and Karl Esch, “The Challenger Disaster: A Case of Subjective Engineering,” IEEE Spectrum, January 28, 2016, accessed March 9, 2018, https://spectrum.ieee.org/tech-history/heroic-failures/the-space-shuttle-a-case-of-subjective-engineering.
- Report of the Presidential Commission on the Space Shuttle Challenger Accident, Chapter V (June 6, 1986), accessed March 9, 2018, https://history.nasa.gov/rogersrep/v1ch5.htm.
- Report of Columbia Accident Investigation Board, Volume 1, Chapter 3 (August 26, 2003), accessed March 9, 2018, http:// s3.amazonaws.com/akamai.netstorage/anon.nasa-global/CAIB/CAIB_lowres_chapter3.pdf.
- Edward Tufte, “PowerPoint Does Rocket Science – and Better Techniques for Technical Reports,” accessed March 9, 2018, https://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001yB.
- Report of Columbia Accident Investigation Board, Volume 1, Chapter 7 (August 26, 2003), accessed March 9, 2018, http:// s3.amazonaws.com/akamai.netstorage/anon.nasa-global/CAIB/CAIB_lowres_chapter7.pdf.
- Report of the Independent Investigator into the 2008 Listeriosis Outbreak, Chapter 5 (July 2009), accessed March 9, 2018, http://publications.gc.ca/collections/collection_2009/agr/A22-508-2009E.pdf.
- Harlan M. Krumholz, Harold H. Hines, Joseph S. Ross, Amos H. Pressler, and David S. Egilman, “What have we learnt from Vioxx?”, BMJ (January 209, 2007): 120-123, accessed March 9, 2018, doi: https://dx.doi.org/10.1136% 2Fbmj.39024 .487720.68.
- Alex Berenson, “Merck Agrees to Pay $4.85 Billion in Vioxx Claims,” The New York Times, November 9, 2007, accessed March 9, 2018, http://www.nytimes.com/2007/11/09/business/09cnd-merck.html.
- “Takata Airbag Recall – Everything You Need to Know,” Consumer Reports, January 19, 2018, accessed March 9, 2018, https://www.consumerreports.org/cro/news/2016/05/everything-you-need-to-know-about-the-takata-air-bag-recall/ index.htm.
- Jeff Plungis, “What’s Happening With the Takata Airbag Recall?”, Consumer Reports, March 1, 2017, accessed March 26, 2018, https://www.consumerreports.org/airbags/what-is-happening-with-takata-airbag-recall/.
- National Highway Traffic Safety Administration, Recall Spotlight: Takata Air Bags, January 5, 2018, accessed March 9, 2018, https://www.nhtsa.gov/recall-spotlight/takata-air-bags.
- Shara Tibken, Roger Cheng, “Samsung answers burning Note 7 questions, vows better batteries,” CNET, January 23, 2017, accessed March 9, 2018, https://www.cnet.com/news/samsung-answers-burning-note-7-questions-vows-betterbatteries/.