Show  See the article in its original context from TimesMachine is an exclusive benefit for home delivery and digital subscribers. About the Archive This is a digitized version of an article from The Times’s print archive, before the start of online publication in 1996. To preserve these articles as they originally appeared, The Times does not alter, edit or update them. Occasionally the digitization process introduces transcription errors or other problems; we are continuing to work to improve these archived versions. A computer malfunction apparently caused excessive radiation doses for two cancer patients at a treatment center, causing the death of one man, a Texas official said today. An identical machine at a Georgia treatment center unleashed a large radiation dose in June 1985 on a patient undergoing treatment for breast cancer, causing nerve damage. Bob Free, an investigator with the Texas Bureau of Radiation Control in Austin, said the two incidents at the East Texas Cancer Center appeared to have been caused by a malfunction that occurred after the operator of the machine entered an erroneous command into the Canadian-made Therac 25 linear accelerator and then corrected it. Use of the Therac 25 linear equipment has been discontinued, said Dick Higginbotham, administrator of the cancer center. Corrective Action The manufacturer, Atomic Energy of Canada, a Government-owned company, is taking corrective actions, which include a modification of the software, Mr. Higginbotham said. Dr. Kenneth Haile, director of the Kennestone Regional Oncology Center in Marietta, Ga., said an investigation into an accident there, begun as a result of the Texas cases, blamed a ''software glitch'' involving ''an unforeseen sequence of computer commands.'' When a certain group of commands were typed into the machine at a rapid rate of speed, it produced higher radiation than was called for, he said. ''What the computer screen indicated was going on was not what was going on,'' Dr. Haile said. Officials at the Texas center said a 66-year-old man, confined to a wheelchair and suffering from emphysema, died in April after receiving an excessive dose of radiation earlier in the month. Another patient, a 33-year-old man, received an excessive dose from the same machine March 25 and is being treated at a Dallas hospital, center officials said. How Overdose Happened In the case of the 33-year-old man, the operator began radiation treatment with the equipment after correcting an operator-entry command. The machine shut down immediately and flashed a malfunction message, Mr. Free said. The operator believed the patient had not received the treatment and reset the machine, he said. But the patient had experienced something like an electrical shock and was rolling off the table when the second dosage caught him in the shoulder and neck, Mr. Free said. Mr. Free said representatives of the clinic and manufacturer examined the machine and determined there was no electrical or other hazard. As a result, the machine was put back into use. Three weeks later, the same error was made on the 66-year-old man, Mr. Free said. This time, he said, the operator immediately shut down the machine. Gerald Vince of the Federal Food and Drug Administration's Dallas office said the two incidents at the East Texas Cancer Center have been under investigation for more than two months. In the Georgia case, Katy Yarbrough, 62, was undergoing treatment for breast cancer when she felt pain and irritation below the collarbone in a spot that over several weeks developed into a deep, discolored hole ''between the size of a quarter and a half-dollar,'' said her lawyer, Bill Bird. The radiation injury destroyed a vital nerve center and robbed her of the use of her left arm, Mr. Bird said. She has filed a lawsuit in Superior Court in Atlanta against Dr. Haile, Kennestone and the manufacturer of the machine. Dr. Haile said the Georgia center was continuing to use the machine, which has been reprogrammed to prevent similar incidents. Victor Garcia considers himself lucky to be alive. Three years ago, a combination of cancer and miscalculation almost killed him. In the 40 months that have passed, 21 patients have died. While its unclear how many of the patients would have died of cancer anyway, the International Atomic Energy Agency (IAEA) said in May 2001 that at least five of the deaths were probably from radiation poisoning and at least 15 more patients risked developing “serious complications” from radiation. Garcia, being treated for prostate cancer, survived but suffered damage to his intestines. He now has a colostomy. “I am very lucky,” he says, shaking his head in wonderment. “Thats what the [investigating] doctors from Houston told me. You are so lucky.” The three Panamanian medical physicists who used the software to figure out just how much radiation to apply to patients are scheduled to be tried on May 18 in Panama City on charges of second-degree murder. Under Panamanian law, they may be held responsible for “introducing changes into the software” that led directly to the patients deaths, according to Special Superior Deputy Prosecutor Cristobal Arboleda. The physicists, of course, thought they were helping the patients. Having consulted a doctor at the hospital and the softwares manual, they thought they had figured out how to place five radiation shields over each patients body, instead of four, to protect against possible overdoses. “I thought I was home free,” one of them, Olivia Saldaña, recalls now. This is not a cautionary tale for medical technicians, even though they can find themselves fighting to stay out of jail if they misunderstand or misuse technology. This also is not a tale of how human beings can be injured or worse by poorly designed or poorly explained software, although there are plenty of examples to make the point. This is a warning for any creator of computer programs: that software quality matters, that applications must be foolproof, and that— whether embedded in the engine of a car, a robotic arm in a factory or a healing device in a hospital— poorly deployed code can kill. Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com Next Page: Not the first time. Not the First Time In this case, a St. Louis company, Multidata Systems International, has found itself in and out of courts in two countries for much of the past three years, fending off charges that its product is at fault in a score of fatalities. The deaths occurred more than 2,000 miles from its home, at an installation of a customer it claims it did not even know it still had-until the death toll began mounting. Now Multidata may face judgments that could damage-if not destroy-the company itself, if the firm is found guilty and is forced to pay damages sought by the victims. No one can accurately predict the amount Multidata would have to pay if the victims succeed in suing in the U.S. So far the plaintiffs have failed. But each of the 28 victims could be entitled to as much as $500,000 to $1 million of compensation for such factors as pain and suffering, lost wages and the number and age of surviving dependents, according to Brian Kerley, a defense attorney at a leading New York malpractice firm. Using those numbers, Multidata could be facing total damages in the range of $14 million to $28 million. Multidata, which is privately held, says it has about $2 million in annual sales and fewer than 15 employees. That company could just as well be your company, whether you write software in small or large teams; and whether you operate domestically or in multiple nations in a rapidly globalizing economy. You are at risk if you place your product in conditions where human lives are at stake. Indeed, its not the first time that software has been a suspect in a series of unexpected fatalities. In the mid-1980s, poor software design in another radiation machine, known as the Therac-25, contributed to the deaths of three cancer patients. The Therac-25 was built by Atomic Energy of Canada Ltd., which is a Crown corporation of the government of Canada. In 1988, the company incorporated and sold its radiation-systems assets under the Theratronics brand. There does not appear to be any formal investigation of the Therac-25 accidents, but according to an in-depth examination by Nancy Leveson, now a professor at the Massachusetts Institute of Technology, and the accounts of other software experts, the design flaws included the inability of the software to handle some of the data it was given and the delivery of hard-to-decipher user messages. In a twist of fate, Theratronics, which was ultimately acquired by the Canadian life-sciences company MDS, manufactured the radiation-therapy machine used at the cancer institute in Panama. In February 1991, during Operation Desert Storm, an Iraqi SCUD missile hit a U.S. Army barracks in Saudi Arabia, killing 28 Americans. The approach of the SCUD should have been noticed by a Patriot missile battery. A subsequent government investigation found a flaw in the Patriots weapons-control software, however, that prevented the system from properly tracking the missile. More recently, during Operation Iraqi Freedom, the Patriot missile system mistakenly downed a British Tornado fighter and, according to the Los Angeles Times and other reports, an American F/A-18c Hornet. The pilot in the single-seat Hornet and the two crew members aboard the British jet were killed. The incidents are still under investigation, but Pentagon sources familiar with the Hornet incident told the L.A. Times that investigators were looking at a glitch in the missiles radar system that made it incapable of properly distinguishing between a friendly plane and an enemy missile. Raytheon, the maker of the Patriot missile system, did not want to comment on the 1991 incident. It also said the government was still investigating the more recent incidents and that reports the software may be at fault were “off base.” Next Page: Software anomaly in U.S. tilt-rotor aircraft partial cause of crash. Software Anomaly A software glitch was cited in a Dec. 11, 2000, crash of a U.S. Marine Corps Osprey tilt-rotor aircraft, in which all four Marines on board were killed. According to Marine Corps Maj. Gen. Martin Berndt, who presented the finding from a Judge Advocate General investigation, “the mishap resulted from a hydraulic-system failure compounded by a computer-software anomaly.” A hydraulic line broke in one of the crafts two engine casings as the pods were being moved from airplane mode to helicopter mode in preparation for landing. When the flight-control computer realized the problem, it stopped the rotation of the engine pods. The pilots, trained to respond, tried to reset the pods by pressing the primary reset button, but the finding stated that a glitch caused “significant pitch and thrust changes in both prop rotors,” which led to a stall. The plane crashed in a marsh. The craft is made by a partnership of Boeing and Bell Helicopter. A Boeing spokesman said changes were made in the software but referred requests for details about the software anomaly to the government. A spokesman for the Navys Air Systems Command, which investigated the incident, confirmed the software problem, but was not able to provide additional details. Nor are these incidents likely to be the last. In 2002, the Food and Drug Administration (FDA), which oversees medical-device software, said of 3,140 medical-device recalls conducted between 1992 and 1998, 242, or 7.7 percent, were attributed to software failures. The FDA also says the number of software-related recalls may be underreported because its often hard to determine the exact cause of a problem in the immediate aftermath of an accident. Theres
a financial cost to all organizations that use badly designed and deployed software as well. Poor-quality software costs U.S. businesses $59.9 billion annually, according to a 2002 report from the U.S. Commerce Departments National Institute of Science and Technology (NIST). The NIST study looked not just at the cost of finding and fixing software problems, but also at costs incurred from lost retail transactions and manufacturing product delays. Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com Next Page: Expert predicts an increase in software-driven medical device problems. Medical Device Prediction Problems The medical-device software market is becoming a particular area of concern. The FDA says about half of the 10,000 medical devices on the U.S. market are software-driven-everything from pacemakers to infusion pumps to radiation-therapy machines. FDA watchers say many of the companies developing medical-device software are small. Because of the amount of research-and-development money that goes into medical devices, companies are under pressure to get products out the door. “We will see more problems” in the medical-device field, says Alan Kusinitz, managing partner of SoftwareCPR, a consulting company that specializes in medical-device software. One of his biggest worries is the ever-increasing number of networked medical devices. Independently, software might function normally, but when connected to code in other machines, it may act unpredictably. “Its the abnormal stuff that always shows up later in weird circumstances,” Kusinitz says. “Thats most often where safety problems occur.” There are defense and industrial efforts underway by organizations such as the Sustainable Computing Consortium (SCC) and the Software Engineering Institute, both located at Carnegie Mellon University, to foster programs and standards to reduce software defects. There are also organizations in the healthcare industry, such as the Association for the Advancement of Medical Instrumentation, that are trying to establish standards for software used in medical devices. In addition, new testing tools and services, such as Software Development Technologies ReviewPro, which examine not just the code but the methodology behind the code, are starting to offer software professionals assistance in vetting their output, as they create it. Also, code-writing practices such as “agile programming” emphasize breaking big projects into small pieces-and getting early and repeated input from users before proceeding. Such efforts, though, are too late to help Victor Garcia or the physicists at the cancer institute in Panama who were trying to use imported software to save patients, not make them suffer. Next Page: The role of a medical physicist. Medical Physicists Role Perhaps nothing shows the ravages of faulty calculations as clearly as cancer. The patients who were suffering in Panama had cancers of the pelvis. Pelvic organs such as the intestines and kidneys are acutely sensitive to radiation. Before a cancer patient such as Garcia is exposed to radiation, a doctor devises a treatment plan that determines what dose of radiation can safely be directed at the tumor. The physician considers the tumors position and depth in the body, the likelihood that the cancer has spread to surrounding tissue, the location and sensitivity of nearby organs and the best angles of attack. As part of the plan, the doctor figures out how to place metal shields, known as “blocks,” above the area where the tumor is located. These blocks, usually made of lead or a metal alloy called cerrobend, protect normal or sensitive tissue from the gamma rays to come. The doctor hands his plan to a medical physicist, who feeds information on the size, shape and location of the blocks into a software package. These packages generally create a 3-D picture of how the dose will be distributed, showing how the radiation will “sum” as beams coming in from different angles intersect at depth in the patients tissue. Once the doctor prescribes a dosage, the software calculates the duration of treatment. The physicists in Panama were carrying out a doctors instruction to be more protective, adding a fifth block to the four the hospital often used on patients in cancer treatments. The extra block could help protect patients whose tissues were especially sensitive due to previous surgeries or radiation treatments. Multidatas planning software was designed to calculate treatments when there were four or fewer blocks, according to the companys general business manager, Mick Conley. Saldaña, however, read Multidatas manual and concluded she could make the software account for a fifth block. According to an August 2001 report from the IAEA, Saldaña found the software didnt only work if she entered the dimensions of each block individually, up to four. She found it also allowed her to enter the dimensions of all five blocks as a single, composite shape-for instance, a rectangle with one triangular block sitting in each corner and a fifth square block protruding, tooth-like, down into the rectangle from the top. Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com So, using the mouse attached to her computer, she entered on the screen the coordinates of the specially shaped block— first the inner perimeter of the shape and then the outer perimeter. This is when she felt she was “home free.” After all, when Saldaña entered the data for this unusual-looking block, the system produced a diagram that appeared to confirm its dimensions. She seemed to be getting confirmation from the system itself that her approach was acceptable. Next Page: Ravages of miscalculation. Ravages of Miscalculation But inside the software, the calculations of appropriate dosages were going awry. The treatment time would be close to correct if Saldaña entered the data for the inner perimeter of the shape going in one direction, say clockwise, and the outer perimeter in the opposite direction, according to the IAEA report. But if she entered the data for the inner and outer perimeters going in the same direction, so that the two loops defining the perimeters crossed, the software essentially locked up. It was not able to accurately recognize the shape and, as a result, miscalculated the treatment times, the report said. Depending on how many treatments the patients received, they accumulated overdoses ranging from 20 percent more radiation than was prescribed to a double dose of the potentially harmful rays, the IAEA found. Inspectors from the FDA were dispatched to Multidatas offices after the agency received reports of patient “radiation overexposures.” The inspection ran from May 31 to Sept. 21, 2001. A summary of their findings echoed the IAEA report: “The treatment-planning system miscalculated the dose each patient was to receive due to failure of the software to correctly handle certain types of blocks… This resulted in a much higher dose being calculated for each patient.” Multidatas Conley says the FDAs finding “is wrong.” He says that if you read FDA reports, “you find out the FDA isnt always right. “Given [the input] that was given,” he says, “our system calculated the correct amount, the correct dose. It was an unexpected result. And, if [the staff in Panama] had checked, they would have found an unexpected result.” Conley insists his company has done nothing wrong. He says the physicists at the National Cancer Institute never called Multidata asking for advice or support. The physicists admit they did not always verify the results of the softwares calculations, which Multidatas manual said was “the responsibility of the user.” Saldaña says the hospital was treating more than 100 patients per day using the one Cobalt-60 machine. The IAEA also found that whatever steps the hospital took to ensure the radiation machine was operating properly only addressed the hardware. There was no quality-assurance program for the software-or its results. In the day-to-day operations of the cancer institute, that meant the physicists were not required to tell anyone they had changed the way they entered data into the cancer-therapy system. As a result, no one on staff questioned the softwares results. Had the hospital verified the dosages, by manually checking the softwares calculations or by testing the dosages in water before radiating patients, a procedure that Conley argues is standard medical practice in much of the rest of the world-the staff would have caught the overdoses in time to avoid harming anyone. But independent experts not associated with the case say software that controls medical equipment and other life-critical devices should be designed to pause or shut down if told to execute a task its not programmed to perform. “If a computer can make a user kill people, its like a loaded gun,” says Jack Ganssle, an engineer whose Ganssle Group advises companies and developers on how to create high-quality software. “A user shouldnt be able to do anything that causes a machine to be dangerous.” But the Multidata software continued to operate. Next Page: Cause of death. Cause of Death As tragic as it is, the Panama incident does not stand alone. In all, Baseline has found no fewer than a half-dozen cases in which software has contributed to loss of life. (See Eight Fatal Software-Related Accidents, Baseline, March 2004.) At least three deaths were blamed on a software glitch that crippled the East Coasts power grid last summer. In 1997, the safe-altitude warning system at Guam International Airport inexplicably generated an excessive number of false alarms that planes were flying too low. As a result, air-traffic controllers cut back the distance scanned by the system from 54 miles to 1 mile. The change prevented controllers from warning pilots of Korean Airlines Flight 801 that they were flying toward a mountain. The crash killed 225. There are also scores of personal injuries in which software was at least partly to blame. A rider on a gyroscopically controlled Segway scooter suffered a head injury because of a software-design gap and, according to the National Highway Traffic Safety Administration, more than 476 people have been hurt because of a problem with a General Motors anti-lock braking system in use from 1991 to 1996. GM said the braking system wasnt designed to check for certain drive-train variables. Certainly, deaths and injuries that can be in some fashion tied to software are statistically rare. Overall, software quality is “generally pretty good,” says James Gosling, a Sun Microsystems vice president. But Gosling, regarded as the father of the Java programming language, which can be used to build applications that can run across diverse computers, says code-writing in many cases is still flawed. Many specifications and designs arent thought out well enough. Programmers, no matter how good, make logical mistakes. In addition, testing procedures often arent rigorous enough, he says. And today, with so many software programs interacting with other software programs, theres no way to predict what will happen when two pieces of code come in contact with each other for the first time. “The quality fight is never-ending,” he says. The threat of physical harm and crippled lives is escalating, now that software drives not just healthcare machinery, but our cars and our household appliances as well. It runs elevators and amusement-park rides. It controls just about every manufacturing plant, utility and business office in the country. As software becomes more pervasive, software quality— long a discussion confined to software-development circles— becomes an issue for business executives, product managers, factory floor supervisors and, as the physicists in Panama found out, anyone who uses software in the workplace. Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com Next Page: All software has bugs. All Software Has Bugs But all software has bugs. For every thousand lines of code developed by commercial software makers or corporate programmers there could be as many as 20 to 30 bugs, according to William Guttman, the director of the SCC, a group of businesses and academic institutions looking for ways to make software more dependable. Many common programs have a million or more lines of code. Sun says its Solaris operating system has more than 10 In a one-million-line piece of code, even if you only have one bug per thousand lines, youre still going to have 1,000 bugs, says Michael Sowers, executive vice president at Software Development Technologies, a software-testing company. In todays software, says Khosla, “you have to assume there are some bugs in the code.” Just look back at the first major case of code that killed, in healthcare. The Therac-25 was one of the first “dual-mode” radiation-therapy machines, which meant that it could deliver both electron and photon treatments. Electrons are used to radiate surface areas of the body to kill cancer cells. A photon beam, normally called an X-ray, can be a hundred times more powerful and as a result is used to deliver cancer-killing radiation treatments deeper into the body. According to Prof. Levesons account, the machine was “more compact, more versatile, and arguably easier to use” than its predecessor machine. But, according to Prof. Levesons 1995 book “Safeware” and other accounts, there were a number of flaws in the software that led to the Therac-25 radiation overdoses at health facilities in Marietta, Ga.; Tyler, Texas; Yakima, Wash.; and elsewhere. In all, three people died. One of the problems manifested itself in 1986 when a physicist tried to change machine set-up data-such as radiation dosage and treatment time-that had been keyed into the software. The machine went through a series of steps to set itself up to deliver either electrons or photons and the dosage of the selected beam. As data was given, the machine recorded the information and then followed the instructions. In some cases, however, operators realized while setting up the machine that they had entered an incorrect piece of information. This could be as simple as unintentionally typing in an “X” for an X-ray (or photon) treatment instead of an “E” for an electron treatment. In “fixing” that designation, an operator would move the cursor up to the “treatment mode” line and type in an “E.” The monitor displayed the new entry, seemingly telling the operator that the change was made. But in the case of the Therac-25, the software did not accept any changes while going through its eight-second-long set-up sequence. No matter what the screen might show, the software grabbed only the first entry. The second would be ignored. Unaware the changes did not register, operators turned on the beams and delivered X-rays, when they thought they were delivering electrons. According to Levesons account, patients received such incredibly high quantities of radiation that the beams burned their bodies. Patients who should have received anywhere from 100 to 200 rads of radiation were hit instead with 10,000 to 15,000 rads, in just one or two seconds. A thousand rads is a lethal dose. Next Page: How the FDA, and few other U.S. agencies, regulate dangers in software. Regulating Dangers The Therac-25, according to John Murray, head of software regulatory efforts at the FDA, was a “seminal event” for the agency. After the incident, the FDA for the first time turned its attention to the software that had begun to control medical devices. The FDA has the power to inspect the work of manufacturers; to ask manufacturers to recall products; to have federal marshals seize products if a voluntary recall isnt done; and to ask the courts to issue injunctions against the distribution of products if a manufacturer does not have good manufacturing procedures in place. To help software manufacturers, the FDA issues “guidance” documents that recommend that manufacturers follow generally accepted software-development standards; keep track of their design specifications; and conduct formal reviews and tests of the code they produce. Arne Roestel, Multidatas president, says the company followed the FDA recommendations. But there are few specifics. According to the FDAs “General Principles of Software Validation,” which went into effect in January 2002, “This guidance recommends an integration of software lifecycle management and risk management activities. Based on the intended use and the safety risk associated with the software to be developed, the software developer should determine the specific approach, the combination of techniques to be used, and the level of effort to be applied.” In the wake of Panama, some industry experts wonder if theres enough oversight of medical-device software-or, for that matter, software development in general. They say the time might be right for tougher regulation. Software engineer Ganssle, for one, notes that programmers dont need any form of certification or license to work on commercial software, including life-critical medical device software. Yet, he says, “In Maryland, where I live, if you want to cut hair, you need to be licensed.” Besides the FDA, there are few federal agencies policing software-development practices. The Federal Aviation Administration oversees the flight-control software in commercial aircraft. The Nuclear Regulatory Commission (NRC) watches over the software that runs nuclear plants. And thats about it, for oversight of commercial software. The Occupational Safety and Health Administration, the Consumer Product Safety Board and other agencies charged with protecting factory workers, professionals and consumers say they dont worry about the quality of software in tools or toys. Next Page: What went wrong. What Went Wrong Overlooking the skyscrapers of downtown Panama City, amid towering palm trees and gracious homes in the old Canal Zone, sits Gorgas Hospital, an imposing concrete structure which now houses Panamas National Cancer Institute. This is a public hospital. No Panamanian is turned away. On a Monday morning in January, at least 50 patients and their family members, including Victor Garcia and his wife, are visiting the institute. The patients walk slowly up the driveway; sit quietly on the patio under the lush vegetation that surrounds the building; stand in the lobby. They are all waiting for treatment. This is not even the hospitals busiest day of the week. That is Tuesday, when the clinic offers every citizen, even those without a doctors referral, free diagnoses of the skin cancer that tends to flourish under the equatorial sun. Cancer is a leading cause of death in Panama: prostate cancer for men, endometrial and cervical cancer for women. And those are unlikely to be just the suns fault. Many Panamanians blame the United States testing of the chemical defoliant Agent Orange in the Canal Zone during the Vietnam War. Since 1997 the number of new cancer patients in Panama has more than quadrupled, according to the cancer institute. The hospital now sees 10 to 15 new patients per day and performs 300 cancer surgeries per month. The victims of the faulty radiation treatments in 2000 and 2001 span the breadth of Panamanian society. Among the dead are Margarita Sevillano, a folksinger; Walter Chandler, a professor at the University of Panama; and Rosa Vergara, a nun. Many of the dead lived in the barrios in the hills above downtown, where chickens peck along the roads, laundry flaps from porches and brightly painted stucco houses are interspersed with small shops and Internet cafés. The hospitals radiotherapy unit is critical to Panama. When the IAEAs investigation, in May 2001, slowed the hospitals routine, patients lined up waiting to be treated. That led the Panamanian ambassador in Austria, Jorge Perez, to urge the Vienna-based agency to hurry up. “Those who could afford to went to the private clinics,” Perez says. “Those who could not, waited.” The difference in cost to the Panamanian government is stark. Garcias treatment, for example, which cost him virtually nothing at the National Cancer Institute, would cost $4,000 at a private hospital, using a Cobalt-60 machine. Using a higher-powered, more-precise linear accelerator, the bill would escalate to $10,000. The current chief of the cancer institutes radiotherapy unit, Dr. España de la Rosa, asserts that some patients died in 2001 waiting for treatment, while the overdoses were being investigated. She says she does not know how many. Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com But the survivors of the overdoses didnt fare well, either. The governments of France and Argentina each offered to take two of the over-radiated patients and treat them for a year at no charge. Panama sent no one. “We are a small country, and everybody knows everybody,” Ambassador Perez says. “How do you decide who to send?” The overdoses occurred not in the newly renovated Gorgas Hospital on the hill, but in the cramped Justo Arosemena Avenue facility downtown, which the hospital was in the process of vacating. The Multidata software and the Cobalt-60 teletherapy machine manufactured by Theratronics had been installed there in 1993. According to a letter written to Multidata by ProMed, the Panamanian distributor that sold the hardware and software, the hospital was looking for cheaper software because it couldnt afford the software that Theratronics typically supplied with its radiation machine. ProMed services manager Camilo Jorge says he doesnt remember the price difference, but he knows the hospital never purchased a maintenance contract for the software-only for the radiation machine. By 1997, hospital staff was so concerned about the possibility of unintended excess exposure that they warned in a report requested by the Ministry of Health of “overexposure of radiation-therapy patients due to human error” unless conditions at the hospital improved. Next Page: The machine was being used nearly twice what the maintenance program recommended. Machine Overused In the report, the staff claimed the hospital was understaffed and poorly equipped, and they asked for more frequent maintenance on the Cobalt-60 teletherapy machine. The contention: the machine was being used 3,780 hours per year, nearly twice what the maintenance program recommended. The staff also asked the hospital to have Multidata do “preventive maintenance” on the software. But the software was never maintained, and by 2000, the hospital was using just the one Cobalt-60 machine to treat all patients, according to Saldaña. A second, older machine was retired. By then, two of the hospitals five radiation physicists had quit. Saldaña says the remaining three did the work of five, which sometimes required 16-hour days. Victor Garcia remembers waiting five to six hours for every treatment. And after each of those six treatments, he felt sicker. As his intestines struggled to slough off cells killed by the radiation, he developed diarrhea. Burns seared through the flesh on his back. He lost 30 pounds. Hospital doctors told him the symptoms were normal. One reason it took hospital staff seven months to discover the overdoses, according to hospital director Juan Pablo Bares, is that patients with pelvic cancers often show symptoms of radiation toxicity, and the number of patients overdosed was small compared to the number being treated. But another reason was the complexity of the software. The glitch involving Multidata was activated only under very specific circumstances-when the dimensions of the blocks that defined the patients treatment area were entered in a particular way. If the blocks were treated as a single, composite shape, and the descriptions of their dimensions were entered so that the “loops” that defined the inner and outer perimeters of that shape crossed, the software would increase patients treatment time, the IAEA report said. As patients began to sicken and then die, the staff hunted for the cause. Saldaña remembers that by March 2001, she was thinking the problem had to be the software. But even then she discovered it by accident: On the morning of March 2, according to a statement she gave to the prosecutors office, she was calculating dosages for two patients with equivalent treatment areas and treatment depths and suddenly realized that the treatment times that came out of the software werent even close. And so began the hospitals effort to unearth the causes of the overdoses. Radiotherapy expert J. Francisco Aguirre, who investigated the overdoses for the Panamanian government as part of a team from the M.D. Anderson Cancer Center in Houston, says the calculation error was a problem that occurred with algorithms in older software used to plan treatments, a linkage that Multidata president Arne Roestel denies. Aguirre says the error was so obscure he wouldnt have thought to look for it-except that while he was in Panama, he remembered seeing a physicist in the U.S. cause a similar error 10 years before. “The trick is how to tell the computer what are [empty] holes and what is solid,” Aguirre says. “If the lines you are digitizing cross along the way, you fool the computer.” Next Page: The investigations. The Investigations Multidatas Mick Conley, in an interview with Baseline on Feb. 19, maintained that his company first heard full details of the overdoses from the FDA and the NRC in June 2001. But Aguirre says that is not true. His team notified Multidata in April, he says, two days after they returned from Panama to the U.S. “We told them, Your equipment has had an accident, go and make sure that for all the systems youve sold in the world, people are aware this is a problem,” Aguirre says. The IAEA report confirms that the Houston team notified Multidata in April, “and that it was impressed upon Multidata to send someone to Panama as soon as possible to resolve this problem.” But Roestel says Multidata could not get enough information from the hospital or the Houston team on exactly what had happened. In addition, Conley says, in order for the company to send representatives to Panama, the hospital would have needed a service contract with the company, which it didnt have, although he adds that Multidata provides telephone support to anybody with its product. But before sending a team out, Conley says, “you really have to have an idea of what the problem is.” If another customer called in a similar situation, he says, Multidata would want a clear explanation of what the problem was so that they “knew what kind of person to send to a place and what kind of tools to send along with them.” On May 18, after several patients had already died, the Panamanian government announced the overdoses, and international agencies began to act. In the U.S., the NRC sent out warnings on June 1 and again on June 6 alerting hospitals licensed to perform radiation therapy of the overdoses-and that Multidatas software was involved. Multidata issued its first warning to customers on June 22, although it did not say which versions of its software were affected or how the overdoses happened. The company directed users to “follow instructions in the user manual…follow a written quality assurance procedure…and perform verification measures.” Multidata issued what Conley calls a “voluntary in-field correction,” starting in the fall of 2001. The software patch checked treatment-planning calculations and rejected anything that was not identified by the system as a valid shape. “We changed the software so that [the Panama incident] could not happen again,” he says. Conley maintains the company was initially unaware the software was still being used in Panama. “We never heard from [the hospital],” he says. Conley and Roestel also contend the software was fine and that the problem was user error. The IAEA report did note that quick action by the hospital staff could have prevented the overdoses. At the end of May 2001, right after the FDA became aware of the accidents in Panama, it sent its examiners to inspect Multidata. The FDA found that Multidata had received at least six complaints about “calculation errors related to the failure of the firms radiation treatment planning software to correctly handle certain types of blocks (polygons).” The report said: “As of 6/1/01, there was no documentation that any complaint or incident report analysis had been performed, or corrective action developed or implemented [by Multidata]. In addition, the firm had been aware of this failure since at least 9/92.” Separately, the NRC published the findings of the IAEA in an Information Notice to operators of radiotherapy machines dated Nov. 20, 2001. The notice said: “Specifically, the staff modified its procedures and entered data for multiple shielding blocks together (digitized the blocks), as if they were a single block. The data were accepted by the treatment planning system, but the [Multidata] software calculated incorrect treatment times. Using incorrect treatment times resulted in significant radiation overexposure to patients.” Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com Next Page: Multidata repeatedly taken to task by the FDA. FDA Takes Multidata to Task”> In 2003, Multidata signed a consent decree with the FDA that precludes the company from making or selling software for radiation-therapy devices in the U.S., although it can still export its products. According to the FDAs injunction announcement, Multidata failed to meet the FDAs manufacturing practices and design standards. Of all the criticism the firm received, perhaps the harshest was a statement made by FDA Commissioner Mark McClellan when the injunction was made public on May 7, 2003. “Multidata Systems has a nine-year history of violations and failure to correct them,” he said. “Despite repeated warnings, the company continued to manufacture its medical devices in a way which put the public health at risk.” Indeed, the FDA has taken Multidata to task several times over the last 10 years for its software-development practices. The FDAs policy is to inspect any company that makes medical devices about every two or three years. Each time its examiners visited Multidata they found problems. The FDA would not say how common it is to find problems on each company visit, but Gladys Rodriguez, head of the FDA enforcement unit that deals with medical devices and radiological health products, says an injunction is a “rare” action. According to documents obtained by Baseline under the Freedom of Information Act, the FDA inspected Multidatas products and manufacturing operations four times
in the past dozen years. Next Page: FDA sends warning to Multidata in 1998. Warning in 1998 After the FDAs 1998 inspection, the agency sent a warning letter to the company, outlining the findings from its review and instructing the company to look into its software-development process and correct the deficiencies noted in its report. The agency told Multidata that if the company didnt take care of the problems, the FDA would be forced to take further action, which could include fines and an injunction against the company. The FDA also told Multidata it wanted the company to notify the agency within 15 days of the steps the vendor was going to take to address the problems. According to the subsequent FDA reports, Multidata never responded to the 1998 letter. “We usually get action from our warning letters,” Rodriguez says. Multidatas Roestel admits that the company did not respond to the FDAs letter. In 2001, after the deaths in Panama, FDA inspectors found many of the same problems they found earlier. Multidata, it said, had no mechanisms for addressing incomplete or ambiguous software requirements, customer complaints were not being properly recorded, and there was no comprehensive testing plan to demonstrate that its software was “fit for use.” Conley says the company has been working hard to rectify the complaints identified in the FDAs 2001 inspection. He says Multidata, in fact, was looking for ways to better track software fixes and problems, including the installation of a computerized system to record and store customer complaints, when the FDA issued its injunction. Conley admits Multidata didnt address issues quickly enough for the FDA. “Were a small company that didnt always react in a timely fashion,” he says. “We did what we were supposed to do, [but] we didnt file the proper reports for it.” To have the injunction lifted, the FDA says Multidata must improve its design and manufacturing methods; upgrade its record-keeping mechanisms; and retain a medical-device design expert to inspect the companys manufacturing activities, check over its software code, and report back to the FDA. The outside expert must have no financial ties to the company other than the consulting agreement for this series of tasks. Conley said in February 2004 that Multidata will have its development practices reviewed
by Bio-Reg Associates Inc., which describes itself as a “regulatory consulting firm conveniently located close to the FDA in Washington, D.C. Want the story latest news in programming environments and developer tools? Check out eWEEKs Developer Center at http://developer.eweek.com Next Page: Examining the regulators. Examining the Regulators Despite the strong action it took against Multidata, FDA watchers say the agency-one of the few empowered to regulate software in any fashion-still does not go far enough to insure the integrity of the computer programs that are the brains of medical devices. “Thinking the FDA is some sort of watchdog group is an exaggeration,” says Bob Morton, a software-quality expert and a former head of the FDA unit in charge of radiation-therapy equipment. The FDA approves medical devices under one of two mechanisms: premarket approval or premarket notification. Premarket approval is reserved for technologies that are radically different from anything on the market, such as a breakthrough pacemaker product. These products are run through scientific reviews and tests to ensure that they are both safe and effective. Products which fit into an existing category of devices are subject only to the premarket-notification process, under which the FDA neither tests products nor requires a manufacturer to conduct its own field trials. Multidata won approval for its software in March 1997 under the notification process, since a number of radiation-treatment planning packages were already on the market. In this procedure, a manufacturer such as Multidata submits paperwork that details how they designed, produced and tested the new product. Agency officials then pore over the documents looking for potential flaws, problems in testing or other likely trouble. If everything is in order, the FDA gives the company clearance to sell its product. Last year, 3,500 of the 4,000 medical devices approved by the FDA came through the notification process. “Can we rely on the FDA to police the medical-device industry in a very reliable way, via the premarket submissions and the inspections?” asks SoftwareCPRs Kusinitz. “No. I think were primarily dependent on the…good intentions of the medical-device manufacturers.” In their defense, FDA officials say its not feasible to test every product. “We regulate 10,000 different types of products. How do you come up with tests for 10,000 products?” says Murray, the FDA software expert. Besides, he says, the FDA has no conclusive data that outside testing would improve the quality of software anyway. Critics, however, still take issue with the FDA being so reliant on human review. Next Page: Back in Panama. Back In Panama Even though Multidata did send representatives down to Panama to deliver a patch for the hospitals software and provide its staff with additional operating instructions, the hospital had stopped using the software in June. Patients were still being treated with the Cobalt-60 teletherapy machine, but the physicists calculated the patients treatment times the old-fashioned way-by hand. As a result, the hospital could only handle around 60 to 70 patients per day, instead of 100. That led to an even longer waiting list and forced the Panamanian government to start subsidizing private hospitals, where it sends those patients who are employed and therefore covered by social security. So far, says Dr. de la Rosa, the government has spent $10 million subsidizing private treatments. Meanwhile, the three physicists are free on their own recognizance while they await trial. Two of them-Saldaña and Alvaro Mejia-continue to work at the National Cancer Institute. The physicists are funding their own defense, even though Saldaña, for instance, makes $585 a month. Saldaña, who has worked at the institute since 1988, says it is difficult to continue after the overdoses, but “if we did not work, the patients would die.” Ricardo Lajon, the chief physicist, calls Saldaña “one of the best physicists we have.” The Houston team also praises the physicists. “This was an unfortunate occurrence, which we believe was not foreseeable,” its report said. “However once discovered the actions taken were appropriate and the cause was quickly found. The personnel involved are to be commended.” Prosecutor Cristobal Arboleda acknowledges the peculiarity of having hospital workers accused of second-degree murder continue to treat patients. But he says they must be presumed innocent until found guilty and cannot be fired before the trial. Besides, he says, “administratively, the hospital needs them.” Due to a dispute with the Ministry of Health, the entire radiotherapy department has been operating without a license to deliver radiation, a fact that Multidata is using as part of its legal defense. But if the department were shut down, Arboleda says, “90 percent of the cancer patients in Panama would die.” Saldaña today appears calm for someone who faces the possibility of two to four years in prison. Her mother takes care of her 13-year-old son in a town in the highlands, and that will continue if she goes to jail. The families of two of the dead patients have hired a private prosecutor to pressure the judicial system to convict the physicists, a common practice in Panama. Arboleda expects other civil suits to be filed in Panama depending on the outcome of the criminal trial. Next Page: Lawsuits dismissed for lack of jurisdiction. Lawsuits Dismissed Meanwhile, the families lawsuits against Multidata and MDS have been dismissed in both countries for lack of jurisdiction in Panama and for “forum non-conveniens” in the U.S.. On Jan. 15, 2004, St. Louis County Circuit Court Judge Emmett M. OBrien told the families to re-file their suit in Panama, where the overdoses occurred. Yet Judge Zoila Rosa Esquivel of the First Court of Justice of the Civil Circuit in Panama had dismissed the suit on April 30, 2003, saying that the case could not be pursued in two countries at the same time. In effect, by taking the companies to court in St. Louis County first, the families forfeited the right to take them to court in Panama, according to Esquivels ruling. Now the families are trying again in Panama. Judge OBrien said the companies need to be given the chance to respond to the charges in Panamanian court, before the case can be reconsidered in the U.S. Judge OBriens decision is a victory for Multidata and MDS, which fought to get the suit tried in Panama. Judgements awarded in Panama tend to be low, compensating victims just for actual damages, notes Edgardo Molino-Mola, a former Panama Supreme Court Justice. And since Panamanian judges permit both sides to engage in delaying tactics, such as filing motions with no substance, cases may not be resolved for 10 or 15 years. Regardless of how the court cases end up, some good has come out of the tragedy. The Government of Taiwan donated two new linear accelerators to the National Cancer Institute, to replace its single, aging Cobalt-60
machine, and the Ministry of Health purchased a third linear accelerator that is expected to be installed soon. Training of hospital staff is greatly improved. A foundation led by a prominent Panamanian cancer survivor, Marta Estela C. de Vallarino, has raised hundreds of thousands of dollars that have helped the hospital buy new mammography and endoscopy machines. Was a famous case in which computerized machine gave lethal radiation overdoses to some cancer patients?Therac-25 causes radiation overdoses. The Therac-25 was a radiation therapy machine manufactured by AECL in the 80s, which offered a revolutionary dual treatment mode.
What was the problem with the TheracThe Therac-25, a computerized radiation therapy machine, massively overdosed patients at least six times between June 1985 and January 1987. Each overdose was several times the normal therapeutic dose and resulted in the patient's severe injury or even death.
Who programmed the TheracThe Therac-25 machine was a state-of-the-art linear accelerator developed by the company Atomic Energy Canada Limited (AECL) and a French company CGR to provide radiation treatment to cancer patients.
What are the software errors of the TheracAt least four bugs were found in the Therac-25 software that could cause radiation overdose. One shared variable was used both for analyzing input values and tracking turntable position. Quickly entering the data on the terminal could, therefore, result in leaving the turntable in the wrong position (race condition).
|
A computerized machine that gave lethal radiation overdoses to some cancer patients
zusammenhängende Posts
Urheberrechte © © 2024 berikutyang Inc.
