Medical devices are divided into three classes (I, II, or III) depending on their "risk." Pacemakers and ICDs are considered "high risk" devices (Class III) because people's lives or health may very well depend on the adequate function of these devices.
So how does a new pacemaker go from the drawing board on the inventor's desk at a manufacturer to being ready for implant into the human body? It has to be approved by the Food & Drug Administration (FDA). For Class III devices, the process is called pre-market approval or PMA.
A PMA is actually an application that a company makes to the FDA asking for its new pacemaker or ICD (or other high-risk device) to be commercially cleared for release. In other words, the company wants to be able to start selling its products to the American healthcare consumer.
Part of the PMA process involves collecting data on the device. The FDA is mainly interested in safety (which is self explanatory) and efficacy (which is a fancy term that means the device does what it says it does). Among the information that the FDA wants to see in a PMA are data from scientific studies proving the device is safe and efficacious.
In order to get that data, the manufacturer conducts studies or arranges for third parties to conduct studies on its behalf. The system works on checks-and-balances, that is, the manufacturers run the studies but the FDA has to approve them.
If the company can show the FDA sufficient data backing up its product, the product is cleared for market release. At that point, the paperwork of the PMA, including most of the study data, product manuals, and some correspondence, is made public.
The JAMA paper is the first time (to my knowledge–I may be wrong here) that anyone has gone back and dug through the studies used to support PMAs.
No one is charging that the process for approving pacemakers is flawed. The PMA methodology of submitting scientific study data to support a product's claims seems to be working. The notion that manufacturers conduct the studies and the FDA approves them has not been called into question, either. The problem raised in the Dhruva article in JAMA is this: the scientific data used to support these PMAs is not as scientifically rigorous as it ought to be.
Are they right? The JAMA article makes a pretty compelling case, but we do not know exactly why manufacturers ran the studies the way they did and why the FDA accepted them. There may be more to the story than we are seeing.
Furthermore, there are a lot of pressures right now on the entire healthcare industry. Americans want better medical care at lower prices and faster turnaround times from the FDA. In demanding so much, it appears we have come up short in some areas. The problem may not be that studies were imperfect but that we need to make hard choices: do we want more stringent, scientifically sound cardiac device approvals and, if so, are we willing to take the time and pay the cost?
The prestigious Journal of the American Medical Association (JAMA) just published an article by Sanket Dhruva, MD; Lisa Bero, phD; and Rita Redberg, MD about the FDA approval process for pacemakers. It's pretty typical of what you might read in JAMA in that it's long and very detailed and sort of complicated. You can probably read the article in the original by going to the public library and requesting the December 23/30, 2009 issue of JAMA. (JAMA is published weekly but this is a two-week "holiday" schedule edition.)
But you don't have to. We can give you the Cliff's Notes version.
The first thing you need to understand is how a new pacemaker or defibrillator gets approved by the FDA so that it can be sold commercially. That's so involved it appears in another article here called "The PMA Approval Process." For now, let's just say that there is a rather lengthy involved procedure before the FDA will clear a new cardiac device for market release.
Part of that approval process involves studies which are typically conducted by the device manufacturer. The FDA uses the data or results from those studies when it makes a decision on whether the device is safe and effective.
The Dhruva article went back and pulled these old studies. The authors looked at studies in support of new cardiovascular devices (pacemakers, ICDs, stents) that were conducted from 2000 to 2007 and it "studied the studies."
Scientific articles are very precise, so let's get precise. The Dhruva team looked at studies involving a total of 78 different cardiac devices. That amounted to a total of 123 different studies. What they wanted to do was find out if these studies were scientifically sound. Were they well designed? Were they clear? Did they really measure what they set out to measure?
Most scientific studies used for medical evidence rely on certain well-known scientific principles to help make the studies sound. One of them is called "randomization." A randomized study basically puts people into the study or into different groups in the study "at random." It has been observed that when a study is not randomized, the investigators or people doing the study can sometimes consciously or subconsciously steer certain patients into or out of the study or into or out of specific groups. When that happens, it is called "selection bias."
For example, if you are testing a high-blood pressure medicine and you only test it on very young, fit people, you'll probably get better results than if you test it on old people with a bunch of health problems. Randomization attempts to filter out selection bias.
Most of the time, randomization is done with computer support. Patients are evaluated to see if they meet study criteria (they must meet certain criteria) and then to see if they have exclusion criteria (things that would automatically rule them out) and then they should be entered into the study or into study groups at random.
Another type of scientific system used in studies is the "control." A control group is a group of patients who basically undergoes everything in the study, except they do not get the treatment–they get either no treatment or whatever the current state-of-the-art treatment is. For instance, if you're testing a new blood pressure medication on patients who have high blood pressure, the "treatment group" would get the new medication and the "control group" would get the old medication or whatever is currently considered the state-of-the-art.
The idea is that the control group shows how people would do if they were treated the way we treat patients today. It gives you a benchmark against which you can measure the treatment group.
When patients enter a controlled study, they are assigned to either be in the treatment group or the control group. (Okay, some studies are more complex and have multiple groups, but let's use a simple example here.) The patient should be assigned to the group at random. However, it has been observed that sometimes bias enters the study when doctors and patients know what group they're in. For instance, if a patient knows he is in the treatment group, he may respond better or imagine he's doing better than he really is because he knows he's getting the "improved new medicine." Likewise, doctors doing the study may rate patients better or worse depending on what they think of the treatment. This is another form of bias.
One way to cut that bias out–and make the study more rigorous and scientifically sound–is to "blind" the study. In some studies, the patients are blinded but the doctors aren't. This means the patient does not know if he is in the treatment or control group–he has no clue. But the doctors and other healthcare professionals do. That is a single-blind study.
Sometimes a study can be designed so that neither patients nor those who care for them (doctors, nurses, and so on) know who is in which group. That's a double-blind study. Double-blind studies are considered the most scientifically sound because it reduces bias to the maximum extent possible.
Last but not least, scientific studies have "endpoints." An endpoint is a goal at which you measure the result. For instance, if you test that high blood pressure pill, the endpoint of your study might be blood pressure readings that are at or below 120/80. An endpoint should be something measurable and objective. The idea is that if the patient achieves the endpoint, it's counted; if the patient does not achieve the endpoint, it's counted. At the end of the study, you have an objective result.
In other words, the endpoints of a study cannot be things like "the patient felt better" or "the patient looked better" or "the treatment worked." It has to be measurable and specific.
Now that you know all of this, we're ready to see what the Dhruva team found out and published in JAMA.
1. Most of the studies used to support these cardiovascular devices were not randomized. (In fact, only 27% were randomized, less than one third).
2. Only 14% of these studies were blinded (we're talking any kind of blinding, that is single-blind).
3. About 52% of the studies used a control, but some of these controls were not as rigorous as they could have been.
4. Most of the endpoints (88%) used what is called a "surrogate measure." That means they had an endpoint and it was something that could be measured, but what they were measuring was something that stood in for something else. For instance, in some ICD studies, the study endpoint was that the lead could be implanted successfully. That was a surrogate for proper lead function since successful implant is associated with lead success. Surrogate measures are actually quite common in scientific studies and they are practical, but they have to be used carefully since not all surrogates work well.
5. About three-quarters of the studies (78%) had some numerical discrepancies, that is, the number of patients enrolled in the study did not work out to match the amount of data collected. It often happens that some people enroll in a study and fail to complete it for any number of reasons (their choice, a move, other health problems). However, studies should account for all patients, even if not everyone completes the study.
That is what the JAMA paper found and it is pretty alarming. But let's point out what the JAMA authors did not say.
The paper did not say that the FDA approved any cardiovascular devices that were dangerous. The paper made no links between shortfalls in studies and devices of questionable quality being approved. While the paper does not rule that out, you can't jump to conclusions. The paper mainly calls the FDA and the medical community to action to help improve how cardiovascular devices are approved.
Second, the paper makes a comparison between drug studies and device studies. Drugs are approved in a similar fashion, through testing, but drug tests are typically randomized, controlled, double-blinded study. To be fair, though, device testing is a different animal. You can test a drug by giving one group of people an active pill and the control group a sugar pill. But how do you test a pacemaker? Do you implant pacemakers in one group but not in another? (And wouldn't the people who did not get the pacemaker realize they were the control group?) And how would you blind physicians treating these patients … if a physician had to program a pacemaker, he would know who had a pacemaker!
While I am being overly simplistic, it is a fact that device studies pose a bit more of a challenge than drug studies in terms of meeting randomization, controls, and blinding goals. Drug studies may be the "gold standard," but device studies cannot necessarily match the drug study methodologies.
It is natural to read an article like this and think: whose fault is this? The article does not really say and I cannot say, either, but I can state that manufacturers (who conduct the studies) and the FDA (who allow and accept the studies) must both share some of the blame.
However, the FDA side of the story is not that simple. Right now, the FDA and all organizations involved in health care are facing unprecedented scrutiny and cost containment issues at the same time that the public wants higher standards but faster turnaround times. Not all of these things are simultaneously possible.
Meanwhile, medical device companies are not immune to these same things. They are being scrutinized in the wake of healthcare reform. The economic downturn has impacted them as well. They also feel the public's demands for more advanced technology and better devices, but how can they deliver those things with more regulation and less money?
There is no doubt more to this story. The FDA itself conducted a study which uncovered that it could do a better job in device approvals. No doubt manufacturers are going to feel public pressure.
Meanwhile, as a person with a medical device, there is no reason to believe that your pacemaker or ICD is not safe and effective. Most medical devices are built on the "platform" of older, established technology with just some new innovations and upgrades added. Devices are monitored for problems, and there is no evidence I have ever seen that our current device monitoring system and recalls does not work in terms of identifying and dealing with potentially dangerous devices. Devices approved by the FDA have been studied and evaluated, even if these investigators found shortcomings in the studies.
In other words, your pacemaker or ICD has been tested by the manufacturer, studied, and approved by the FDA, even if some of the methods used were not the most rigorous ones from a scientific standpoint. It may indeed be time to tighten up the approval testing methodologies and to try to find better ways to run device tests that align more closely with the pharmaceutical model.
