Neuronetics had tried last year (Jan 2007) to get FDA approval for its transcranial magnetic stimulation (TMS) device, but failed spectacularly. Transcanial magnetic stimulation is a way of non-invasively stimulating the brain with electromagnetism. So what the heck happened with the trial? Neuronetics showed the results of a clinical trial of TMS for major depression to a board of FDA advisors.

The primary efficacy outcome in the clinical trial was the reduction in the Montgomery-Asberg Depression Rating Scale (MADRS) symptom score after 6 weeks. Secondary outcomes included changes in the 17- and 24-item Hamilton Depression Rating Scale (HAMD) (pdf). In the trial, the mean decrease in the MADRS scores was about 5.6 points in the active TMS and around 3.2 points in the sham TMS. Unfortunately for neuronetics, the p-value at the six week mark was .058. A p-value less than .05 is usually that difference between sham and active TMS that is considered statistically significant.

Anything higher than a value of .05 means that the trial technically failed to reach statistical significance. Since the MADRS was the primary outcome measure and it had a p value over .05, this was the statistic that caused the FDA to recommend against approval for the TMS device. The FDA could not get over the fact that the mean score on the MADRS was technically not statistically significant over placebo in the mean change in depression scores for patients. Statisticians argued to the FDA that a p-value of .058 is clinically indistinguishable from a p-value of .05, but that wasn't good enough for the board evaluating the TMS device.

Looking at the other statistics this specifc trial, the active TMS did reach statistical significance over sham when measured by the secondary outcome symptom scales (HAMD-17 and HAMD-24). So why is it that the TMS device did reach statistical significance on some measures but not others? Well, the MADRS, HAMD-17 and HAMD-24 (pdf) are 3 different scales with completely different rating items.

The TMS is activating a specific area of the brain (the left dorsolateral prefrontal cortex (LDPFC)). Left dorsolateral prefrontal dysfunction (LDPFC) is associated with pseudodepressive symptoms. These type of symptoms include apathy, indifference, anergia, poor concentration and psychomotor retardation. So what likely happened on the neuronetics trial is that specific ratings scales may load more or less heavily on LDPFC dysfunction. The MADRS may not have reached statistical significance because not enough items measured the pseudodepressed type of symptoms associated with LDPFC dysfunction.

This past november (2007), neuronetics had gotten results from a new trial. Surprisingly neuronetics again used the reduction in mean MADRS scores as the primary outcome measure but after 4 weeks instead of 6 weeks. For this new trial, active TMS does show statistically significant improvement over sham TMS with a p-value of .038. Unfortunately there is only a trend for more improvement evident at six weeks with a p-value of .052. The difference between active and sham just missed statistical significance at the six week mark. Both HAMD-17 and HAMD-24 had a reduction in mean scores that was statistically significant for the active treatment over sham (P=0.006 and P=0.012) at four weeks and at six weeks (P=0.005 and P=0.015).

So in this second trial, they at least got clinical efficacy on their primary measure. However that fact that the MADRS mean score didn't distinguish itself from sham after six weeks is not so good. Will the FDA see past this fact and approve the device? It seems that it will be a close call, but I don't have confidence that it will be approved. Neuronetics really messed up on both of these trials. If they had looked at the specific rating scale items, they probably could have predicted which rating scale to use as a primary outcome measure.

My guess is that there was a greater reduction in symptoms as measured by the HAMD rating scale because the items load more on LDPFC dysfunction than the MADRS scale. Not to mention that neuronetics used treatment resistant depressed patients in the trial. They are much less likely to respond to treatment than normal patients and make the end results of the trial look, well, depressing.