Binder, Rohling and Larrabee Meta-Analytic Review Criticized Again
Neuro law attorneys are very familiar with the meta-analytic review of neurological studies that was authored by Binder, Rohling and Larrabee (Binder 1997). It is a study that is often cited by defense attorneys to support the myth that recovery after mild traumatic brain injury (mTBI) occurs within the first three months, with any subsequent changes in performance being of limited statistical and clinical significance. Binder, et al. found only a small overall effect size and concluded that the size of the overall effect was unimpressive and suggestive of clinical non-significance. That study was later updated by Frencham, Fox and Maybery (Frencham 2005). These meta-analyses are sometimes cited as evidence that mTBI has no lasting effect on neuropsychological status.
In 2009, Pertab JL, James KM and Bigler ED (Pertab 2009) conducted a study designed to clarify opposing conclusions in the mTBI literature by re-analyzing meta-analytic data sets. The study was conducted as there had not been, “A critical review of meta-analytic techniques in mTBI to help clarify discrepancies and conclusions that are drawn from the literature” (Pertab 2009). “Since Binder (1997), there had been a large volume of research published in the literature showing that a sub-set of those who sustained mTBI have long-term significant problems.” (Pertab 2009). Pertab “revisited the data combined in the meta-analysis of Binder and Frencham, specifically addressing four areas: (1) mechanism of injury, (2) diagnostic criteria employed, (3) type of neuropsychological assessment tool employed, and (4) whether symptomatic or non-symptomatic mTBI subjects were assessed separately.” Pertab concluded, after re-analyzing the data, “By using different mTBI characteristics as inclusion/exclusion criteria to re-examine the existing meta-analyses, the current results indicated significant statistical heterogeneity (a) the effect sizes of neuropsychological measures employed in the post-acute phase and marked qualitative heterogeneity in (b) the criteria used to define mTBI and mTBI severity, and (c) the populations and mechanisms of injury from which the mTBI samples were selected.”
As indicated in the introduction, we are not disputing the overall conclusions in mTBI meta-analyses where it appears that the majority of individuals who sustain an mTBI suffer no significant neuropsychological residue after three months. In fact, the largest effect size in this re-characterization was moderate, with most being small to trivial. It is well known and widely understood that group inferential statistics can obscure important individual differences.
As indicated in the introduction, the Binder, et al. and Frencham, et al. studies have been cited to support the non-effect of any lasting sequelae of mTBI as a general principal for the outcome of all mTBI’s. From the total group standpoint, that likely remains a true statement, but not one that necessarily applies to an individual within that sample.
If small sub-samples of clinically symptomatic mTBI patients do exist within a broader sample of non-symptomatic subjects where the mechanism of injury differs between studies or the methods of assessment differ, etc., the averaging process of meta-analysis is likely to obscure, rather than highlight, clinically relevant features of any minority sample within an mTBI group that may have residual symptoms and/or deficits.
Following the publication of Pertab (2009), Binder, Rohling and Larrabee updated their meta-analysis to include studies through 2002, observing similar results to Binder, et al. (1997). Rohling, et al. (2011) have now critiqued the methods used by Pertab (2009) and performed even more meta-analytic comparisons on the original twenty-five studies, again concluding that there were no lasting cognitive effects of mTBI.
Now, Bigler (2013) and his colleagues have again reviewed Binder (1997), Frencham (2005) and Rohling (2011). In their new paper, Bigler responds to the Rohling (2011) critique, reaffirming the original findings of Pertab (2009), and providing additional details concerning the flaws in prior meta-analytic mTBI studies and the effects on neuropsychological studies. “Reaffirmed Limitations of Meta-Analytic Methods in the Study of mTBI: A Response to Rohling, et al.”, Bigler ED, Farrer TJ, Pertab JL, Kelly JM, Petrie JA and Hedges DW, The Clinical Neuropsychologists (2013).
Bigler (2013) again acknowledges that:
The majority of mTBI patients over time enjoy a full functional return to their pre-injury baseline and their recovery follows a quick and rather benign course. However, some mTBI patients do experience persisting neuro cognitive and neural behavioral deficits and symptoms, even after controlling for such factors as depression and potential response bias.
Bigler (2013) responds further:
While such a statement may apply to the majority of mTBI participants who experience a positive outcome, this perspective does not match contemporary animal-model neuropathological investigations and human neuroimaging and neuropathological studies of mTBI which do indicate that permanent changes may occur after mTBI.
A major point of Pertab (2009) was that meta-analytic technique as applied to group data will not detect embedded impaired performance by the few, because it becomes averaged and therefore hidden within the overall group mean.
The Rohling (2011) critique strongly disagrees with the idea that embedded effects may go undetected in a meta-analysis and in fact cite literature demonstrating, ... in some cases, low-powered individual studies of various medical topics have failed to show differences, while meta-analytic studies of the same topics have revealed significant results.
Bigler (2013) also addresses some important issues in the field of mTBI. Discussing the sensitivity or lack of sensitivity of neuropsychological measures to persistent effects of mTBI, Bigler (2013) notes, “Regardless of the type of neurologic and/or psychiatric disorder, neuropsychological assessment faces its biggest challenge in detecting subtle impairment,” citing B. Johnson, et al. (2011).“An fMRI investigation showing persistent deficits in the brain’s default network in athletes with mTBI, explicitly state that: ‘Neuropsychological testing and conventional neuroimaging techniques are not sufficiently sensitive to detect’ neurological changes.”
On this point, Bigler (2013) includes:
All of this suggests that traditional neuropsychological techniques, including those used in the twenty-five source studies for meta-analyses discussed herein have inherent sensitivity problems after the acute recovery time period; issues of sensitivity not discussed by Rohling, et al. in offering their conclusions. If the traditional neuropsychological measure is insensitive after the acute time frame, what should be used? The answer is likely to be found in the integration of functional neuroimaging with cognitive processing speed tasks that measure performance in milliseconds, a more direct representation of brain processing speed.
Bigler (2013) also criticizes Rohling’s failure to separate mTBI from complicated mTBI cases. Bigler (2013) finds:
Rohling and co-authors state that their conclusions were not necessarily applied to those with complicated mTBI. Is that conclusion not being made because in mild-complicated TBI there is objective indication (i.e., a bio-marker?) of neuro injury (see Green, Koshimori and Turner, 2010) and that mild-complicated TBI participants have been shown and even classified by some as having sustained a moderate TBI? As already stated, the distinction of whether mild-complicated TBI was present or not was not uniformly addressed whatsoever in the twenty-five source studies. In regards to technology, CT imaging - which was the basis for mild-complicated TBI classification for essentially all studies prior to the early 1990's - is the least sensitive of the imaging technologies for detecting hemorrhage which is a typical marker for the mild-complicated TBI distinction. ... Because current-day technology did not exist during any of the twenty-five source studies used to perform the meta-analyses, the issue of objective indications of neuro injury simply cannot be addressed, including the mild-complicated issue.
Rohling, et al.’s conclusions assume that neuropsychological measures represent the best approach for detecting impairment. However, as shown above, neuropsychological methods may simply be ineffective (i.e., lack of proper sensitivity) in detecting the subtleties of any type of residual impairment following mTBI from the sub-acute to chronic phase.
With regard to their initial evaluation of the twenty five source studies, Bigler (2013) write:
Given everything discussed to this point, the real mistake of Pertab et al. (2009) was that even after they demonstrated the various assumption violations and listed all of the limits of the Binder et al. (1997) and Frencham et al. (2005) meta-analyses, they went ahead and used some of the same studies in their metaanalysis. In fact Pertab et al. should have stopped and offered no additional analyses of the 25 mTBI articles because of the study quality issues and assumption violations. The real message of Pertab et al. is that past mTBI studies are problematic for all the reasons discussed herein, and if meta-analysis is to be used to address the mTBI problem, the highest quality of research and clinical standards need to be used.
Bigler (2013) concludes:
In conclusion, we show methodological violations, all of which were originally identified in Pertab (2009) that limit Rohling (2011) critique where their findings merely perpetuate type-II statistical errors. As already stated, it is an untenable and non-supportable position that neuron-based symptoms and deficits do not persist in some individuals who experience mTBI. (Citations omitted). Current neuroimaging and neuropathological findings indicate that mTBI results in permanent structural and physiological injury to the brain in some individuals. As such, neuropsychology’s goal should be to refine its methods, including its interface across disciplines to detect those individuals who have experienced an mTBI and in whom brain dysfunction and its symptoms persist, not to continue arguing that permanent deficits do not occur from mTBI.*
* It should be noted that Larrabee (2013) published a rebuttal to Bigler (2013) in an article entitled “Meta-Analytic Methods and the Importance of Non-TBI Factors Related to Outcome in Mild Traumatic Brain Injury: Response to Bigler, et al. (2013), Larrabee GJ, Binder LM, Rohling ML and Ploetz DM, The Clinical Neuropsychologist (2013).
Search Post in Google | MSN | Yahoo! | Google Blogs | Technorati