Research bias in ME/CFS
Research in every field deals with bias. Research into myalgic encephalomyelitis/chronic fatigue syndrome faces specific kinds of bias issues as it navigates significant gaps in scientific knowledge and several opposing viewpoints.
- 1 Emphasis on randomized controlled trials and systematic reviews
- 2 Reliance on unpublished data and withholding outcomes
- 3 Clinicians' disbelief in unsuccessful trials
- 4 Bias and inadequate peer review
- 5 Editorial bias
- 6 Failure to retract, withdraw or correct flawed research
- 7 Prevalence
- 8 Bias recognition
- 9 Notable studies
- 10 Articles and blogs
- 11 Possible causes
- 12 See also
- 13 Learn more
- 14 References
Emphasis on randomized controlled trials and systematic reviews[edit | edit source]
Following an Evidence-Based Medicine (EBM) model gives greater weight to systematic reviews and randomized controlled trials. However, some systematic reviews and a number of randomized controlled trials have been shown to have significant flaws, including a failure to follow pre-defined procedures or standards, and altering the primary outcome(s) part way through unblinded trials either without adequate justification or without a referring to the change, and failure to comply with prespecified trial plans, serious failings in ethics including false ethical approval statements in the publication of more than a dozen research outputs about children with CFS, and research incompatible with child protection and safeguarding, e.g. in the SMILE trial. The most influential examples of these are the Cochrane review of exercise therapy for CFS, which Cochrane initially planned to withdraw after admitting it did not meet the set standards, and the PACE trial, which is the largest treatment trial for CFS, and resulting in a lengthy legal battle for the release of the original primary outcome data including the results of the six-minute walk test.
PACE trial[edit | edit source]
Davenport et al. (2018) state the lack of randomized controlled trials in ME/CFS research mean that an evidence-based model can be unduly influenced by a very small number of studies, for example the PACE trial found that treating ME/CFS with graded exercise therapy or cognitive behavioral therapy did not result in clinically significant improvements - but this trial heavily influenced the clinical use of those treatments. The PACE trial was so problematic that it was used an the example for Prof. Bruce Levin's presentation How Not To Conduct A Randomized Clinical Trial, and The Lancet received open letters signed by over 100 people, including researchers and academics, calling for the trial to be retracted.
Cochrane systematic review[edit | edit source]
The highly influential Cochrane review of exercise therapy for chronic fatigue syndrome was also found to have significant flaws. Cochrane announced it would be withdrawn, but when the authors refused to agree Cochrane then changed its position and published repeated amendments and editorial notes instead. Cochrane allocated to reviews about chronic fatigue syndrome to its Common Mental Disorders group, despite the long standing classification of CFS as a neurological disease by the World Health Organization, and the absence of any psychological symptoms in any of the differing diagnostic criteria. Cochrane have withdrawn the individual patient data used in the exercise review.
Reliance on unpublished data and withholding outcomes[edit | edit source]
A number of journals have disregarded their data access policies or published studies that did not comply with them, resulting in editorial corrections or withdrawn data rather than retractions, for example in a PLoS PACE trial follow up publication and the Cochrane systematic review of exercise as a therapy for chronic fatigue syndrome. When outcome swapping has taken place during a trial, in some cases there has either been no mention of the change in the published trial results, or the change has been referred to but the data has been omitted, and researchers have refused to provide it to other academics, for example in the PACE trial which received £5 million in funding.
Clinicians' disbelief in unsuccessful trials[edit | edit source]
Some clinicians appear to have become so convinced that a treatment must work during a trial that a null or negative response to the treatment has been misattributed to a patient mistake. Examples include the FINE trial, which was the sister trial to the PACE trial but for severely ill patients, where a null result was published along with a comment that some nurses delivering the treatment felt that "the bastards [patients] just don't want to get better". Clear conflicts of interest with representing both insurance companies in dismissing claims for illness insurance, in private training companies run by clinician researchers, and in prior published popular self-help books for chronic fatigue or CFS commonly occur, and are often not disclosed in publication, including in all of the three PACE trial principle authors, at least one other PACE trial author and the majority of biopsychosocial treatment studies. A publication by Simon Wessely, one of the clinicians who delivered treatment in the PACE trial, described the management of the trial using a comparison of steering a ship to its pre-determined destination.
Bias and inadequate peer review[edit | edit source]
Many journals do not have open peer reviews, however some have been found to have clearly biased peer reviewers, or authors are invited to suggest "independent" peer reviewers but suggest biased ones. In one published peer review, the peer reviewer commented that they did not read past the abstract, but the research was published regardless. In another case, an author submitted an invited editorial and was asked to nominate independent peer reviewers but nominated only those with well-documented adherence to the same paradigm of ME/CFS. In this case, the journal rejected the publication, and the author then published it on a popular health website which did not require peer review.
A series of publications about chronic fatigue syndrome in UK school children were published using the same invalid ethics approval number without this being picked up in peer review. This lead to an investigation into pediatrician Esther Crawley by the University of Bristol, and published corrections for all articles - none of which had an ethics approval. The publications were reclassified as "service evaluations" which did not require ethics approval.
Editorial bias[edit | edit source]
A number of highly influential journals have published editorials either strongly in favor of controversial treatments, criticizing key publications which challenged the journals' previous ideas about ME/CFS, or harshly criticizing advocates, patients and even some researchers or clinicians who raised genuine scientific queries or made scientific criticisms of research; such criticism has continued for decades despite changes in editorship. Journals criticized for editorial, publication or other bias including the BMJ, the Lancet, which published the PACE trial; and Archives of Disease in Childhood, which published an extremely lengthy editorial correction rather than retract the SMILE trial, a pediatric trial of the controversial Lightning process, which is based on neurolinguistic programming.
Failure to retract, withdraw or correct flawed research[edit | edit source]
A number of journals plus Cochrane, have published research which failed to meet their own standards, including clinical treatment trials of graded exercise therapy and cognitive behavioral therapy for ME/CFS, and have then failed to retract some which could not be appropriately corrected in a way that met the journal standards. A significant number of editorial corrections, expressions of concern or corrections by authors have been published, merged data from a successful feasibility study into a treatment trial; the journal's editor described the correction as "extensive". Cochrane announced it would withdraw a systematic review of exercise as a therapy for CFS after a detailed complaint described its failure to meet Cochrane's own standards and incorrect use of the GRADE framework required by Cochrane - but then reversed this decision after the authors refused to agree to the withdrawal of the review.
The Lancet responded to multiple open letters requesting an investigation into the PACE trial and an independent re-analysis of the data by refusing those requests and continued to do so after the full PACE trial data was released which showed that patient's six-minute walk test results had barely improved, and the 13% of those taking part in the trial met at least one of the "recovery" outcomes at the start of the trial due to the definition for recovery being lowered part-way through the trial.
Prevalence[edit | edit source]
Bias recognition[edit | edit source]
Geraghty et al. (2019) looked at evidence behind the popular cognitive behavioral model of ME/CFS, and found that a number of research papers contained significant evidence that did not support the model combined with contradictory abstracts or conclusions at odds with the authors' results, and notable selective interpretation of evidence, such as ignoring any contradictory research.
Sonia Lee (PhD) assessed research waste in ME/CFS by comparing the PACE trial with two cellular trials, finding that "selective reporting", an indicator of research waste, was greater in the PACE trial; however this paper (2017) was not peer reviewed.
Nacul et al. (2017) discussed the effects of selection bias, highlighting how psychosocial/biopsychosocial studies frequently used the broadest diagnostic criteria - the Oxford criteria - and did not break results down by subgroup. The use of the Oxford criteria lead to a 100-fold increase in disease prevalence compared to the Canadian Consensus Criteria (CCC); with 14 out of 15 patients meeting the Oxford criteria typically failing to meet the Canadian Consensus Criteria for ME/CFS.
Notable studies[edit | edit source]
- 1982, Suffering and the Social Construction of Illness: The Delegitimation of Illness Experience in Chronic Fatigue Syndrome(Abstract)
- 2005, Chronic Fatigue Syndrome: Editorial Bias in The Lancet(Abstract) - Not peer-reviewed
- 2011, Chronic Fatigue Syndrome: Editorial Bias in the British Medical Journal(Abstract)
- 2017, Bias, misleading information and lack of respect for alternative views have distorted perceptions of myalgic encephalomyelitis/chronic fatigue syndrome and its treatment(Full Text)
- 2017, How have selection bias and disease misclassification undermined the validity of myalgic encephalomyelitis/chronic fatigue syndrome studies?(Full text)
- 2017, Research Waste in ME/CFS(Full text) - not peer reviewed
- 2018, Checking our blind spots: current status of research evidence summaries in ME/CFS(Abstract) - Editorial
- 2018, Are ME/CFS Patient Organizations "Militant"? : Patient Protest in a Medical Controversy.(Abstract)
- 2019, The ‘cognitive behavioural model’ of chronic fatigue syndrome: Critique of a flawed model(Full text)
Articles and blogs[edit | edit source]
- 2017, MEA Review: The SMILE trial - a lesson in how not to conduct clinical trials in people with ME/CFS(Full text) - by Dr Charles Shepherd, ME Association medical advisor
Possible causes[edit | edit source]
See also[edit | edit source]
- Retractions, corrections and expressions of concern in chronic fatigue syndrome
- Clinical trial terminology (category)
- Research terminology (category)
- Bruce Levin
- Wessely school
- Ethical issues
- Ben Goldacre
Learn more[edit | edit source]
References[edit | edit source]
- Davenport, Todd E; Stevens, Staci R; VanNess, J Mark; Stevens, Jared; Snell, Christopher R (July 17, 2018). "Checking our blind spots: current status of research evidence summaries in ME/CFS". British Journal of Sports Medicine: 1–2. doi:10.1136/bjsports-2018-099553. ISSN 0306-3674.
- World Health Organization (2016). ICD-10.
- Tuller, David (May 31, 2020). "Trial By Error: Letter to BMJ Paediatrics Open About that CBT-Music Therapy Study". Virology blog. Retrieved September 18, 2020.
- British Medical Journal Publishing Group (November 1, 2019). "Correction: unidentified chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics". BMJ Open. 9 (11): e000252corr1. doi:10.1136/bmjopen-2011-000252corr1. ISSN 2044-6055. PMID 31712351.
- BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health (February 1, 2020). "Correction: Obesity in adolescents with chronic fatigue syndrome: an observational study". Archives of Disease in Childhood. 105 (2): e1–e1. doi:10.1136/archdischild-2016-311293corr1. ISSN 0003-9888. PMID 31969354.
- Dowsett, E. G.; Colby, J. (October 11, 1997). "Chronic fatigue syndrome in children. Journal was wrong to critizise study in schoolchildren". BMJ : British Medical Journal. 315 (7113): 949. ISSN 0959-8138. PMC 2127623. PMID 9361553.
- Geraghty, Keith; Jason, Leonard; Sunnquist, Madison; Tuller, David; Blease, Charlotte; Adeniji, Charles (January 1, 2019). "The 'cognitive behavioural model' of chronic fatigue syndrome: Critique of a flawed model". Health Psychology Open. 6 (1): 2055102919838907. doi:10.1177/2055102919838907. ISSN 2055-1029.
- Lee, Sonia (May 7, 2017). "Research Waste in ME/CFS". bioRxiv: 133926. doi:10.1101/133926.
- Goudsmit, Ellen M; Howes, Sandra (2017), "Bias, misleading information and lack of respect for alternative views have distorted perceptions of myalgic encephalomyelitis/chronic fatigue syndrome and its treatment", Journal of Health Psychology, 22 (9): 1159-1167, doi:10.1177/1359105317707216
- Tuller, David (September 17, 2020). "Trial By Error: A Short Talk for UK Docs and Researchers". Virology blog. Retrieved September 18, 2020.
- Goudsmit, Ellen M; Stouten, Bart; Howes, Sandra (October 5, 2005). "Editorial bias in the Lancet". www.axfordsabode.org.uk. ME Research Online. Retrieved April 20, 2019.
- Blease, Charlotte; Geraghty, Keith J. (September 2018). "Are ME/CFS Patient Organizations "Militant"? : Patient Protest in a Medical Controversy" (PDF). Journal of Bioethical Inquiry. 15 (3): 393–401. doi:10.1007/s11673-018-9866-5. ISSN 1176-7529. PMID 29971693.
- Brown, Nick (July 2019). "Editor's Note on Correction to Crawley et al 2018". Archives of Disease in Childhood.
- BMJ Publishing Group Ltd and Royal College of Paediatrics and Child Health (October 1, 2019). "Notice of correction and clarification: Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial". Archives of Disease in Childhood. 104 (10): e4–e4. doi:10.1136/archdischild-2017-313375corr1. ISSN 0003-9888. PMID 31296601.
- Nacul, Luis; Lacerda, Eliana M; Kingdon, Caroline C; Curran, Hayley; Bowman, Erinna W (March 1, 2017). "How have selection bias and disease misclassification undermined the validity of myalgic encephalomyelitis/chronic fatigue syndrome studies?". Journal of Health Psychology: 1359105317695803. doi:10.1177/1359105317695803. ISSN 1359-1053. PMC 5581258. PMID 28810428.
- Ware, Norma C. (December 1992). "Suffering and the Social Construction of Illness: The Delegitimation of Illness Experience in Chronic Fatigue Syndrome". Medical Anthropology Quarterly. 6 (4): 347–361. doi:10.1525/maq.1992.6.4.02a00030. ISSN 0745-5194.
- Goudsmit, Ellen M.; Stouten, Bart (2011). "Editorial bias in the British Medical Journal". Journal of Chronic Fatigue Syndrome. 12 (4): 47–59. Retrieved January 23, 2019.
- Edwards, Jonathan (March 28, 2017). "PACE team response shows a disregard for the principles of science". Journal of Health Psychology. 22 (9): 1155–1158. doi:10.1177/1359105317700886. ISSN 1359-1053.
The PACE trial of cognitive behavioural therapy and graded exercise therapy for chronic fatigue syndrome/myalgic encephalomyelitis has raised serious questions about research methodology. An editorial article by Geraghty gives a fair account of the problems involved, if anything understating the case. The response by White et al. fails to address the key design flaw, of an unblinded study with subjective outcome measures, apparently demonstrating a lack of understanding of basic trial design requirements. The failure of the academic community to recognise the weakness of trials of this type suggests that a major overhaul of quality control is needed.
- Shepherd, Charles (October 12, 2017). "MEA Review: The SMILE trial – a lesson in how not to conduct clinical trials in people with ME/CFS". ME Association. Retrieved April 21, 2019.
randomized controlled trial (RCT) - A trial in which participants are randomly assigned to two groups, with one group receiving the treatment being studied and a control or comparison group receiving a sham treatment, placebo, or comparison treatment.
Canadian Consensus Criteria (CCC) - A set of diagnostic criteria used to diagnose ME/CFS, developed by a group of practicing ME/CFS clinicians in 2003. The CCC is often considered to be the most complex criteria, but possibly the most accurate, with the lowest number of patients meeting the criteria. Led to the development of the International Consensus Criteria (ICC) in 2011.
International Classification of Diseases (ICD) - A system of medical diagnostic codes, created by the World Health Organization (WHO), to classify diseases and other health related conditions for the purpose of international diagnostic consistency. By having common diagnostic codes around the world, health researchers are better able to quantify and track disease burdens. The most current version is called ICD-11. (Learn more: www.who.int)
subjective outcome An outcome of a clinical trial that depends on the judgement or opinion of the assessor or patient, e.g. asking if fatigue has increased or decreased "a little" or "a lot", patient questionnaires like the Chalder Fatigue Scale, and other patient-reported outcome measures (PROMs).