Back to Messageboard | Home | Site Map  

Flaws in Lancet 2006 study - new report
Posted by ALP February 10 2008, 11:12 » Uploaded 10/02/08 12:38  

New 50-page research paper from Professor Michael Spagat of Royal Holloway, University of London:

Ethical and Data-Integrity Problems in the Second Lancet Survey of Mortality in Iraq

February 2008

Abstract: I consider the second Lancet survey of mortality in Iraq published in 2006. I give evidence of ethical violations against the survey’s respondents including endangerment, privacy breeches and shortcomings in obtaining informed consent. Violations to minimal disclosure standards include non-disclosure of the survey’s questionnaire, data-entry form, data matching anonymized interviewer IDs with households and sample design. I present evidence suggesting data fabrication and falsification that falls into nine broad categories: 1) non-disclosure of key information; 2) implausible data on non-response rates and security-related failures to visit selected clusters; 3) evidence suggesting that the survey’s figure for violent deaths was extrapolated from two earlier surveys; 4) presence of a number of known risk factors for interviewer fabrication listed in a joint document of American Association for Public Opinion Research and the American Statistical Association; 5) a claimed field-work regime that seems impossible without field workers crossing ethical boundaries; 6) large discrepancies with other data sources on the total number of violent deaths and their distribution in time and space; 7) two particular clusters that appears to contain fabricated data; 8) irregular patterns suggestive of fabrication in claimed confirmations of violent deaths through death certificates and 9) persistent mishandling of other evidence on mortality in Iraq presented so as to suggest greater support for the survey’s findings from other evidence than is actually the case. (Intro & supporting links) (Paper - PDF file)

COMMENTS Post comment


Comment 01 – ALP February 10 2007, 13:21

The following comparisons are not included among these L2 [Lancet 2006] footnotes despite being far more relevant to the case of Iraq than the articles cited. They all suggest substantially more than 20% coverage for media-based monitoring in Iraq, contrary to the L2 claim that “we can find no conflict situation where passive surveillance recorded more than 20% of the deaths measured by population-based methods”:

1. L1, conducted by mostly the same authors as L2, estimated 56,700 violent deaths of civilians plus combatants outside Al-Anbar governorate (a large outlier in L1) compared to 17,687 deaths of civilians in Iraq outside Anbar recorded by IBC for the L1 period.

2. The ILCS estimated 24,000 war-related deaths of civilians and combatants compared to an IBC figure of about 14,000 deaths of civilians for the ILCS coverage period.

3. A study of Afghanistan done by colleagues of the L2 authors at Johns Hopkins compared mortality estimates from a population-based survey with a body count based on media monitoring that used methods that inspired IBC’s approach.107 The survey found 5,576 killed. This compares to a media-based count of 3,620 civilians killed.

I draw two conclusions from the material discussed in this section. First, L2 is much more of an outlier in the Iraq mortality literature than would be suggested by L2’s treatment of the literature. Second, the treatment of the evidence on Iraq mortality in L2 displays a persistent pattern of data and information falsification.

[p48 of PDF paper]

Comment 02 – ALP February 10 2007, 14:40

Jon Pedersen (of ILCS) has publicly criticised the Lancet study, but hasn't yet come in for the kind of character assassination (mainly from Medialens) experienced by Spagat and IBC. He's also widely respected - even, apparently, by world-class experts such as Stephen Soldz and Les Roberts.

In an email to me (4/12/06) Pedersen wrote, in praise of the Main Street Bias (MSB) research, by Spagat et al:

I very much agree with the MSB-team that there is some main stream bias, and that this is certainly an important problem for many surveys - not only the Iraq Lancet one. [...] The MSB people have come up with some intriguing analysis of these issues.

And the New Scientist quotes Pedersen as saying that the methodological criticisms coming from people such as Spagat will probably benefit the field:

It was not easy for the Johns Hopkins and Iraqi team to have their work dissected in public (see New Scientist, 25 April, p 44), but the process may ultimately benefit the field, says Pedersen. The discussion has prompted many researchers to think about how potential biases such as those outlined by Spagat could creep in. "The debate has had a positive effect in many ways," he says.(From issue 2615 of New Scientist magazine, 01 August 2007)


Post comment

Display email
  Don't display    Display
Lifespan of comment   Delete after 3 weeks    Keep permanent if post is permanent  


Optional link URL:
eg ""
Optional link text:
eg "Media Hell"


  Messageboard Back to top