Ed. Note: Chris Halkides has been kind enough to try to make us lawyers smarter by dumbing down science enough that we have a small chance of understanding how it’s being used to wrongfully convict and, in some cases, execute defendants. Chris graduated from the University of Wisconsin-Madison with a Ph.D. in biochemistry, and teaches biochemistry, organic chemistry, and forensic chemistry at the University of North Carolina, Wilmington.
At James Otto Earhart’s trial for the murder of Kandy Kirtland, compositional (comparative) bullet lead analysis (CBLA) was used to conclude that the bullets seized from his home and car were “analytically indistinguishable” from one found with her body. FBI agent John Riley testified that he could determine whether or not bullets are from the same box of ammunition.
With respect to all of the 0.22 caliber bullets manufactured in one year, he testified the probability that two 0.22 caliber bullets came from the same batch is approximately 0.000025%, give or take a zero. Upon cross examination, he conceded that he had not taken into account the fact that there were three kinds of 0.22 caliber bullets. Despite having little other probative evidence, the jury convicted Mr. Earhart of Kandy’s murder and he was sentenced to death. Even during Mr. Earhart’s appeals, it would have been difficult to find an expert to question CBLA. Justice, however, was not served.
CBLA can be divided into three phases: analysis, grouping, and inference. The analysis phase uses neutron activation analysis (NAA) or inductively coupled plasma atomic emission spectroscopy (ICPAES) to determine the amounts of up to seven elements found in trace amounts in lead bullets. If the seven elements were found in the same concentrations within error, then the reference and questioned samples were analytically indistinguishable. Both NAA and ICPAES are used in various branches of chemistry and are noncontroversial.
The grouping phase, in which bullets were declared analytically indistinguishable or distinguishable, was more questionable but could have been fixed. The FBI used three different statistical procedures, two-standard deviation test, a range test, and data chaining, the first and third of which probably overstated the strength of the evidence. Data chaining, at the very least, misleads the unwary, but it also increases the chances of false matching. Suppose that bullet A matched bullet B; bullet B matched bullet C; and bullet C did not match bullet A. Yet bullet C could be said to match the group made up of A and B and thus match A. The National Research Council recommended dropping these methods in favor of the successive t-test approach or the T2 test statistic.
However, what prevents CBLA from being a salvageable forensic technique are the four assumptions that underlie the inference phase:
(1) The forensic specimen (about 50 milligrams) is a representative sample;
(2) The hundred-ton batches of lead were homogeneous.
(3) No two molten sources are ever produced with the same composition.
(4) Bullets were distributed in retail markets in a way as to make it unlikely that a chemical concentration of bullets was common in any geographical area.
All four assumptions are either dubious or wrong.
Lead is heterogeneous at the microscopic level, suggesting that (1) might not be correct. When the beginning, middle, and end of one pour was examined, the amount of copper was found to vary by 142%, indicating that (2) is incorrect. One problem with (3) is that tin is not generally present in bullet lead, and cadmium almost never is. The range of silver concentrations is modest, and the range for bismuth is even smaller. Therefore, most of the discrimination is dependent on just three elements, antimony, arsenic and copper. Moreover, the concentrations of the seven elements are not independent, meaning that there are some statistical correlations. The assumption of independence probably increases the possibility of a false match. There is not enough data to be certain about (4).
Empirical testing also challenged the basis of CBLA. According to the California Innocence Project (https://californiainnocenceproject.org/issues-we-face/firearms-analysis/lead-bullet-analysis/), an internal FBI study showed that two batches manufactured months apart had the same composition and that bullets in the same box often had different compositions. The National Research Council noted that “the FBI’s own research has shown that a single box of ammunition can contain bullets from as many as 14 distinct compositional groups.”
And the problems did not end there. The FBI was not forthcoming with respect to people who wished to study their database. Moreover, they may have deleted some of their own data. The language the agents used gave widely differing impressions of the value of the evidence. It ranged from “Could have come from the same box” to “Must have come from the same box or from another box that would have been made by the same company on the same day.” The National Research Council concluded that “The available data do not support any statement that a crime bullet came from a particular box of ammunition. In particular, references to ‘boxes’ of ammunition in any form should be avoided as misleading under Federal Rule of Evidence 403.”
CBLA evidence was generated in 2500 cases before the FBI discontinued this type of analysis in 2005. About 20% of those cases went to trial, and it presumably factored into the thinking of some of the defendants who took plea bargains. After the NRC report and subsequent media investigations, the FBI sent pro forma letters to prosecution and defense organizations. Paul Giannelli was unimpressed: “Yet the letters neither highlighted the problem, nor its significance, and therefore were grossly inadequate means of communication.”
Wendy J Koen concluded that Agent Riley’s estimate of 0.000025% probability of two 0.22 caliber bullets coming from the same batch was “incorrect and not based on reality.”
Mr. Earhart was executed before several state courts stopped accepting CBLA as evidence. Was this a miscarriage of justice? In addition to Ms. Koen another scholar of wrongful convictions, Tucker Carrington, indicated that Mr. Earhart might have been guilty or innocent and that we will never be certain.
For further reading
Giannelli P, “Comparative Bullet Lead Analysis: A Retrospective,” 2010 Criminal Law Bulletin 47(2) 306-315.
Koen W and Houck MM, “Compositional Bullet Lead Analysis,” in Forensic Science Reform Protecting the Innocent, ed. Koen W, Bowers CM. 2017 Academic Press. ISBN 978-0-12-802719-6.
Tobin WA et al., “Absence of Statistical and Scientific Ethos: The Common Denominator in Deficient Forensic Practices” 2017 Statistics and Public Policy, 4:1, 1-11.
National Research Council “Forensic Analysis: Weighing Bullet Lead Evidence (2004) ISBN 0-309-52756-2.