An email from Forensic Bioformatics alerted me to yet another rather surprising problem with the dreaded magic of DNA.  I, like most lawyers, maintain a somewhat idealized vision of science.  While "junk science" has long been on my radar, "real science" is still on a pedestal, mostly because I want to believe that somebody out there is more dedicated to the truth than to the conviction.  Naive, as we well know, but still true.

Bioforensics, however, burst my bubble again by alerting me to yet another problem with DNA testing, the Context Effect.  As we push to improve identification procedures in light of the overwhelming studies that show that eyewitness IDs are replete with flaws, it appears that many of the same problems exist with DNA.

Picture the DNA analyst given a sample from a suspect in a crime and asked to compare it for a match to a sample found at a crime scene.  That's the scenario address in this White Paper by Bioforensics.  The analyst, knowing what he's dealing with, will try very hard to make the match, and his knowledge of what he's got in hand will influence the way he "interprets" fuzzy areas.  Objective interpretation becomes subjective, and fuzzy becomes clear when the analysts knowledge affects his judgment.

When I asked if there were any good examples of this happening, a big one came back at me.

Examples of context effects in forensic science are unfortunately not difficult to find.  One of the most famous is the Brandon Mayfield case, where several FBI fingerprint experts "matched" his print to fingerprints found on a bag of explosives material found in connection with the Madrid train bombing.  The FBI's fingerprint software listed Mayfield as a potential match, so the analysts were more likely to try to match the prints.  Mayfield was later found to be completely innocent.

That's certainly a fairly clear example of how objective science turns subjective.  Jason Gilder has provided me with a powerpoint showing the problem in more scientific terms, DNA Subjective Interpretations.

The danger in all of this is that an innocent person could possibly be linked to an item of evidence through DNA.  That is precisely what happened to Josiah Sutton in 1998 when he was convicted of rape.  There was a subjective reading of the test results to match Sutton to evidence (an argument could have been made that he was excluded based on the results alone).  There is no doubt, however, that the lab interpreted the evidence profile along side the defendant's DNA profile.  Even when you consider him to be a match to the evidence, the lab incorrectly reported the weight of the evidence.  They stated that the chance of a coincidental match was 1 in 694,000 individuals (the chance of matching the defendant's profile) rather than the chance of matching the item of evidence (which was 1 in 15 individuals). 

This is precisely what we've been led to believe could not and does not happen with DNA.  Yet apparently it does, raising not only another red flag, but one which will elude notice if we rely on the analyst's putatively "objective" testing that, as it turns out, isn't nearly as objective as we think.

The answer to the problem, as discussed in the White Paper, is to keep the DNA analyst in the dark as to whose sample he's using to avoid the human tendency to interpret in favor of a match of the suspect.  This is similar to many of the proposals to improve eyewitness identification procedures to eliminate the same effect.

These problems can be minimized by preventing analysts from knowing the profile of submitted references (i.e., known samples) when interpreting testing results from evidentiary (i.e., unknown or questioned) samples. The necessary filtering or masking of submitted reference profiles can be accomplished in several ways, perhaps most easily by sequencing the laboratory workflow such that evidentiary samples are interpreted, and the interpretation is fully documented, before reference samples are compared. A simple protocol would dictate a separation of tasks between a qualified individual familiar with case information (a case manager) and an analyst from whom domain-irrelevant information is masked.

As we focus on "fixing" problems such as false positive identifications, this reminds us to remember that the same "effects" that impair reliability of one are likely to do the same with others.  It also reminds us that science, even good science, is only as good as the person ultimately doing the analysis, and that any science can end up being reduced to human frailty if not given serious scrutiny.