Just Say No, Forensics Edition

Jeffrey Gamso, in his inimitable way of tying loose ends together, went from 9th Circuit Judge Alex Kozinksi to F.W. Murnau’s missing head to the decision in People v. Collins.  And if that doesn’t give you whiplash, reading Brooklyn Supreme Court Justice Mark Dwyer’s opinion will make your head spin.

Justice Dwyer begins with something that ought to be obvious to all, but few judges would openly admit:

This court recognizes that judges are, far and away, not the people best qualified to explain science. That observation is doubly applicable when novel scientific techniques are at issue—and that of course is precisely what Frye analysis involves. But courts are bound to do their best.

I just want to bask in the warm glow of metacognition for a moment. Damn, he’s good.  From there, Justice Dwyer launches into a general dissertation about DNA that will give you a headache. At least it did me.  He then reaches the issue in the case, whether a new-fangled way of using DNA, called High Sensitivity Analysis, together with a new program to do it, the Forensic Statistical Tool, should be admitted into evidence.

The TL:dr as to these new DNA weapons is that sometimes, there isn’t enough DNA available to use the usual, and approved, means of analysis, so forensics guys came up with a way to take inadequate samples and make them work. Or so they say.

It’s a flawed process, resulting in additions and subtractions to the 15 requisite loci that skews the outcome.  But by reducing the statistical certainty from one in a gazillion to a mere one in a bit of gazillion to account for the fact that it’s flawed, the prosecution experts explain that they’ve been sufficiently conservative to cover up the flaws.

Because juries will be far less likely to embrace the idea that the certainty of 1 in 19,000 isn’t nearly as certain as 1 in a trillion.

Justice Dwyer takes an aside into the standard by which scientific evidence is determined to be admissible in New York.

This opinion has cited Frye, yet has not explained it. But in New York legal circles, little explication is required. Novel methods of scientific analysis can produce admissible evidence only if the relevant scientific community generally (though not necessarily unanimously) considers those methods to be reliable. Frye v. United States, 293 F. 1013 (D.C. Cir. 1923).

Put another way, a court assessing the admissibility of evidence under Frye is not charged with deciding the validity of novel scientific procedures. It would hardly be sensible to assign that task to the judiciary, most of which is as patently unqualified to perform the task as is this court. Judges should be “counting scientists’ votes,” and not “verifying the soundness of a scientific conclusion.” Parker v. Mobil Oil Corp., 7 NY3d 434, 446-47 (2006), quoting People v. Wesley, 83 NY2d 417, 439 (1994)( Kaye, C.J., concurring).

For those who may not be aware of what he’s getting at, the alternative standard, Daubert, would require the court to make an independent conclusion that the science is valid, which, as Justice Dwyer notes, isn’t exactly a judge’s strong suit.

Where things really go crazy wild, however, is that the admissibility of these new methods has already been allowed:

One New York trial court has, after a Frye hearing, upheld the use of high sensitivity DNA analysis. People v. Megnath, 27 Misc 3d 405 (Sup. Ct. Queens Co. 2010) ( Hanophy, J.). Another has, after a Frye hearing, held that the FST approach to mixture analysis is generally accepted in the DNA community. People v. William Rodriguez (Sup. Ct. NY Co. October 24, 2013) (Carruthers, J.) (unreported).

This is where the rubber meets the road, and why this decision is important.  Most judges, faced with the issue of whether to admit new scientific evidence, would look to whether someone before them held a Frye hearing, made a decision, and then punt.  Another judge let it in? Boom. Move along, counselor. It’s a done deal.

Judge Dwyer did not punt.

With all respect to the authors of those decisions, this court has counted the scientists’ votes differently, and disagrees with their conclusions.

The law is saddled with decades of bad forensic science.  Everything from the beloved dog sniffs to fingerprints, all of which has come under scrutiny over time only to learn that the certainty of this evidence was anything but.  Indeed, some of it, bite mark analysis for example, has proven to be a total crock, while others, fingerprints, have been modified over the years to overcome its original inadequacies, and yet remains subject to huge flaws.

But everybody knows that fingerprints are magic.  And doggies are special because they can smell everything.

Gamso concludes:

[Justice Dwyer] did something that almost never happened with expert evidence: he did it properly. That’s worthy of some note. Frye is readily abused. Fingerprints, for instance, pass the Frye test and are admitted in court because the relevant community of experts, fingerprint examiners, pretty much conclusively believes they’re meaningful. (They ought to flunk a fair Daubert challenge since they have no demonstrated reliability, but they’re fingerprints f’rgodssake, so they’re universally admitted because everybody knows fingerprints.)

To the uninitiated, what happened in Collins may seem, well, normal, expected even, since it’s perfectly reasonable to expect judges to do their job.  But what Justice Dwyer did was remarkably bold, maybe even exceptional.  He said “no” to a new wrinkle in DNA analysis, and he did so after other judges said, “sure, why not?”

With pathetic regularity, the by-product of bad forensic science is revealed when defendants convicted with the utmost certainty are later exonerated.  We know what’s wrong. Hell, even Judge Kozinksi says so, and he’s a judge.

But leaping from what we all know to actually refusing to admit something the government swears is all science-y is huge. Now, if we can only get a dog sniff case before Justice Dwyer.  Junk science has got to stop, and the fact that Justice Dwyer is willing to take the heat for doing the hard, and unpleasant, work of judging deserves both recognition and appreciation.

21 thoughts on “Just Say No, Forensics Edition

  1. John

    As someone who does PCR professionally and has worked at a company that produces forensic DNA test kits, I read the People v Collins explanation. I would like to see the papers on “High Sensitivity Analysis” whose existence is mentioned without any specific citations given*. Also, publishing is not necessarily validation for industry, medical or forensic purposes. I am extremely skeptical of this and, based on what I read, agree with everything the defense witnesses said.

    The premise behind HSA is a commonly used one in research, but the fact that only one lab is using it and no company is producing kits FOR FORENSICS suggests to me that no company’s developers are satisfied with its reliability enough to stake their reputations on it. When I use the triplicate or quadruplicate amplification trick to control for stochastic effects, I am open about not being able to trust any single sample “but all 8 experimental samples are going in the same direction relative to controls.” It is not “beyond all reasonable doubt” for any single sample.

    *I am also reminded why I hate reading court briefs about science – why can’t they give real citations? Is there any bibliography I missed in there?

      1. John

        Yes, I see at some level (maybe not by a defense attorney’s standards). I became acquainted with what passes for scientific evidence in courts a long time ago. I know for a fact there are a few people working in the legal system who are savvy enough about science to not fall for this crap, but they are not nearly common enough.

      2. Jonathan Edelstein

        In the Megnath case, there was a 1400-page Frye hearing with scientists testifying on each side and numerous papers and treatises admitted into evidence. And after all that, the judge decided that Frye analysis didn’t apply because low copy DNA analysis wasn’t a “novel” scientific technique — it was just STR amplification done three more times.

        I wonder what’s been happening with Megnath’s appeal, BTW – it’s been pending since 2010.

        1. SHG Post author

          Don’t know why, but this reminds me of a hearing I did before Judge Owen in SDNY years ago, where the issue was whether my Fuchanese client understood Miranda warnings in Mandarin. After all the expert testimony, Judge Owen told our expert to give the warning in Fuchan dialect, then Mardarin dialect. He then paused for a moment, and said: “sounds the same to me. Denied.”

        2. John

          It was not just 3 additional cycles. It was statistically distinguishing DNA amplified from multiple individuals. Effectively, it was hard to distinguish the sample from background human DNA contamination.

          Anyone who says adding more PCR cycles is not novel and therefore does not require new validation has little or no PCR experience. Companies call it the “dynamic range” in their products: how much you can dilute your samples (and increase cycle number) before the results start changing. This case hinged on relative abundance of signal and background, and I have seen over 100-fold changes in the final results when the abundance of the target drops enough to need 5 additional cycles.

          1. Patrick Maupin

            What is your problem here? Obviously, both DNA amplification and homeopathy are proven science with deep roots, and any genius can see they belong together.

      3. Jonathan Edelstein

        Actually, upon rereading, I see that Judge Hanophy did do a Frye analysis in Megnath in addition to finding that low copy DNA wasn’t novel. I still think he had a somewhat skewed view of what “general acceptance” means, but he did at least go through the formula.

  2. anonymous coward

    Whew, I’m glad this article finished with the judge throwing it out, because when I started reading about the lab analysts making “additions and subtractions” my first thought was oh carp! now the police labs can make the DNA fit the suspect.

    1. SHG Post author

      You assume malfeasance. If that’s the case, they just don’t test it at all and say it fit the suspect. It’s that easy, if someone is so inclined.

  3. Fubar

    Further, the fact that FST software is not open to the public, or to defense counsel, is the basis of a more general objection. This court understands the city’s desire to control access to computer programming that was developed at great cost. But the FST is, as a result, truly a “black box”— a program that cannot be used by defense experts with theories of the case different from the prosecution’s.

    Justice Dwyer deep Fryes the reliance
    On results from a black box appliance.
    If your methods are hidden,
    Their use is forbidden.
    To convict, you can’t use secret science!

      1. Fubar

        [
        Heh!

        The late great Paul Wilbur Klipsch had a special button he wore behind his lapel for use in those special moments.

        The company still sells ’em. Maybe somebody should send Justice Dwyer one. He could wear it in conference for those special occasions when somebody claims they have magical sekrit black boxes like Forensic Statistical Tool software.
        null

        ]

  4. Chris Halkides

    One problem with high sensitivity DNA analysis is that it is often not possible to determine the body fluid or tissue that produced it. It is “sub source” DNA, which is poorer quality evidence than DNA that can be tied to a source. Another problem is that it is very easy to deposit and to transfer small amounts of DNA, as the article “Making the invisible visible” in the New Zealand Herald clearly describes. This issue may be pertinent when considering a mixture of DNA that includes the suspect and one or more other individuals. If the DNA from the other individual(s) were deposited in a way that is unrelated to the crime, then how can one be sure that the suspect’s DNA was deposited during the crime and not in some other way.

  5. Pingback: An Attempt to Explain High Sensitivity Analysis in Collins - Windypundit

  6. Chris Halkides

    “[Barry] Scheck had been demanding the Office of Chief Medical Examiner make public its internal validation studies on LCN, which it has refused to do. At one point, after an otherwise subdued hearing, he yelled to some of the subcommittee members: “YOU ARE ALL ******* LYING!” LCN is an abbreviation for low copy number, a rough synonym for high sensitivity in DNA forensics. IMO one has to show one’s cards.

Comments are closed.