But For Judges, They Wouldn’t Be Dead In The Name Of Science

This isn’t the sort of thing anybody wants to say, but I hope they were guilty. Because the alternative pushes the boundaries of even the hardest hearted bastard.

The Washington Post published a story so horrifying this weekend that it would stop your breath: “The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000.”

What went wrong? The Post continues:“Of 28 examiners with the FBI Laboratory’s microscopic hair comparison unit, 26 overstated forensic matches in ways that favored prosecutors in more than 95 percent of the 268 trials reviewed so far.” The shameful, horrifying errors were uncovered in a massive, three-year review by the National Association of Criminal Defense Lawyers and the Innocence Project.

Chillingly, as the Post continues, “the cases include those of 32 defendants sentenced to death.” Of these defendants, 14 have already been executed or died in prison.

That forensic science has been a monumental disaster isn’t news.  It runs the gamut of lies, incompetence and, mostly, a scam on the legal system, that has been well-known for years now. That this meant that decades of convictions were based upon knowingly false junk science was similarly presumed.

But this review of actual cases puts a price tag on the fraud.  Of the cases reviewed, 32 defendants were sentenced to death. Death. No doubt the crimes for which these defendants were convicted were horrible, but were they the perpetrators of those crimes?  Fourteen of those defendants have been executed, fast or slow. So yeah, I hope they were guilty as sin, because if they weren’t, the fraud perpetrated by the forensic science industry is a crime of enormous magnitude. Want to talk about mass murder? Are 14 enough bodies for you?

Since the report of the National Academies of Science came out in 2009, “Strengthening Forensic Science in the United States: A Path Forward,” there hasn’t been any serious doubt that most of forensics is a scam.  We believe, because we don’t really have a clue about science and we’re susceptible to bias and the influence of television magic. We’re so infatuated with the idea that science can solve all the nettlesome problems that we let our eyes fog over and give people with insignificant degrees and articles published in quasi-existent trade journals the authority to destroy lives.  It’s so much better to be told by an expert that someone is guilty than to have to figure it out based on evidence.

But that report came out in 2009, and still courts admit the same crap into evidence, and in the same manner as to prevent any effective counter to its falsity.

This problem doesn’t stop with the FBI labs or federal prosecutions. The review focuses on the first few hundred cases, involving FBI examiners, but the same mistakes and faulty testimony were likely presented in any state prosecutions that relied on the between 500 and 1,000 local or state examiners trained by the FBI. Some states will automatically conduct reviews. Others may not. Much of the evidence is now lost.

Systemic change, in other words, is being left to the discretion of the system itself.

If there was a deep concern over the integrity of the process, it should have ground to a halt upon release of the 2009 report, and a big meeting held for all the judges at which somebody in a position of authority screamed at them, “stop admitting shit evidence, you dopes.”  But that meeting never happened.  And shit evidence continued to flow, as it always did. And judges, to the extent they weren’t blithering idiots or complicit in the critical goal of assuring the conviction of as many defendants as possible, shrugged and went about their business as if junk science had nothing to do with them.

Dahlia Lithwick raises some of the now-historic fixes, noting specific areas of junkiness, such as crime labs that fake results, or “100%” hair analysis that confused dog hair with human. Whether fixes were expensive or easy, complex or simple, may miss the point, however. The point is that we know, with certainty, that almost all forensic science is a scam and yet it continues to be admitted into evidence to convict people.

These solutions are not all that expensive or complicated. Among them: giving defendants their own forensic experts, untethering crime labs from the prosecutors and cops to which they now answer, verification and standards. But no matter how many times we may reiterate that the status quo is intolerable and that simple corrections would yield significantly better data, no real energy for reform exists.

All good solutions, but hardly worth the bother as long as judges continue to allow the fraud to be perpetrated on the witness stand. Want to generate some “real energy for reform”? Just say no, Judge. Refuse to be complicit in the perpetuation of lies and scams in your courtroom. Granted, it means that some guilty people will walk, but the alternative is that you wrap yourself in the warm comfort of precedent while knowing, with absolute certainty, that the “expert” you just qualified is a fraud.

And as long as we’re talking about perpetuating scams, there’s one more that wasn’t part of this study, but likely happens more than any other in the legal system.  The beloved drug doggie. Why not let the dogs go to good homes, where they can play with children and romp in the backyard, and just use a coin toss to determine whether police have probable cause to search.

We all know that a coin toss is just as valid as a dog hit.  If you adore the doggies that much, free them from their labor and end the lie.  But then, if you really loved the doggies that much, you would get upset when cops murdered them for being doggies. Or does it just make it all seem so much more real to pretend that the doggies are magic when they get their doggie treats from the good guys and alert? Science, right? Who are we kidding?

 

48 thoughts on “But For Judges, They Wouldn’t Be Dead In The Name Of Science

    1. Lawrence Kaplan

      I note that the good judge avoids the main issue, judges who accept and sentence people to death on the basis of junk forensic science. How do you stand on that oh Honorable R G K?

      1. Richard G. Kopf

        Lawrence,

        I don’t mean to be snide. But what the fuck do you think I think about it.

        All the best.

        RGK

  1. albeed

    As a scientist who worked in many disciplines for 35 years (5 of them microscopically examining and trying to identify particulate matter in injectable drugs), I would throw my hands into the air and say there were almost an infinite number of possibilities to determine the sources of the materials which I was trying to identify. I came to realize that until one establishes the precision, accuracy, limits of detection and quantitation, and ruggedness of their methods, it is all BS. Of course, I am not a recognized expert to be able to testify in trials. The number of times I threw objects at the TV watching the “Forensic Files” was more than I can count, and I am not just talking about wonder dogs and bite mark evidence.

    I remember the honorable RGK saying that he “would have no problems” with sentencing a known factually innocent person if all “legal procedures and processes” were satisfied. He also claims to be an atheist, or at least an agnostic. I claim that he is wrong. His deity is his mind and the law, no matter how misguided they both are or can be. The acceptance of forensics as currently practiced is Exhibit A.

      1. DaveL

        I think you misunderstand. In US v. Bentley, the dog in question provided probable cause 93% of the time. It’s just that when it did alert, drugs were only found 59% of the time.

        So really, it’s more like having the cops put a puppet on one hand to serve as a confidential informant.

        The high alert rate is really the more disturbing part, because it implies the dog’s success rate is only as high as it was because of the underlying likelihood the subjects were carrying drugs in the first place, and has nothing whatsoever to do with the abilities of the dog. He did, after all, test virtually the entire sample. The best case scenario would be that the dog never failed to alert on people carrying drugs, in which case the 7% that weren’t alerted on were innocent. That would give him a 0% false negative rate and an 84% false positive rate. The worst case scenario is that all the people the dog didn’t alert on were carrying drugs, which would give him an 11% false negative rate and a 100% false positive rate. Take results like that and put them on a sample where you don’t have more than half the sample carrying drugs, and his success rate would quickly go from “coin toss” to “carnival game”.

  2. REvers

    A coin toss is much more dependable than a dog hit, for the simple reason that cops can’t tell the coin to land heads or tails. Unless the cop is telekinetic, of course, although I’ve yet to see a cop claim that particular superpower.

    1. Dave

      Sure they do. Cops will just command the coin to comply, and if it fails to do so they will beat it with their baton until it bounces up on the side they wanted.

  3. MS

    I saw both of these articles being passed around Twitter yesterday, but both of them are from last year. You even wrote about the underlying FBI report when it first came out: http://blog.simplejustice.us/2015/04/21/the-easy-yet-impossible-solution-to-junk-science/.

    Do you know why there’s been renewed interest in these articles? Don’t get me wrong; the subject couldn’t be more important, and I would be perfectly happy if they were highlighted every week. I’m just curious if there’s a specific reason for the current attention.

    1. SHG Post author

      Not a clue why, but since they came back on the radar and there has been zero change, it’s worth pounding the table again.

  4. Dragoness Eclectic

    I apologize if this is a dumb question, but am I correct in thinking that the problem with drug dogs is that they are too eager to please their handlers, and handlers either subconsciously (if you want to be charitable) or deliberately (if you are cynical) train them to respond to subtle cues that read ‘I really want you to alert on this guy’? I am given to understand, based on a SAR trainer I knew, that canine sense of smell really is that good. (Finding bodies in the rubble before they are beyond rescue, or recovering them afterwards, is not a situation where the handler has a motive to reward false ‘hits’)

    1. SHG Post author

      It’s not the dog’s fault. The dog is just a dog. Don’t blame the dog. I hope that answers your question and that your google is repaired quickly.

          1. Dave

            My attorney advises me that I should remain mute while she argues that it COULD have been nothing more than a typo. Therefore, there is reasonable doubt, and I have no obligation to testify or present any evidence to the contrary, as the burden does not fall on me.

            Unless you are using Title IX on me. Then I will just put my head in the guillotine before trial, as required, and pull the cord labeled “due process for the accused”.

        1. Dragoness Eclectic

          How interesting.. there’s actually a name for it! “Clever Hans Effect”, and it’s a big issue in any kind of cognition experiments….and documented as a problem with drug dogs. I was right in guessing that part of the problem was subtle or overt cueing by the handler, based on nothing more than my knowledge of basic dog training. It’s very easy to mistrain dogs if you are careless–like one of my relatives who always issues the “Sit” command three times to his dog, because he thinks the dog is recalcitrant. No, he’s just inadvertently taught the dog that “Sit-sit-sit”, not “sit”, is the proper command to sit. The dog doesn’t speak English, he just learns what you praise and reward him (or her) for doing–like alerting on someone that makes his handler tense or whatever.

  5. Dave Ruddell

    Okay, I’ll bite. When I find particles containing lead, barium, and antimony on a sample taken from a suspect’s hands, and I state that it’s gunshot residue, what’s unscientific about that? My report will note the three main ways that a person could get GSR on them (only one of which is actually firing a gun), and will mention the limitations of my examination based on the available scientific literature. If called to testify, I will be able to go into much more detail. I also always reach out to the defence when I receive a subpoena, although most of the time they don’t call me back.

    So, what part of that is a scam?

    1. SHG Post author

      So you want personal absolution because this was all about you? Fair enough. Not you. If you says so.

      Unless of course one of the thousand possible things that can go wrong, including falsifying lab results, happens, in which case maybe it is you. How would anyone here know? Because you say you’re legit? Did Annie Dookhan wander around the internet commenting that she falsified results? Not saying you are, but your saying your not doesn’t exactly prove the fact. And that you think it does raises a question as to the soundness of your judgment. See how that works?

      1. Dave Ruddell

        Thank you, but I neither request or require your absolution. I take it you can see how your jeremiad above can be applied to any number of professions, including your own. After all, some defense lawyers are lazy and incompetent. Not you, of course, if you says so. Of course your saying so doesn’t exactly prove the fact. So let’s do away with CDLs and replace the system with Trial by Combat? Or maybe Ordeal?

        1. SHG Post author

          Ah, but you did request absolution, or you wouldn’t have left the comment all about you. And there is no rational connection between some defense lawyers being lazy (note that I didn’t suggest where I was, one way or the other, which distinguishes us. Sorry) and trial by combat.

          So now I’m curious. Having made the mistake of self-vouching to make your point, and having been called out not for your self-assessment as being untrue, but merely not being a good way to make a point, what made you think taking a swan dive down the rabbit hole was going to work out better for you? This comment was idiotic. What makes you think it’s a good idea to go from a small logical error to a big (and kinda bizarre) one? If I hurt your feelings with my reply to your first comment, did you think conclusively branding yourself an idiot would make things better?

        2. Steve

          You may well be a competent lab tech, but you surely didn’t help your cause here. If anything, your initial self-assessment, which would have fine, now looks suspect in light of your totally off-the-wall reply.

          You may want to rethink that trial by ordeal thing, because you won’t do very well.

    2. OEH

      It’s a bit off-topic but this is something that I’m curious about — at what point do legit analytical techniques become persuasive?

      Your example would convince me if you tested it end-to-end on a regular basis. Here’s what end-to-end testing would mean: select random strangers, scrape the particles off their hands, apply your analytical procedure, then interview the test subjects about whether they’d handled a gun recently.

      Because, from the information you provided, your example doesn’t convince me. I’m not saying it’s wrong — maybe it’s a great way to identify gunshot residue — it’s just it doesn’t allow me, given that information, to draw that conclusion on my own. I handle those three chemicals on a daily basis and I never go near a gun. Most old glazes, pottery, paints, plumbing, various electronics, and the dirt on the ground in industrial areas will contain those chemicals.

      It’s not that your technique isn’t valid, its’ that it’s answering the wrong question. And that’s a real problem, because what you get out of it isn’t evidence; it’s a conclusion. It’s a magic box that says “guilty”. Evidence is not the same thing is a conclusion, no matter how carefully drawn that conclusion may be.

      1. Dave Ruddell

        Well, there are some studies out there that do try to answer that question, and the only ones that seem to have GSR on them are people who handle guns. The point that you make about handling those elements is a good one, but the technique used by the vast majority of labs (SEM/EDX) identifies particles that contain all three elements in the same particle, so even if you handle lead, barium, and, antimony in you daily life, you wouldn’t test positive for GSR. Heck, if you spend all day handling unfired ammunition you won’t test positive. There’s also been extensive investigation into whether there are other substances unrelated to the primer in ammunition that would mimic GSR; so far there’s nothing.

        You final paragraph makes an important point. Having GSR on your hands means you have GSR on your hands. Doesn’t mean you fired a gun, doesn’t mean you fired the gun used in the incident. It’s up to the scientist to communicate the limitations of the examination.

        1. zoe

          David-
          re: “Having GSR on your hands means you have GSR on your hands. Doesn’t mean you fired a gun, doesn’t mean you fired the gun used in the incident.”

          Is this a statement you provide for the benefit of the judge and jury (without prompting)?

          How often do you testify for the prosecution? If the prosecution believed this, you wouldn’t be testifying for them because it substantially weakens their case. Arguably, it calls into question the point of the analytical test. (Absence of GSR does not mean that the suspect did not fire the gun used in the incident. It means the suspect may have washed their hands.)

          1. SHG Post author

            This is why making things personal presents a problem. If Dave says he does, indeed, testify that way, we’re left to believe him, based only on his saying so (like every cop who claims he’s never lied on the stand) or doubt him, because it can’t be extrinsically proven. No one is any wiser for this.

            Regardless, it’s one data point amongst many, and resolves nothing as to what others say on the stand. And here you are, asking Dave what he does, as if what Dave does somehow proves that all witnesses do whatever Dave does, or at least claims to do. Again, no one is any wiser for this.

            And even if we assume away the foregoing problems, it’s at most one tiny slice in a vast array of forensic science evidence. Bite mark analysis was similarly considered scientifically accurate, until it wasn’t. Will that happen to gun shot residue some day? Probably not, but then, that’s what someone would have said about fingerprints, bite mark analysis, etc., ten years ago.

  6. pavlaugh

    “Just say no, Judge” doesn’t solve the problem when everyone–the witness, the prosecutor, the judge, and even defense counsel–thinks the science is “good.” As you’ve pointed out in your blog, everyone used to think fingerprints and eyewitness identifications were the “gold standard.” Why? Because the science wasn’t adequately challenged?

    At least in the cases I see (appeals), it is rare for defense counsel to mount more than a perfunctory challenge to the reliability of an expert’s methods, let alone bring a controverting expert or cross-examine with a learned treatise.

    I’m too baby of a lawyer to know much about the development of science on hair samples or fingerprints. But I’ve seen unconverted testimony plenty of times w/r/t dog sniffs and eyewitness identification. An officer testifies his dog alerts to the smell of cocaine on cash. I doubt every judge knows this testimony is technically wrong, or at the very least subject to some serious challenge. But there’s no objection from the defense, no defense expert, no learned treatise. What’s the judge to do? Pause the trial and read up on the matter sua sponte? (Even just reading a few appellate decisions on the matter reveals there’s wide disparity, and opinions swing like a pendulum year to year.)

    As you’ve said, everyone currently thinks of DNA as the gold standard. It doesn’t get challenged in court very much from what I can tell. The analyst says so-and-so can’t be excluded from this DNA mixture. The likelihood of another Caucasian is one in a sextrillion. No objection; no controverting expert; no learned treatise; barely any cross-examination at all. What’s the judge to do other than admit the evidence?

    1. SHG Post author

      Some defense lawyers are lazy and incompetent. Most can’t be bothered fighting losing battles.

      Prosecutors are supposed to want to do justice, but since that’s a meaningless word, there’s no point in beating it.

      Judges are the gatekeepers. When defense lawyers fight “science,” they get the crap smacked out of them, because everybody knows bite marks are good science. They even have their own Journal, the American Journal of Irrefutably Good Bite Mark Science. Yes, defense lawyer should object and fight, but judges have to be receptive to the objection or defense expert.

  7. Simon Elliott

    A lot of drug dogs are no different from horoscopes: they merely tell you what you want it to tell you. If we don’t admit horoscopes as a basis of probable cause, then badly trained dogs should also not be admissible.

    1. SHG Post author

      Bad analogy. Not that dog hits are or should be acceptable as probable cause, but there is nothing analogous with horoscopes, and no need to make an analogy at all.

  8. Simon Elliott

    Why is it a bad analogy? A horoscope is sufficiently vague that you can read into it whatever you want it to say, including your unconscious biases. Thus, while it appears to be an independent source of prediction, any “predictions” are a sham because they are merely confirming your preexisting thoughts and views of reality. If a badly trained drug dog is merely reacting to the unconscious thoughts of its handler, then it is likewise not an independent source of prediction or detection.

    Note also that there is a lot of confirmation bias in the reports: we only hear about drug dogs detecting drugs when that is used as a basis for PC, but it could be alerting to everything. We see the same thing in the literature on horoscopes, where correct predictions are remembered but incorrect ones are not.

      1. Myles

        Should someone explain to him that dogs are real and can actually smell, while horoscopes are total nonsense? Nah.

        1. Simon Elliott

          Sigh. Its an ANALOGY meant to illuminate the absurdity of accepting bad science as evidence.

          Yes, dogs can detect things, and are pretty good at it (some animals are better). But the way that they are USED is not always commensurate with their predictive ability. Any test should be an independent source of evidence. But, in certain hands a dog is not an independent predictive tool for the presence of drugs, but merely confirms a police belief.

          In that sense they are being used like horoscopes: they appear to be an independent predictive tool those who believe in them, but they are really just telling them what they want to hear. We shouldn’t accept horoscopes, and we shouldn’t accept bad science as a cover for bad police work. We should consider whether a dog is accurate before using it as a test, in the same way that breathalyzers are supposed to be calibrated and checked.

          The courts accept bad science because they believe that a certain test is accurate and because the results confirm their biases. But belief is not the same as fact, and confirmation bias is a bug, not a feature.

          1. SHG Post author

            We got it. You like your analogy, even though others think it sucks. You feel compelled to repeat the obvious, as if no one ever heard of confirmation bias until you got here. Thanks for playing, but the game is now officially over. Just stop.

        2. Simon Elliott

          Sigh. Its an ANALOGY meant to illuminate the absurdity of accepting bad science as evidence.

          Yes, dogs can detect things, and are pretty good at it (some animals are better). But the way that they are USED is not always commensurate with their predictive ability. Any test should be an independent source of evidence. But, in certain hands a dog is not an independent predictive tool for the presence of drugs, but merely confirms a police belief.

          In that sense they are being used like horoscopes: they appear to be an independent predictive tool those who believe in them, but they are really just telling them what they want to hear. We shouldn’t accept horoscopes as a reasonable basis for an action, and we shouldn’t accept bad science as a cover for bad police work. We should consider whether a dog is accurate before using it as a test, in the same way that breathalyzers are supposed to be calibrated and checked.

          The courts accept bad science because they believe that a certain test is accurate and because the results confirm their biases. But belief is not the same as fact, and confirmation bias is a bug, not a feature.

  9. j a higginbotham

    Are there two distinct issues?
    1) How reliable scientific tests are and what the chance of errors are, and
    2) Whether the people conducting the tests are biased, consciously or unconsciously.

      1. Dragoness Eclectic

        I agree. You always have to keep an eye on the human factor–the most rigorous scientific lab test in the world is worthless if the lab tech just wants to meet quota and fills in the paperwork with the results their ‘customer’ wants to see, instead of actually doing the test.

  10. Federico Wibmer

    [Ed. Note: Link deleted per rules.]
    A quote of interest: “When it comes to the assessment of courtroom evidence, it is too often a matter of the blind leading the blind. “We have scientifically illiterate judges, scientifically illiterate lawyers, and scientifically illiterate jurors,” Fabricant said.”

Comments are closed.