The Easy Yet Impossible Solution To Junk Science (Update)

In the New York Times, Eric Lander, founding director of the Broad Institute of MIT and Harvard, provides a ringing condemnation of the use of forensic science in the courtroom.

As miscarriages of justice piled up, it became essential to understand precisely how the criminal justice system made such mistakes and how to prevent them. Troublingly, about a quarter of the cases examined by the Innocence Project (on whose board I now serve) involved forensic scientists who had erroneously claimed to identify defendants with near-certainty by matching hair samples, fibers, shoe prints or bite marks.

There were clearly fundamental problems with forensic science. A 2009 report by the National Research Council, an arm of the National Academies, found that apart from DNA testing, no forensic method had been rigorously shown to consistently and reliably demonstrate a connection between evidence and a specific person.

Or to put it more bluntly, we’ve rested our faith in crap, scientifically unproven and unjustifiable, because people dressed in expert suits took the witness stand, offered jargonized mumbo-jumbo in official sounding tones, and promised that they were right, because science.

As has been made clear for years, neither lawyers nor judges tend to come from strong science backgrounds.  Ask most what the scientific method is and they’ll either stare blankly into space or divert your attention to their shiny iPhone.

As the flaws in the courtroom are obvious to scientists, particularly those who don’t make a living off testifying but rather are honest about their discipline, the solution is similarly obvious:

Under the federal rules of evidence, expert testimony must be based on “reliable principles and methods.” It is now abundantly clear that an expert’s opinion is not a reliable basis for drawing connections between evidence samples and a particular person. No expert should be permitted to testify without showing three things: a public database of patterns from many representative samples; precise and objective criteria for declaring matches; and peer-reviewed published studies that validate the methods.

This reflects the chasm between the scientist’s appreciation of science and woeful grasp of jurisprudence.  It’s not that this couldn’t happen. It could. It’s that every aspect of our system pushes against it happening.

The law is built on precedent.  That means, once an expertise is held to be reliable, it is reliable in every case thereafter, and immune from challenge on the question of its reliability.  The premise is finality, that we don’t relitigate every challenge over and over.  It’s done. Move on.

And so the law is built on the weakest link. The nastiest defendant, whom we want so desperately to convict that we’ll turn a blind eye to the fallibility of the evidence.  The judge who sees her role to facilitating conviction of criminals because she believes that it’s best for society to get its miscreants off the streets and into prisons.  The prosecutor who will conceal the infirmities of his proof because he believes in the righteousness of his cause.  The defense lawyer who is in over his head, disinterested in fighting the junk being thrown at the defendant.

Does anyone really think the justices of the Supreme Court are unaware that dog hits are little better than a coin toss in determining probable cause?  They aren’t blind fools.  They are, however, practical, recognizing that if they end the lie upon which constitutional rights hinge based upon science, there will be an awful lot of people who possess drugs who will go free.  They knowingly trade off the absurdly bad science of the efficacy of dog sniffing for the effectiveness of catching criminals.

What the law should do is proclaim that all science in the courtroom be halted, that no expert be allowed to testify, until it can be scrutinized as Lander suggests.  Except we all know that most of it won’t pass muster. Then what will we do? Our system of convictions depends on our maintaining the belief that we have the right criminal, and that belief depends on our acceptance of junk science which has become the accepted method of proof.

All of this falls within the ambit of the gatekeeper function, that judges shouldn’t permit someone who is deigned an expert to take the witness stand and opine on the meaning of evidence and its conclusiveness and reliability.  Once this happens, it essentially removes the question from the jury, as science says the hair found next to Joe’s dead body came from Bill, as did the bullet and the bloody fingerprint.  So who can possibly dispute that Bill shot Joe?  The question is answered by science, and the jury can sleep well at night knowing that they convicted the right fellow.

While all of the infirmities of proof in the courtroom appear obvious to scientists, they are hardly so clear to others.  Judges may not get it, and often don’t. Lawyers fare no better.  But that shouldn’t be the stumbling block, as neither judges nor lawyers are scientists.  All we need to do is get an expert of our own in whatever discipline is in issue, put the expert on the stand to refute the silliness of the duct tape expert, and turn the trial on its ear.

No one who has ever sought to put an expert on the witness stand would be so cavalier about it. An expert must be found, and most people who possess real expertise have little interest in being in the hot seat.  There are experts for sale all over the place, and most have no greater expertise than the slugs who shill for the prosecution. The good ones, the real ones, aren’t nearly as interested in being cross-examined as they are in writing editorials.

Experts need to be paid.  There really isn’t much more needed to be said about this.  As sincere as their belief may be in whatever their expertise, they don’t appear for love.  It’s not that I blame them for expecting to be paid for their services, but of the many things in short supply in the courtroom, funds for the defense are a perpetual problem.  And real experts aren’t cheap. If you think lawyers are expensive, put an expert on the stand for an hour.

Experts need to be persuasive.  Some people are experts in answering questions at trial, but they rarely are the same as the experts in the underlying discipline.  In contrast, many scientists who possess true expertise in their field suck as witnesses. They equivocate. They are obtuse and incomprehensible. They say one thing in the office, than say something entirely different on the stand. They get flustered or angry. They are called for the purpose of furthering the truth, and end up looking like idiots because cross-examination doesn’t happen with the niceties of an academic symposium.

There is no question nearly all the science offered in courtrooms is unreliable nonsense, and to the limited extent the participants aren’t clueless or blind, they realize it.  But without it, the system would grind to a halt and the guilty, along with the innocent, would go free.  The only thing more important than faith in scientific evidence is faith that our legal system will protect us from the criminals.

It’s not that we accept the conviction of the innocent as the price of a smoothly running conviction machine, but that we are comforted by the certainty that junk science offers, that the system isn’t a huge sham.  And the mechanisms don’t exist to do much to change it, because nobody really wants them to.

Update:  Jeff Gamso adds in a detail about Lander’s op-ed that needs to be recognized:

“Stunned the legal community”?  Really?

THE F.B.I. stunned the legal community on Monday with its acknowledgment that testimony by its forensic scientists about hair identification was scientifically indefensible in nearly every one of more than 250 cases reviewed.

Lander says it was no surprise to actual scientists who’ve known for years that nearly all of what’s called “forensic science” is actually forensic bullshit.  But the lawyers have known that, too.  Really, anyone who’s been paying attention’s known it.

But Jeff misses the key point. While we’ve been screaming about this forever, nobody cared. Nobody listened, because we aren’t trusted government officials and it’s just us criminal defense lawyers complaining about stuff, like evidence and constitutional rights, again.

That the FBI admitted it is what stunned the legal community (of which, apparently, criminal defense lawyers are too minor a part to be worthy of recognition), because they are official.  Now the legal community cares. That’s stunning. Us, not so much.

38 thoughts on “The Easy Yet Impossible Solution To Junk Science (Update)

  1. Bartleby the Scrivener

    I can’t afford to become a lawyer, but you REALLY make me want to get my degree just so I can fight for things like this, even if I don’t do it for a living! I wish there was a way I could just donate to a fund that fights garbage like this with platoons of lawyers that are willing to swarm into courtrooms and make the case heard by everyone.

    I see many people rail about our high rate of incarceration and blame stupid laws for it, but I think the worst of our laws are those that allow the violation of our rights in pursuit of a conviction, even if it’s for a good reason! I don’t care if someone committed murder, if our government can’t get a conviction without violating the person’s rights, then that’s just too bad…we have to let them go!

    *sigh*

    As usual, you are absolutely correct and I completely agree with you, but why do I feel like an old man waving my cane and telling kids to get off his lawn?

  2. Bill

    Your solution is definitely the way to go, but I think a minimum qualification of an A in Stats 101 or above for each jury member would get us two standard deviations of the way there. I know, it’s a stupid thing to point out b/c it’ll never happen, but easy but impossible was the theme of the post

  3. Marc R

    Good points. Even though it’s explained in the retainer and in person and reiterated a lot of clients are of the belief an expert should come out of the retainer. I don’t where this line of thought came from. I can’t think of a trial I’ve had where I don’t have an expert combat the givernment’s witness.

    I see it as two option: (1) have a real scientist argue that hair and ballistics and blood splatter are opinions resting on unproven assumptions with no basis in chemistry or physics. These experts are “forensic scientists,” very rarely do they possess a phd in molecular biology or physical chemistry. So they can attack the witness, the science and through away the “proof.”

    Or you get an expert with similar credentials to the state’s expert who will refute the conclusion, and sometimes the methodology used, but never the premises upon which this “science” relies.

    Obviously in state court you need the latter and for federal it’s a tough call depending upon the case and the judge.

    What’s your desired remedy to evaluate the government scientists and to counter their evidence against your client?

    1. SHG Post author

      Perhaps the solution has to fall into a special science court, where knowledgeable judges, lawyers and actual (and well-funded) experts, including a panel of mutually-agreed neutral expert “jurors,” vet all forensic disciplines before any judge is allowed to admit testimony to a jury? And all current science must go through this as well?

      But that’s pie in the sky. I have the same problems as everyone else with experts, and every time I need one, go through the same struggles and disappointments. Ironically, I know there are real experts out there, but they are always washing their hair the day they’re needed for testimony.

      1. DB

        Except for issue of plaintiffs/defendants being involved, there already is a kind-of-sort-of form of this court in existence: it’s called peer review.

        And the main impetus for writing this blog post seems to stem from the fact that the government’s forensic methods are finally being subjected to that peer review, with disastrous results (considering the number of people irreparably harmed by the flawed methods).

        I also believe there are plenty of lawyers out there with an education in the various scientific disciplines; it’s just that they all go work as patent attorney’s.

          1. df

            As a small-scale analogy, what about using limited expert hot-tubbing, having them consult with each other as to what the technical issues and disagreements are? Or is this the kind of thing that works only (when it does work) in civil litigation rather than criminal?

            1. SHG Post author

              Many of these “disciplines” (see, e.g., American Academy of Forensic Duct Tape Examiners) are such nonsense that outside of the small group of professional testifiers, they don’t actually exist. That’s a large part of the problem.

      2. Dissent

        Wasn’t the Daubert standard intended to address some of the concerns you’ve raised, Scott?
        I think courts may be used to asking questions about the *reliability* of findings, but reliable only means repeatable. If you keep repeating the same confounded design, you will repeat the results, but will likely still be wrong in your conclusions, and peer review may not catch that. And if you define your constructs incorrectly, you may get reliable results that don’t even measure what you want to measure.
        OK, I’ll shut up now… 🙂

    2. bmaz

      These are some fairly decent suggestions previously proposed by Balko and Koppl that are not bad, but the inertia of the system, and strength of prosecutors and police lobbies will probably prevent any of them:

      Forensic counsel for the indigent. In many jurisdictions, indigent defendants aren’t given access to their own forensic experts. As a result, the only expert witnesses are often testifying for the prosecution … This undermines the whole adversarial basis of our criminal-justice system. Indigent defendants should be given vouchers to hire their own experts, who can review the forensic analysis and conclusions of each prosecution expert.

      Expert independence. Crime labs, DNA labs, and medical examiners shouldn’t serve under the same bureaucracy as district attorneys and police agencies. If these experts must work for the government, they should report to an independent state agency, if not the courts themselves. There should be a wall of separation between analysis and interpretation … When the same expert performs both the analysis and interpretation, defense experts are often at a disadvantage, having to rely on the notes and photos of the same expert whose testimony they’re disputing.

      Rivalrous redundancy. Whether the state uses its own labs or contracts out to private labs, evidence should periodically and systematically be sent out to yet another competing lab for verification. The state’s labs should be made aware that their work will occasionally be checked but not told when. In addition to helping discover errors that might otherwise go undetected, the introduction of competition to government labs would all but remove any subconscious incentive to appease police and prosecutors and would strengthen the incentive for a more objective analysis.

      Statistical analysis. The results from forensic labs should be regularly analyzed for statistical anomalies. Labs producing unusually high match rates should throw up red flags for further examination. For example, in 2004 Houston medical examiner Patricia Moore was found to have diagnosed shaken-baby syndrome in infant autopsies at a rate several times higher than the national average. This led to an investigation—and the reopening of several convictions that had relied on Moore’s testimony.

      Mask the evidence. A 2006 U.K. study by researchers at the University of Southampton found that the error rate of fingerprint analysts doubled when they were first told the circumstances of the case they were working on. Crime lab technicians and medical examiners should never be permitted to consult with police or prosecutors before performing their analysis. A dramatic child-murder case, for example, may induce a greater subconscious bias to find a match than a burglary case. To the extent that it’s possible, evidence should be stripped of all context before being sent to the lab. Ideally, state or city officials might hire a neutral “evidence shepherd,” whose job would be to deliver crime-scene evidence to the labs and oversee the process of periodically sending evidence to secondary labs for verification.

        1. Bartleby the Scrivener

          Do the statements of an expert witness that are not rebutted by another expert in the same field have to be accepted at face value?

          This is very troubling for me.

          I’m not trying to speak as an attorney with the following…I’m just as a goof with opinions: I’ve long thought opposing expert witnesses should be included in the right to counsel, and this strengthens that belief. If the expert witnesses that are being used to convict you are not subject to an ordinary level of scientific rigor and we only rarely go back and question the validity of their techniques, then there should absolutely be expertise questioning the conclusions of the experts working to convict.

          1. SHG Post author

            No. The jury is entitled to accept or reject them as they see fit. But it’s sham. If the jury was capable of reaching its own assessment of the foundational information, there would neither be a need for an expert nor, indeed, would the party calling the expert be entitled to do so.

            That’s the point, you are allowed to use an expert for an opinion that requires specialized knowledge beyond that of the jury, so how can the jury ignore the expert? They can’t and they don’t.

  4. DB

    A number of years back, my small company hired someone to oversee production and quality control, as well as manage some animal studies of an investigational drug we were were developing. He’s a Ph.D. toxicologist who, and in addition to whatever other accomplishments and work history he had under his belt, he was proud to brag that he often served as an expert witness (he even listed this fact in his C.V.) in criminal trials.

    And though I’m certain he doesn’t read SJ, he strikes me as the litigious type, so I must be careful in what I want to say about him. For the time that I worked under him, in my opinion, his actions never once reached the level of his claims about experience in manufacturing pharmaceuticals, and even more unforgivable, he made repeated errors in understanding the basic chemistry of our product. To cut short any further description (and/or possible defamation), in my opinion, he was a very poor scientist in the time I worked with him, to the point I had to wonder how he even got his undergrad degree, much less a doctorate.

    He was however, a very smooth talking guy, resolute in his beliefs (especially his wacky political ones), sure of his correctness in whatever he was talking about. He was also a nice guy, and personable, and friendly. He was hard to fluster, was used to talking to audiences (partly from being a professor at times), and most importantly (for the topic at hand), when you listened to him talk about some technical/scientific subject, he conveyed a feeling that made you think he absolutely knew what it was he was expounding on. And unless you were knowledgeable in that same subject, it’d be an almost impossible matter to call him out on anything.

    He seems to be the exact type of preferred expert witness you are talking about, convincing and comfortable on the stand and under cross examination. Not that I’m out there getting myself into trouble, but it scares the absolute hell out me to think I could ever be in court and have somebody like this person provide “expert” testimony against me, when the only thing they’re an actual expert in is playing an expert in court (in my opinion).

  5. David

    It is interesting to note that the study calls into question the reliability of fingerprint analysis. The report summary states: “There is some evidence that fingerprints are unique to each person, and it is plausible that careful analysis could accurately discern whether two prints have a common source, the report says. However, claims that these analyses have zero-error rates are not plausible; uniqueness does not guarantee that two individuals’ prints are always sufficiently different that they could not be confused, for example. Studies should accumulate data on how much a person’s fingerprints vary from impression to impression, as well as the degree to which fingerprints vary across a population. With this kind of research, examiners could begin to attach confidence limits to conclusions about whether a print is linked to a particular person.” It also states: “One study found that fingerprint examiners did not always agree even with their own past conclusions when the same evidence was presented in a different context.”

    Critics have rightly pointed to flawed testimony in some of the non-traditional areas of criminal forensics (e.g., hair samples, fibers, shoe prints or bite marks), but fingerprint analysis always seems to get a pass, perhaps because it has such a long history in criminal forensics.

    Given the fact that fingerprint analysis is admitted in so many criminal cases, it seems likely that many innocent people have been convicted of burglary or other serious crimes because a government-paid fingerprint analyst testified that one latent print demonstrated conclusively that the defendant was at the scene of the crime.

    1. SHG Post author

      You’re beginning to appreciate the full scope of the problem. No, fingerprints are not (sorry, TV guys) 100%, but have a variety of significant flaws, and a long history of confidence level errors. And then there’s Brandon Mayfield.

      And fingerprints are the stuff that everybody thought was rock solid.

      1. John

        Everybody? Lol. It may not be common knowledge, but the problems with fingerprints have been known since long before I was born. The criminal justice system might have acted that way, but they also like to pretend that police, prosecutors and judges don’t have any bias against black men and upcoming elections don’t influence sheriffs, prosecutors and judges…

  6. John Barleycorn

    Upping the accountability ante is like pulling teeth while sweeping the back porch?

    In the end, your guild in the broadest of senses is to blame for allowing this and other charades to merrily wonder on down the trail on a daily basis in courtrooms high and low throughout the land.

    And especially fuck the judge that doesn’t go out of his or her way to actively solicit a sound and grounded understanding outside the law and within before he or she drops the gavel down on the decisions that must be made in order to orchestrate the justice jubilee.

    A justice here,
    an injustice there.

    Here a justice,
    there an injustice.

    Justice juice everywhere.

      1. John Barleycorn

        A Dire Straights original and a Dead Kennedys cover come to my mind if you are needing a tummy rub…

        You sit the expert on the closed circuit court TV

        That ain’t workin’ that’s the way you do it

        Money for nothin’ and your chicks for free

        _____________________

        Cut ’em out, ride ’em in
        Ride ’em in, cut ’em out
        Cut ’em out, ride ’em in
        Rawhide

        Keep movin’, movin’, movin’
        Though they’re disapprovin’
        Keep them doggies movin’, rawhide

        Don’t try to understand ’em
        Just rope, throw an’ brand ’em

        Soon we’ll be livin’ high an’ wide

  7. Jerryskids

    But how can you have a kangaroo court without the kangaroos? Nobody wants to spoil the fun by pointing out that the kangaroos are just a dog and a pony dressed in kangaroo costumes.

    1. Fubar

      [ True story from a sometime expert witness of my acquaintance —

      He is expert in telephony and various communication systems. He’s been in the business most of his adult life.

      A few decades ago, he was retained as expert witness by defense in a federal phone hacking case. Defendant was charged with a crime along the lines of “fraudulently obtaining and possessing telephone company trade secrets, confidential telephone company information and devices through interstate commerce”. Defendant had been arrested in possession of a shelf full of Bell System Practices (BSP) volumes and a roll of #18 bellwire. which the Feds offered as evidence of the crime.

      So, Mr. Expert Witness appeared in court with 1) a stack of BSP volumes on a dolly, 2) a receipt, cancelled check and bill of lading, and 3) a roll of #18 bellwire. He had simply ordered the the BSP volumes from AT&T under his own name, and had them shipped to his home. He bought the bellwire at an electrical supply store.

      Feds dropped charges immediately.

      Lili Tomlin was right, and not just about the phone company. ]

  8. Beth Clarkson

    This situation unfortunately is not an uncommon one. “There is no instant pudding!” was how Dr. W. E. Deming put it, as forcefully as a man in his 90’s could when I attended a seminar he put on for Boeing, when I worked there. The hard truth is not easy to accept and there are systemic pressures in place, as you know all too well, driving people to act in ways that drive down the overall quality of results produced.

    Before you can inspire people to act, they have to accept the reality they’ve been shielded from.

  9. Greg

    This is in the supposedly “hard” sciences. The problem is greatly compounded (if even worse than what is being reported can be imagined) in the “soft” sciences involving psychology. Eyewitness identification because who could believe that a rape victim could not ID her attacker whose face she saw from inches away; or child abuse where the child sexual abuse syndrome continues to be used despite being discredited by the guy who first named it. All of these “experts” are allowed to testify about “facts” that are supposedly beyond the ken if the jury. Seems that they are beyond the ken of the self-called experts

    1. SHG Post author

      Not sure why you suggest this is limited to “hard” sciences. I thought it was abundantly clear that it wasn’t limited at all. That said, have there not been enough posts here about the problems with eyewitness ID that it was necessary for you to raise it in cursory fashion lest anyone think it was without problems?

  10. jac

    Perjury, fraudulent documentation (lab reports), tampering with evidence…why aren’t these crimes perpetrated in the courtroom by the so-called experts prosecuted? Annie Dookhan (Massachusettes) is the only forensic analyst that comes to mind that has ever been prosecuted for falsifying data (but only after she performed 60,000 drug tests). So there is a solution — use the hammer!

      1. William Doriss

        Sincere and unfounded beliefs in faeries and goblins may not be crimes,
        but they do not belong in courts of law where lives can be disrupted, uprooted
        and/or ruined/destroyed beyond repair.

        [Ed. Note: Ruminations on the nature of Disney movies deleted.]

        The appellate courts are no help,… asleep at the switch, normally. They rubber-stamp the trial court like there’s
        no tomorrow. Do they even know where the switch is?!?

  11. John

    Forensic tests are usually GREAT for one thing: exclusion. I can say “the hair found in Alice’s car did not come from Bob” with 100% accuracy. However, “the hair is Carl’s car may have come from Bob” is just not good enough. The cops want the expert to say “there is a less than one in ten thousand chance this hair came from someone other than Bob.”

    Forensics can help an investigation by ruling out suspects. Rarely can it make a case. I worked at a company that developed forensic DNA test kits. I heard the complaints of the people who dealt with law enforcement. Besides their incompetence at following simple evidence storage instructions that a sixth grader should be able to understand (grr…), they want a certainty that often cannot be provided except when excluding suspects.

    1. SHG Post author

      Some forensics are great at excluding. Some forensics, like bite mark and duct tape, are worthless for any purpose.

        1. Bill

          The problem is, that Junk Science is reinforced by many tv shows (supposedly serious ones like Forensic files). This may sound trivial, but it’s precisely the people that will likely be on a jury who are watching these shows and believing ‘it’s real’ b/c some TV expert said the same thing. Bite Mark analysis has been thoroughly discredited, yet there are at least two Forensic files episodes where they highlighted the precision of bite mark analysis as being critical to the conviction.

          SHG doesn’t like going down rabbit holes of stupid so I’ll stop here, but seriously, Duct Tape analysis, Bite Mark analysis and even “Forensic Hypnotherapy” is regularly highlighted as ‘real hard science’. If you have uneducated jurors (that never happens, I know) or even educated by easily impressed jurors, it’s very safe to say they’d be swayed by this.

          If you doubt it, see how many murder experts there were back during the Casey Anthony trial or how many self-defense law experts appeared during the Zimmerman case. If Jurors didn’t fall for it, Junk Science wouldn’t make the discussion topic list, but there are a whole lot that seriously believe in duct tape analysis. Source: Reddit, any blog post discussing law that doesn’t have a curmudgeon filtering the stupid.

          1. SHG Post author

            People who pay attention to such matters as law or science, whether because they’re educated in it or just particularly interested, are frequently astounded to learn that otherwise smart and educated people are just totally clueless when it comes to common assumptions versus reality. It’s a steep learning curve, and extremely hard to get people to move beyond strongly held assumptions/beliefs, even though they are utterly wrong.

            We all have these strongly held but baseless assumptions/beliefs, only about different thingss. We laugh at someone else’s misguided views, but can’t recognize our own any more than they can recognize theirs.

Comments are closed.