Steve Landsburg, an economics professor at University of Rochester, posts a Law School Admissions Test at his blog, Big Questions. It comes in two parts, the first being the “test” itself.
Here I have a couple of urns. The one on the left contains 70 red balls and 30 black. The one on the right contains 30 red and 70 black.
While you weren’t looking, I reached into one of these urns and randomly drew out a dozen balls. As you can see, 4 of them were red and 8 were black.
Here are three questions that I think you ought to be able to answer if you want to be in the business of assessing evidence:
- If you had to guess, which urn would you guess I drew from?
- What’s your estimate of the odds that you’re right?
- Do you think you’re right beyond a reasonable doubt?
Stop here. Without overthinking the problem, the answer should be reasonably clear, because probability is what it is. But Landsburg’s point isn’t about probability. In his second post on the test, Landsburg recounts the reaction from some lawyers at a cocktail party. One lawyer chose the counterintuitive route, which the others rejected.
The other lawyers all agreed that this was not a very rational thing to do — that the evidence was in favor of the right-hand urn.
“But by how much?” I persisted. After a while a consensus emerged: The evidence is meager; the odds might go up from 50-50 to 55-45, but “…as lawyers we are trained to be skeptical, so we would slant our best judgments downward and act as if the odds were still roughly 50-50″.
The correct answer is about 98%. Yes, the balls were drawn from the right-hand urn beyond a reasonable doubt. This story points out the fact that most subjects vastly underestimate the power of a small sample.
Notice where he slides seamlessly from statistics into the abyss? I’ll repeat it in case you missed it:
The correct answer is about 98%. Yes, the balls were drawn from the right-hand urn beyond a reasonable doubt.
Landsburg has put the likelihood that a defendant committed a crime sufficient to overcome reasonable doubt at 98%. You might think he pulled this number out of his butt, but he didn’t. His put his statistical skills to good use in a different arena, citing to one of my favorite law review articles ever, Alexander “Sasha” Volokh’s “n Guilty Men,” 146 U. Penn. L. Rev. 173.
While 10 guilty men might indeed be the industry standard, the legal scholar Sasha Volokh has documented a long tradition that encompasses a wide range of numbers, some as high as 100 or more. It is difficult for me to believe that the largest of those numbers were ever meant to be taken seriously.
Seizing upon the ancient aphorism, an example of a concept memorialized in a variety of expressions to encompass the idea that we do not accept the conviction of innocents as the price for conviction of the guilty, Landsburg has taken them literally and used the 10 to 1 ratio as the measure of collateral damage.
While acknowledging that 98% suggests that some of those convicted are in fact innocent, and in a nation where so many are convicted, that’s a whole lot of people, he further explains (for the benefit of lawyers, I trust) that it’s not quite a straight two percent who are not guilty.
One commenter suggested that if we adopted a “98%” standard, then 1 out of every 50 people on death row would be innocent. That’s not true, because under that standard, we’d convict everyone who’s 98% sure to be guilty plus everyone who’s 99% sure, plus everyone who’s 99.5% sure, and so forth. So among the entire convicted population, the fraction of innocents would likely be well below 2%.
So rather than 2% of our convict population being innocent, statistical progression suggests that it would “likely be well below 2%,” a surprisingly imprecise expression given Landsburg mathematical bent.
Lest we think Landsburg a hardcore badge-licker, he offers some basis to suggest that his numbers far exceed what we’re staring at from jurors already:
Indeed, I learned from yesterday’s comments [based upon Wikipedia] that as an empirical matter, potential jurors appear to set their cutoff for conviction at something like 70-74% certainty.
70-74% certainty sounds like roughly the right standard to me in a world where the police can be counted on not to take advantage of that standard by falsifying evidence against people they don’t like. Given that prospect, though, I think I prefer something a little tougher — though not as tough as 98%.
With this background, I offer my Law School Admissions Test: Why is Professor Steve Landsburg completely wrong?
H/T Eugene Volokh
Discover more from Simple Justice
Subscribe to get the latest posts sent to your email.
He says “Is Raiffa right that 98% is “beyond a reasonable doubt”? Given a reasonable interpretation of what “reasonable” means, I think the answer is pretty clearly yes. There’s not much in life that we can be more than 98% sure of.” That’s a lot of qualifiers right there.
Oh, and let’s not forget that this is complete bullshit, anyway. How the hell does he plan on comparing an isolated instance of drawing a known number from a known variable, with definite knowledge of the result to… I don’t know, say half remembered testimony from biased parties about something they didn’t actually see?
Excellent points.
What a great post.
Like the cocktail party lawyers, I’m surprised that the probability was as high as 98%. I would have thought a lot lower, but still clearly in favor of the right urn. Something like 75% then.
Maybe the statistical methodology is a little skewed? But I haven’t been a student of statistics.
Although my own informal estimates indicated that in making arrests and bringing charges police are largely correct about 80% of the time.
The biggest problem I see is that from there almost nothing gets corrected. The final results are substantially wrong about 20% of the time, plus or minus a little.
Until the robot overlords devise a perfect justice system for us, we have to tolerate some degree of error or another. It’d be nice if the degree of error in all social policy was “zero,” but, well, life’s complicated.
The analysis is off-putting at first — you cannot directly apply quantitative probabilities to qualitative analyses like whether or not a doubt is “reasonable” — but, if you get over that hump, then the analogy is not that far off the mark. The standard by which we convict people is not “beyond any sort of doubt whatsoever, including doubts which are frivolous or stupid,” but rather “beyond a reasonable doubt,” i.e. a doubt based on reason. At what point does a doubt become reasonable? If we accept that we can analogize “reasoned doubts” to probabilistic figures, then, sure, a 2% doubt does indeed seem like a doubt founded more on speculation about a highly unlikely event than on reason.
Robot overlords?
Each case is sui generis, such that probabilistic outcomes may or may not have any nexus with reasonable doubt. By the way, my understanding of reasonable doubt is not a doubt for which there is a reason, but rather a doubt that is not unreasonable. What does that mean? It’s not at all clear, which is part of the problem with “reasonable doubt” as a standard and why no court has ever devised a meaningful instruction.
If this was a test in a real statistics test with a pedantic stats professor, the answers would be as follows:
1. Right one, but it is kind of meaningless to pick
2. Impossible to calculate
3. Impossible to state
The reason why it is impossible to calculate odds is that there is a probability factor with unknowable odds in the equation. The professor DOES NOT SAY how he decided which urn to pick.
The odds can be only be calculated if these two conditions are met:
– The professor picked the urn randomly (e.g., by throwing a coin)
– The professor would have asked the question regardless of the outcome.
Otherwise, the professor can walk around the room, do various things, and whenever an unlikely event occurs, he can ask you to calculate probabilities.
Even though you’ll think you are 98% certain with each answer, you’ll still be wrong every time. This might seem like a pedantic point, but the distinction a random variable and an unknowable variable is very important in statistics.
… converting what I wrote to “lawyer topics”. 🙂
A prosecutor presents a suspect to the jury and asks, “Look at all those black dots! What’s the chance of this guy living 5 blocks from the victim and not coming to work on the day of the murder?”
If the suspect was chosen randomly, the odds are very small. But if the guy was picked by police as the most likely suspect out of 1000 people checked – precisely because he was near the crime scene and has no alibi – the odds are very different.
The test also had limited and controlled variables. Nothing is so clear and clean in law that the probability can be so readily established. And as my old buddy, Rummy said, then there are the unknown unknowns, which we can’t even imagine because they’re, well, unknown.