Why Does The Sentence-O-Matic 1000 Hate Blacks?

Following a debate between Judges Kopf and Bennett over the whimsy of discretionary sentencing, the omni-geeky Jake DiMare came up with the idea of the Sentence-O-Matic 1000.

Judge Richard Kopf raised the question of whether the language was so devoid of meaning and guidance as to render § 3553(a) worthless.  Judge Mark Bennett responded “Sentencing requires us to weigh that which cannot be measured,” to which Judge Kopf replied: “Let’s be honest then and declare that sentencing is entirely a matter of discretion…”  If so, this raises the specter of sentencing being so arbitrary and capricious, so captive to any judge’s whim, as to be a total crapshoot.

Jake offered this reaction:

Was there ever a task in the courtroom more ripe for automation?

Jake had a point.Remove the feelz component from the mix and reduce sentencing to cold, hard empiricism.

As a representative of the ignorant masses, I find comfort in the notion that everyone would be given sentences using the same criteria, and never again subjected to the whimsy of some of the judges.

There is, without question, an allure to the notion that people’s lives would never again be at risk of the vicissitudes of judges. Then again, the outcome would be at the mercy of the algorithm. No one could create an algorithm that would cover every variable, or balance them adequately, given the nature of the human condition.

As Judge Bennett noted, “Sentencing requires us to weigh that which cannot be measured.”  If it can’t be measured (and it can’t), then it can’t be input into the Sentence-O-Matic 1000.  And even if it was input to the best a bunch of programmers could manage, it couldn’t be subject to the arguments that real life human beings require to address the individualized circumstances that exist in every case and every sentence, because that would obviate the goal of consistency and return us to the whimsy of the judge.

Pshaw, you geeks and empiricism-lovers reply? Or perhaps, it’s imperfect but it’s less imperfect than one crazy old coot in a black robe who thinks he has magic powers because somebody handed him a gavel? Fair enough, except the hard, cold brutality of the algorithm has already found itself in the crosshairs of social justice.

The ACLU has begun to worry that artificial intelligence is discriminatory based on race, gender and age. So it teamed up with computer science researchers to launch a program to promote applications of AI that protect rights and lead to equitable outcomes.

You have science. You have “equitable outcomes,” whatever that means. So mix and shake?

Opaque and potentially biased mathematical models are remaking our lives—and neither the companies responsible for developing them nor the government is interested in addressing the problem.

This week a group of researchers, together with the American Civil Liberties Union, launched an effort to identify and highlight algorithmic bias.

Can math be biased?

Algorithmic bias is shaping up to be a major societal issue at a critical moment in the evolution of machine learning and AI. If the bias lurking inside the algorithms that make ever-more-important decisions goes unrecognized and unchecked, it could have serious negative consequences, especially for poorer communities and minorities. The eventual outcry might also stymie the progress of an incredibly useful technology (see “Inspecting Algorithms for Bias”).

Of course it’s not the fault of math, but measuring outcomes demonstrates a disparate impact based on race. If there is a disparate impact coming out the back end, then there must be bias in the algorithms on the front end. How do we know this? Because we believe that race cannot be a factor. It’s not that algorithms expressly take race into account, but use proxies for race, such as poverty, educational attainment and prior arrests and convictions. And reliance on algorithms is showing bias elsewhere as well.

Examples of algorithmic bias that have come to light lately, they say, include flawed and misrepresentative systems used to rank teachers, and gender-biased models for natural language processing.

Cathy O’Neil, a mathematician and the author of Weapons of Math Destruction, a book that highlights the risk of algorithmic bias in many contexts, says people are often too willing to trust in mathematical models because they believe it will remove human bias. “[Algorithms] replace human processes, but they’re not held to the same standards,” she says. “People trust them too much.”

This is true, and simultaneously false. An algorithm is only as good as its creation, the factors and weights given. It’s not as if AI has ill intent, but may be programmed in a way that incorporates biases. Prior convictions, for example, are certainly a critical factor in determining the efficacy of punishment, but if cops are far more inclined to toss black guys than white guys, then black guys are going to have more significant criminal records. Garbage in, garbage out.

The problem, of course, is that infusing the algorithms with bias to thwart bias means that we traded one bias for another, not that we’ve eliminated bias from the mix and the Sentence-O-Matic 1000 suddenly works. To blame it on the algorithm deflects responsibility for the problem. Algorithms only do what they’re told to do. They don’t feel happy or sad, love or hate, fair or unfair. They’re cold. Not like some judges are cold, but cold in the exercise of their harsh objectivity, which is only as objective as the inputs.

More to the point, we trust the math. Even worse, by trusting the math, the power of empiricism, it relieves humans of the emotional and intellectual burden of performing unpleasant tasks. Algorithms have no feelings, so if the output says one year or one lifetime, we can shrug, blame it on the Sentence-O-Matic 1000 and sleep well that night, knowing that it wasn’t our fault that some guy got life, but the machine’s.

So are the geeks and the ACLU fixing the problem?

MIT Technology Review reports that the initiative is the latest to illustrate general concern that the increasing reliance on algorithms to make decisions in the areas of hiring, criminal justice, and financial services will reinforce racial and gender biases.

If the inputs reflect bias, then so too will the outputs. Yet, if geeks game the inputs to eliminate perceived bias, it will still reflect bias, but of the opposite sort. The underlying premise is the belief that there can be no racial or gender distinction, such that any disparate impact must, by definition, mean that the algorithm is biased.

But if we’re to hand over discretion to ruin lives to machines, as geeks would like, and relieve judges from the unpleasant task of making hard decisions, will the purely objective outcome of AI be better if tweaked to reflect the purely subjective input of feelz? More to the point, no matter how we tweak the algorithm, will we ever be capable of a less biased outcome than we do under a regime where we can argue to judges whatever factors specifically apply to the individual before the court, rather than the 32 inputs some geek decided fits all defendants?

In many instances, an algorithm may well produce a less emotional, less biased outcome than a human judge who is, or isn’t, in touch with his feelings. But given the way in which the legal system falls back on technology, mindlessly accepting whatever outcomes it spews out, are we prepared to hand over lives to a machine whose purely objective, and absolute, determinations are as reliable as the humans who screwed with the objective factors to tweak the impact? Once we tweak empiricism with feelz, the only guarantee is that the output isn’t objectively sound.

Bear in mind, even if you’re good with the ACLU playing with the geeks to game AI at the moment, somebody will invite DoJ or the NDAA to the table at some point to seek their input on your client’s outcomes. If it’s not going to be purely empirical, purely objective, then whose finger is on the button matters. And even with the ACLU in the mix, will eradication of racial and gender bias mean black guys get a break, or that white guys and women get longer sentences?

20 thoughts on “Why Does The Sentence-O-Matic 1000 Hate Blacks?

  1. LTMG

    Some questions:

    Would it be possible to devise sentencing algorithms the produce a range for a given sentence rather than an absolute number? Yes, it is technically feasible.

    Would it be possible to legislate that judges have discretion to vary the output from a sentencing algorithm +/- a certain percent? Yes, it is possible for legislatures or policy makers to do this, but will they?

    Would it be possible for defense attorneys to be allowed to know the math behind the algorithms and, more importantly, the assumptions that go into the equations? Based on what I’ve read in the past year, companies making these algorithms claim trade secrets and will not reveal what’s inside the algorithms. This is a serious problem, I believe.

    Would it be possible for the ACLU, or anybody else for that matter, to do statistical hypothesis testing to determine whether algorithmic or judicial sentencing are in any way biased. Yes, absolutely. What are the barriers to doing so? Just need to gather the publicly available data. The number crunching is straightforward.

    1. SHG Post author

      All of this would be possible. None of this remotely addresses the problem. (No, I’m not interest in your next five comments explaining why you think it does. They will be trashed.)

  2. PDB

    “It’s not as if AI has ill intent, but may be programmed in a way that incorporates biases.”

    I’m not sure how true that statement is. If these are learning systems, then they’re not really being “programmed.” Is is more likely that the machines are fed large volumes of data and then come to the conclusions on their own (through pattern recognition and the like). So essentially, I think these machines are coming to “biased” conclusions on their own, not through anything that the programmers have done in their hard coding. Kind of like the way judges, through seeing years or decades of cases, come to their own conclusions through pattern recognition and the like.

    Ultimately, what this means is that AI machines – which are supposed to ultimately mimic human thought – can become as “biased” (or racist, sexist, pick your favorite epithet) as humans. So for all this, we are right back where we started. Until black people stop committing crimes in disproportion to their population in this country, until women behave 100% identically to men, you’re going to have these biased outcomes, human or computer.

    1. SHG Post author

      Take that same thought in a slightly different direction: the machines are fed raw data and reach conclusions that are contrary to our rejection of disparate impact. Does that make the machines wrong or us?

  3. jim ryan

    Step 1:
    Machine Learning/Deep Learning. Take all the cases (Big Data) and results (sentences) and feed all known factors into then machine (tensorflow and/or MXNet) and let the machine learn. Then run in parallel with actual and note the differneces and feed them back into the machine. Soon you’ll have your answer. But be careful of what you wish for.
    Step 2:
    Why bother with the trial at all? Just take all the facts and information and develop the AI model. It would certainly lower costs (no DA’s, CDL’s or Judges) and the results could arguablly be proved more fair and balanced than the current system.
    Step 3:
    Then for our next act of inanity, we’ll extend this and call it “Minority Report”

    1. SHG Post author

      “I want empiricism to eliminate human bias from the system.”
      “Empiricism results in racially disparate impacts”
      “I want empiricism that comports with my good bias, not their bad bias.”

  4. Richard Kopf


    The ABA’s manic compulsion to rid the world of bias will I predict with certainty ignore empirical truths (such as race is a valid predictor of re-offense) when the ABA studies risk prediction instruments. On the other hand, the ABA will surely latch on to data that warms SJW hearts–like that predicting women are less likely to offend again than men.

    There are really good people around the world (such as Dr. James C. Oleson, a former US Supreme Court Fellow, at the University of Auckland in New Zealand) who could help but such persons don’t flinch from data they otherwise hate. As Colonel Nathan R. Jessep once said in a surprisingly related context, the ABA can’t handle the truth.

    All the best.


    1. SHG Post author

      As we’ve discussed in the last go-round on the Sentence-O-Matic 1000, when *all* factors are adequately taken into account, who knows what the “truth” will be. But as much as I don’t trust judges to be free of bias, I don’t trust programmers, devs or AI machines to be rid of algorithms that use proxy criteria for bias either. And the worst part of it is that if were every capable of truly ridding bias from the mix, as you’ve noted, we may really hate what we find out.

      1. Richard Kopf


        Please forgive the reference to the ABA when you wrote about the ACLU. Perhaps you will appreciate my confusion and conflation.

        You write: “[W]e may really hate what we find out.” Indeed, swords cut both ways.

        All the best.


    2. B. McLeod

      ABA has no intention of ridding the world of bias. The goal is to enforce conformity with ABA-approved bias.

  5. Patrick Maupin

    As an Anonymous Coward posted on slashdot:

    It’s simple, really — we just need to develop a SJW AI to harangue the other AIs about their biases, real or perceived.

    We can then offload all political nonsense to the AIs, who will be too busy fighting with one another to go full Skynet on the rest of us.

  6. Jeff Gamso

    Of course, the Sentence-O-Matic 1000, even if it worked with the perfection fantasized, wouldn’t do a damn thing to solve the larger problem of prosecutor’s controlling the system. For that we’d need the AUSA[and-your-state-equivalent]-O-Matic 2500.

    Then again, the Sentence-O-Matic 1000 is really just Guidelines 2.0. And we know how well the Guidelines work.

  7. B. McLeod

    It looks like part of the issue here must be whether the inputs coming to the algorithms are already tainted by the alleged bias, for example, the use of prior arrests and convictions. It is absolutely typical today for criminal codes to provide enhanced penalties for prior convictions. If the thought is that this is unfair to minorities because they have excessive priors, due to police bias in disproportionately arresting them, or prosecutor bias in disproportionately charging them, or jury bias in disproportionately convicting them, that all happens upstream from the algorithm. A racially neutral algorithm is not going to be able to “adjust” for that, because the bias is invisibly inherent in the inputs. So, sentencing algorithms can’t solve the problem, unless we also install police, prosecutor and jury algorithms, and disqualify all prior convictions that were entered prior to the time those algorithms were in place (or alternatively, stop using priors as a factor, for which a rational case can certainly be made).

    1. SHG Post author

      Some of the best indicators also happen to serve as racial proxies. You can’t get rid of one without the other.

Comments are closed.