The Intercept, which does some of the most consistently excellent and accurate long-form reporting on criminal law issues, does a blistering expose on facial recognition technology. Much as I’m not a fan of the trend of hooking the reader with an anecdote, Ava Kofman’s nails her point:

It was just after sundown when a man knocked on Steve Talley’s door in south Denver. The man claimed to have hit Talley’s silver Jeep Cherokee and asked him to assess the damage. So Talley, wearing boxers and a tank top, went outside to take a look.

Seconds later, he was knocked to the pavement outside his house. Flash bang grenades detonated, temporarily blinding and deafening him. Three men dressed in black jackets, goggles, and helmets repeatedly hit him with batons and the butts of their guns. He remembers one of the men telling him, “So you like to fuck with my brothers in blue!” while another stood on his face and cracked two of his teeth. “You’ve got the wrong guy,” he remembers shouting. “You guys are crazy.”

Of course, Talley was the wrong guy. Totally the wrong guy, not that it did him much good.  Talley is the same guy Noel Erinjeri told you about at Fault Lines last April, an unforgettable reminder than you can beat the rap but not the ride.

Talley was released in November, and the charges were apparently dropped. In the months that followed, a series of medical exams revealed that Talley had sustained several injuries on the night of his arrest, including a broken sternum, several broken teeth, four ruptured disks, blood clots in his right leg, nerve damage in his right ankle, and a possibly fractured penis. “I didn’t even know you could break a penis,” he told me.

But the facial recognition software said he was the bank robber, and technology is so very scientific. If cops can’t rely on technology to break a guy’s penis, what can they rely on?

But while the accuracy of other visual, pattern-matching methods like blood-splatter analysis has been subject to vigorous public debate, the fallibility of facial comparison, or facial identification, has received less attention.

Facial recognition technology is relatively new, and has all the gloss that makes us want to love it. After all, not only will it solve crimes, but it will do so with science, that magic that most of us find so alluring despite having no clue how the magic happens.

This may be because comparing facial images can seem like an easy or even intuitive task. “We assume, wrongly, that we are good at recognizing faces,” said David White, an Australian psychologist who researches facial perception. In reality, however, we are for the most part terrible at comparing photographs, video stills, and composite images of unfamiliar faces — and this remains the case even with high-quality, full frontal images.

The photographs of Talley and the suspects were sent to the FBI’s Forensic, Audio, Video and Image Analysis Unit, where trained forensic examiners manually compare points of similarity between faces to help investigators confirm or eliminate the identities of potential suspects. After selecting frames from the video, they work back and forth between evidence from the crime scene and images of their suspect to develop a conclusion regarding the type and number of similarities.

Look behind the curtains and you find that this tech, like its ancestor, fingerprints, even drug analysis and DNA, leaves tons of room for error. The tech itself may be reasonably valid, but it remains exposed to the weak links in the chain, which usually involve humans. The problem for those of us who either don’t have a clue how the magic is achieved or merely prefer to believe in anything that sounds remotely science-y is that we believe in its inherent magic and never waste time worrying about Steve Talley’s broken penis.

Read Kofman’s post at the Intercept and remember what it says about our love of things that seem scientific and reliable because, well, science. Then remember Talley’s broken penis and ask yourself whether you’re prepared to take that for the team.