One of the saddest realizations that comes of spending time online is that there are a lot of really sick, disgusting people out there. I mean, a lot more than I ever suspected before the internet.
Sure, we always knew there were sick people out there, but they weren’t in our faces. And they weren’t able to find each other easily, which meant they didn’t have the support and comfort of knowing that they weren’t alone in their embrace of whatever warped thing turned them on.
But then, what protects the rest of us from suffering their pleasure? Adrian Chen at Wired gives us a view into the perverse world of content moderation.
As social media connects more people more intimately than ever before, companies have been confronted with the Grandma Problem: Now that grandparents routinely use services like Facebook to connect with their kids and grandkids, they are potentially exposed to the Internet’s panoply of jerks, racists, creeps, criminals, and bullies. They won’t continue to log on if they find their family photos sandwiched between a gruesome Russian highway accident and a hardcore porn video.
Let’s dispense with the obvious up front. There is no First Amendment violation when private businesses decide that they don’t want to host images of beheadings. Only the government can violate your rights. Private businesses or citizens have no similar obligation, even though you may really wish they did.
And because the article was written by Adrian Chen, who struggles with separating his personal agenda from his journalism, he had to toss in his pet peeves like racists and bullies, which are sufficiently vague to cover conduct that might well be moderated, but would be more likely not, as the point isn’t to silence thought, but to protect grandma’s eyes.
Social media’s growth into a multibillion-dollar industry, and its lasting mainstream appeal, has depended in large part on companies’ ability to police the borders of their user-generated content—to ensure that Grandma never has to see images like the one [21-year-old Filipino Michael] Baybayan just nuked.
And that’s where it all gets totally weird.
So companies like Facebook and Twitter rely on an army of workers employed to soak up the worst of humanity in order to protect the rest of us. And there are legions of them—a vast, invisible pool of human labor. Hemanshu Nigam, the former chief security officer of MySpace who now runs online safety consultancy SSP Blue, estimates that the number of content moderators scrubbing the world’s social media sites, mobile apps, and cloud storage services runs to “well over 100,000”—that is, about twice the total head count of Google and nearly 14 times that of Facebook.
Perhaps you knew of this army of people whose purpose was to delete the images that make other people gasp and wretch, but I didn’t. It reminds me of how naïve I am, how hard it is for me to fathom why there is any need for this. But clearly, there is a need, and the need is huge.
A list of categories, scrawled on a whiteboard, reminds the workers of what they’re hunting for: pornography, gore, minors, sexual solicitation, sexual body parts/images, racism. When Baybayan sees a potential violation, he drills in on it to confirm, then sends it away—erasing it from the user’s account and the service altogether—and moves back to the grid. Within 25 minutes, Baybayan has eliminated an impressive variety of dick pics, thong shots, exotic objects inserted into bodies, hateful taunts, and requests for oral sex.
This job isn’t nearly as much fun as it may seem.
Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue
“Oh fuck! I’ve got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos.
Who are we? What are we doing? I realize that there are many who see this as part of the lulz factor, the occasional bizarre image that gives us that taboo thrill of being naughty and grossed out. But when you realize that it takes 100,000 people to clean up this cesspool, does it not make you realize that this reflects far too much sickness to explain away?
The cost of cleaning up the internets, both social and economic, is huge, even when the word is outsourced to the Philippines so that internet companies don’t have to pay US wages to get it done. But that doesn’t make it any better or more justifiable. Filipinos are human beings too, and they don’t need to see these images any more than some dude in Idaho.
But what this says about people is the most basic concern. Are we really as sick and disgusting as this article suggests? Do we really have to be? Is the anonymity of the internet an aphrodisiac for our worst, most disgusting impulses?
By no means do I suggest that the solution is to enact law to control the worst of human nature to express itself. But the flip side is that we don’t have to indulge our worst natures, even if we can do so anonymously, even if we can get away with it.
Clean up your act. Before posting some pic that will make grandma cry, ask yourself if this is really who you are. Ask yourself if this is really who you want to be. If not, then don’t do it. Just because you can be sick and disgusting doesn’t mean you have to be.
Sometimes it does seem like a worldwide electronic bathroom wall.
It may not be that bad, though. For one thing, one really dedicated crazy person can put a lot of stuff on the internet. They’re the reason we can’t have nice things. (I don’t want to name names, but I ‘ll bet you can think of an example or two.) And although the content moderators only have to remove the bad content, they nevertheless have to examine all the content. So that figure of 100,000 moderators is for everything, not just the crap. Also, there are an estimated 2.9 billion people on the internet, which means it takes about 29,000 internet users to keep a moderator busy. That’s bigger than a lot of suburbs and towns, and many of them need more than one cop
Or we can do our best to persuade others to use some better personal judgment, responsibility and impulse control, rather than rationalize their failure to do so.
Anonymity makes it difficult to get people’s attention in real life. Maybe we need more Curt Schillings around to show the would-be provocateurs that true anonymity is not that easy to achieve, and that being really mean can adversely impact your own quality of (real) life.
Sure, why try to persuade people to do the right thing when we can cause harm to people instead. Cool idea, bro.
I’m beginning to have some serious concern about the people who read SJ.
Obviously, it’s much better to teach lessons without inflicting pain if possible. But it seems that the lesson that “karma’s a bitch” is one that some people can’t be taught without pain, and there will always be people ready, willing, and able to mete out the pain. Some, like Curt, will do it publicly with words. Wait, how can he do that? Words don’t hurt, do they? Others — well, maybe they become your clients, I don’t know.
In any case, I sincerely hope that wide publication of Curt’s actions will give some people pause, and I’m sorry if my idle speculation about whether the world would be a better place if people knew their hurtful words might come back to bite them creates more pain.
Don’t be ridiculous, there’s no way content moderators are examining all the content. Practically every major content site has a mechanism for customers to report offensive content. If the site is sophisticated, and you’re logged on when you express your displeasure, they may track how well your reporting matches the final outcome. This would help them prioritize what to examine, and maybe save money by allowing them to lay off some moderators, since they can count on your good judgement.
Second, there’s no discussion of the international side of this business. If the online service conducts significant business (i.e. payment processing) in a particular nation, they are far more likely to do the work to stay abreast of the standards of that nation. Nothing like seizable bank accounts to spur obedience to the local authorities. You’ll find that other nations have significantly different tolerance for speech then here.
Finally, online businesses want to build significant value in the accounts that you hold with them. As a result, there are a bunch of evil people who believe they can make money by stealing your account credentials and laundering them in some fashion, or using your account in some sort of money making scheme. These kinds of folks only make money in volume, so they need to spam a lot of crap to find the suckers, and they will do so in entirely automated ways. Pro-actively pulling down this content can be a smart business move – you can measure your success against the costs you incur in support costs restoring people’s accounts.
If you want to see some examples, just search youtube for something like ‘How to hack Facebook account’. You may come across the thoughtful advice of some hacker who has anticipated your desires, and has kindly provided all the software you need to pursue your passion, easily available in the download link provided with the video. But in a twist that’s common to many hollywood movies about con-artists, you may actually be the mark!
There are many dirty jobs. I don’t know how many content moderators is too many, but the guesstimate of 100,000 comes from a single source with a financial interest. He may be well-informed, or not.
The better-supported figure of 2 million people who staff the great firewall of China gives one an idea of how hard it can be to keep 650 million people from finding what they want.
I’m skeptical about the article’s implication that the work is so awful it injures those who do it. Dirty work isn’t for everyone, but there are literally millions of people who do them and are found to be at least as happy of clean-hands workers. If the story had quoted some people who enjoyed their work, or stated that the author tried and couldn’t find any, it would have been more credible.
That may be, but then that’s really not what this post is about. I wish computer guys didn’t constantly get hung up on the piece that directly involves them and could see the larger context. Maybe some day.
If the actual total is closer to 10,000 internet cesspool cleaners, would you have felt the same amount of concern? The post suggested to me that it was the size of this army, not its mere existence, that concerns you. Perhaps I continue to misunderstand. With nearly 3 billion people on the internet, the vast majority of them already keep things pretty clean, or the army would be much larger. Exhortations to do better are fine, but it seems to be a rather small minority who won’t heed them.
Sigh. First, have you met my good friend the reply button? Second, your pulling 10k out of your ass doesn’t make it real, or create a basis to make it worthy of discussion. Third, you assume my reaction. The 100,000 cleaners really didn’t mean much to me. And finally,
A “rather small minority” because you say so. Got it.
My apologies for the reply gaffe.
Yes, I made up the 10,000 figure – and so did the self-proclaimed expert in “safety, security, and privacy” who was quoted about 100,00 cleaners in the article. That field is so polluted with fakirs that unsubstantiated claims should be treated with the same degree of skepticism used in evaluating figures tossed out for “sex trafficking”
The mere existence of this particular dirty job makes you sad?
I infer a rather small minority because the army of internet cesspool cleaners isn’t 2 million or 10 million strong. That would make cesspool cleaning a huge industry and probably the largest employer in the Philippines. Hard not to notice. Yet Facebook and its ilk are fairly clean, so it seems unlikely that there is even 10% of internet users – roughly 300 million people – who are in the habit of posting filth.
While I’m not a fan of Adrian Chen’s (which means I take what he writes skeptically), this wasn’t a critical number and I will assume, for the safe of his article, that he didn’t pull the number out of his ass. He may be way off based, but that doesn’t change what he describes in the article, and that’s really what matters here.
You have no idea what I see just moderating the comments here. It’s ridiculous. I am disgusted. We don’t have to be this way, and to the extent the internet empowers people to indulge in the worst, it’s a shame. My buddy, Marco Randazza (who is hardly an enemy to vulgar language and content), is of the view that Section 230 has got to change because people are absolutely out of control online. I disagree, but then, the point here is to get people to stop being assholes of their own accord before the ruin this place.
And you’re not helping.
I do not get that impression from Mr Randazza at all. He seems a Section 230 supporter to me, saying things like, “Section 230 has been a wonderful thing. It has allowed the Internet to grow, and allowed services like Facebook, Craigslist, Fling.com, Pissedconsumer.com, and any number of other fun websites to exist. It allows me to have a comments section on each post, without worrying about whether I’ll be liable for something posted there. It does foster free speech online. So hooray Section 230.”
https://randazza.wordpress.com/2014/03/15/no-section-230-does-not-prohibit-you-from-being-responsible/
Of course. Forget my conversations about this with Marc and let’s focus instead on your impressions.
“the point here is to get people to stop being assholes of their own accord”
Start at home. You assign an opinion to someone else online where he publicly holds the opposite? Asshole move man.
Are you brain dead? Not only has Marco said so publicly, but he and I (we actually know each other and are friends) have actually discussed this in actual real life. And if you think this was an “asshole move man,” why not shout out to Randazza, tell him about my asshole move so he can correct me?
Hey Dumbass (that would be you, Mike). When someone who uses a real name and is actually identifiable takes personal responsibility for saying a discussion actually happened, he isn’t “assigning an opinion,” you flaming moron, he is telling a dumbass like you what the fuck actually happened.
Asshole move, Mike.
Nice ad hom, and appeal to authority. I quoted my source, you did not.
I am the source. There’s something seriously wrong with you.
I’m a glutton for punishment I guess. If I wrote an article about a subject a year ago, but then recently expressed my doubts to you privately, I would be rightfully upset about being called out as a flip-flopper online. I couldn’t find anything on his blog or twitter about a change of heart.
Awaiting your apology, which I expect to include some small demonstration that you grasp the myriad ways you were absurdly wrong.
Whatever. Trolls don’t apologize. I’m not even convinced I’m wrong. I get pissed when a friend posts an unflattering picture of me online, let alone says I’m in favor of reduced protections for free speech online.
Your choice.
Mike,
You’re the asshole here.
Sure, you he’ll believe. Me? I’m not a source.
Edit: Apparently, he doesn’t believe you either. Sheesh.
Am I missing something? Marc Randazza publicly and unequivocally supported Section 230. “Hooray for Section 230” is about as clear as one can get. This was not just an impression of Mike’s. If Marc then privately in a conversation with SHG told him that he has changed his mind– and I have no doubt that such a conversation took place– why hasn’t Marc put up a post to that effect? If he already has, why don’t we see a link to it?
Seg. Schultz: Your habit of always adding your own insults to those of SHG is really annoying, particularly since you are anonymous. I is sycophancy of the worst degree.
Yes, you are missing something, as you so often do. Marc’s position on Section 230 is nuanced and deeply thoughtful. (Note: he has made this position clear online in the past, but I can’t recall where and appeasing dumbasses isn’t worth that much of my time to scour the internet to find it. As Marco has noted here, Mike was wrong. He too wasn’t sufficiently interested to spend his time explaining his position here and now to a dumbass. I am doing you the courtesy of responding to your comment. You don’t deserve my courtesy or my time.)
Marco lauds much about Section 230, but it is not the simplistic grasp you (and Mike) take away. One can be strongly in favor of a law as applied to many aspect of conduct, while being capable of seeing others where the law may not work well, may require finer tuning, may require modification.
A deeper understand would enable a lawyer to appreciate that Marc thinks this because if the law doesn’t get “fixed,” we may well lose the law in its entirety, all that “Hooray for Section 230” stuff, because of the problematic aspects. Of course, Mike isn’t a lawyer, so his inability to appreciate a nuanced understanding of law is somewhat understandable. You, on the other hand, are just a dolt.
As for Sgt. Schultz, he tends to be a bit flagrant in his annoyance with stupidity. Others send me emails asking why I allow comment threads to devolve into non-lawyer ignorance rather than delete all the stupid comments. But then, I allow your comments as well as his. Want to guess whose comments would not be missed?
Exactly.
Certainly, the referenced article does lament that Section 230 is not long for this world, “One day, I expect that Congress will take away Section 230 (at least in part).
Why?
Because we can’t have nice things… One day, it is going to be the wrong Senator’s daughter who gets a rape threat or defamed on a comment somewhere, and you’re going to see some rollback to the freedom that Section 230 has given us. And that will be too bad, because it didn’t need to be that way.”
But then he goes into the same “stop being assholes of their own accord” type argument as presented here. I’m curious where can I go to get a better understanding of how such high-minded lawyers as yourselves believe the law should change to prevent such a disaster? Better yet, how can we convince the legislature to do anything without a cause celebre?
So I’m trying to decide whether I should post your comment given that you’re an unapologetic troll posting comments on my dime. But then, that would be petty of me.
You’re still making assumptions about what Marco is saying and what I’m saying that aren’t accurate. This isn’t a post about fine tuning Section 230, and I regret bringing it up at all given where it’s gone, so I’m not going to perpetuate it because you’ve decided that you really need to discuss this further.
I expect a time will come when Marco or I decide to write about changes to Section 230, but today isn’t that day and this isn’t that post.
I figure the least I can do is give Lawrence an explanation, so here goes.
Almost everyday, I read SJ. I don’t pay for it. It’s there for me whenever I want to read. I usually learn something here. Even when I don’t necessarily agree with it, it’s still interesting and makes me think. And when I disagree, or don’t learn something, I lose nothing because some guy named Scott writes it, puts it up on his blog, and let’s me read it.
Am I his sycophant? Maybe I am. I appreciate the fact that Scott does what he does. I appreciate the fact that SJ exists, that Scott keeps writing and that it’s here for me, even though I do nothing to deserve it. Does that make me a sycophant? If so, then I am.
And I am amazed at the nonsensical comments people leave. Not all, but some. Most of the people who write comments that disagree with Scott just lack the grasp to understand what he’s talking about. It’s not because they disagree, but because it’s obvious from their disagreement that their problem is their own stupidity, not actual disagreement.
There is no doubt that Scott is fully capable of dealing with stupidity on his own, and he doesn’t need my help. But reading some of the comments, I think the least I can do to contribute to him in exchange for what he’s given me is to put my two cents in. So I do.
You have a problem with that? Tough shit. I owe Scott some appreciation. I owe you nothing. As Scott wrote, he spent his time explaining things to you as an undeserved courtesy. I do the same.
If you don’t want to be called stupid, then don’t be stupid. And it that makes me a sycophant, then it’s the least I can do for what I’ve gained at SJ.
You’re welcome.
No doubt because I am a dolt I fail to understand why Marc Randazza could not have written a short one sentence response to Mike to the effect that “While I have written in support of Section 230, I have also posted about the need for its fine tuning.” But I guess it’s much more “nuanced” to write “You’re the asshole here.”
Sgt. Schultz: I too appreciate SHG’s blog and think he is performing an important public service. Indeed, I have told many of my friends that the blog has made me more sensitive to defendant’s rights and police and prosecutorial abuse than I was previously. And most of my comments on the blog have been supportive. And there was one time I corrected an error of SHG’s which he was kind enough to acknowledge. But I still think that at times SHG unjustifiably insults bloggers for simply having the temerity to disagree with him. If that makes me stupid or a dolt or an asshole, so be it.
So your next issue is with Marco for not responding in a way that you find suitably pleasing? Because he owed Mike (or you) a suitably pleasing response?
And I don’t meet your approval “at times” either? Well then.
Sgt. Schultz: At least I, like SHG and unlike you, post under my own name. I believe that people who post anonymously, as you do, should think twice before calling people, particularly those who have names, stupid and assholes. Incidentally , I did not call you a sycophant. I said your habit was sycophancy.
Apparently using a pseudonym makes you count less according to Kaplan . by his reasoning, i use not only my real name but link back to my site which makes me super duper authentic. With that said, I second everything Sgt Schultz said. Realizing I’m violating the rule I’m about to espouse – the “listen, shut up and learn” rule works really well here. On an all too often basis, stupid comments annoy SHG to the point he starts to rethink whether or not to keep SJ going . When that final comment pushes him over the edge and when it’ll be is hard to know, but I’m 100% positive that he won’t decide to stop SJ b/c he didn’t get enough commentary on his posts. I’m willing to take Judge Kopf’s advice for SCOTUS and STFU, maybe if Kaplan and Mike did the same, Sgt Schultz wouldn’t need to come in and clean house, and SHG and Marc could back to making the world a better place. If it’s good enough for SCOTUS, it’s good enough for me.
Bill: I did not say nor mean to imply that the substantive opinions of people who post anonymously count less. I said that such anonymous posters should think twice before insulting other posters, particularly those who post under their own names, and calling them stupid or asshole. Since you are clearly not stupid, I must attribute your failure to appreciate this rather elementary distinction to your being deliberately obtuse.
The fact that that overall both SHG and Marc do such outstanding wok, for which I and so many others are grateful, does not mean that they should be immune from criticism for their often intemperate replies to critical disagreement. Evidently you fail to appreciate this distinction as well.
Lawrence, please just stop. SHG hates , HATES stuff like this, your comments and mine alike. Irrelevant, OT back and forth exchanges have been the soup de jour on the internet since Prodigy and Compuserve. You think they shouldn’t be immune, I think i’ts bad manners to annoy your host especially when he invites you over for free dinner (and dinner is top notch). He’ll find this comment (the spirit of which he agrees with) every bit as annoying as your’s or Mike’s so please, can we just collectively shut up, flip back over to read-only mode and leave it there?
There hasn’t been bread and circus in a couple millennia, so the internets have to do for now… Even the Romans had a small army working in the Hypogeum filling up their “Spoliarium” – back then Grandma’s didn’t like looking at or smelling bodies in the Tiber either.
Are you vying for worst analogy of the day? Maybe week?
I wish, I am not even close.
While I agree completely with the point you’re making, I think your comment
is an example of the more offensive part of the spectrum. People have never had so much access to information and yet without including such a caveat, there are probably countless idiots that would yap about 1st Amendment violations. Cries about private companies violation freedom of speech are as common as people claiming a blog post is slander or that any insult is an ad hominem . Being an’edgy’ asshole is certainly overplayed, but methinks sheer stupidity is much more of a scourge than dic pics.
They’re not mutually exclusive.
Speaking of something being hard to fathom and beheading videos, for the life of me, I cannot fathom why anyone would click on one to watch. I haven’t watched one and I will not. Call me crazy but I expect I’ll be better off for it.
I haven’t watched one either, but then, that’s my choice (as it is yours).
“But that doesn’t make *is* any better or more justifiable.” Just brazenly correcting this sentence before cowardly disappearing back into the ether. PS Good title.
Everybody else missed that, so thanks.