Example: Content Moderation

Before we introduce VSD, let’s take a deeper dive into a specific and pervasive example of ethics and values in technology: content moderation. Many online services act as platforms that host user generated content: Facebook and Twitter host text (as well as other forms of media), Instagram and Snapchat host images, YouTube and Tiktok host videos, Yelp and Tripadvisor host reviews, Amazon and ebay host product listings. These platforms like to claim that they are neutral conduits for information, open to all people and all forms of speech. For example, when being questioned by US lawmakers about the potential for political bias in Google Search results, Google chief executive Sundar Pichai said:

“We build our products in a neutral way… We approach our work without any political bias.”

The idea of neutrality is key to the ethos of large online platforms for several reasons. First, it resonates with the thinking that technology in general is neutral (an idea that we are in the process of debunking). Second, it positions the online platforms as “free and open” in contrast to “old media” companies, like television and newspapers, that are heavily curated by editors. Third, it enables the platforms to try and sidestep responsibility and shift blame to users when they post problematic content. Fourth, it is a useful stance for minimizing liability with respect to laws like Section 230 of the Communications Decency Act.

Yet, content moderation is fundamental to how online platforms work. Some content is moderated to comply with the law, for example when copyrighted material or child abuse images are taken down. But other content is moderated by online platforms voluntarily – all of the major online platforms have rules, sometimes called “community guidelines” or “community standards,” that delineate what kinds of content are not acceptable. Examples of prohibited content often include hate speech, bullying, and pornography. In other words, these platforms are not neutral: they make choices about what kinds of content that they believe are unacceptable.

Furthermore, content moderation is not just about filtering or removing content, it is also about prioritizing and highlighting content. All of the platforms offer search functionality that attempts to identify and sort the most relevant content related to a particular query. Amazon uses a recommendation algorithm to highlight products that it believes are relevant to each shopper. Facebook is organized around a news feed that prioritizes “engaging” content. And if you don’t want your content to be subject to the whims of algorithmic ranking or recommendation, you can often move your content to the top of the queue by paying money to promote it as an advertisement.

Again, it is clear that online platforms are not neutral: they are highly editorial and curatorial, relying on a mix of algorithms and human beings that make decisions about content availability and visibility on their platforms. Moreover, we can see that the design of these moderation systems reveal some of the values held by platforms. Most platforms espouse a commitment to free expression, preferring not to take content down if at all possible. Community guidelines that prohibit hate speech and bullying suggest upholding the values of human welfare and respect. (Although some platforms’ degree of commitment to these values is perhaps questionable, as we will see).

If you are interested in a much deeper discussion of content moderation by large platforms, we recommend the book “Custodians of The Internet” by Tartleton Gillespie. A PDF copy is available for free here.

Content Moderation Controversies

Content moderation presents a cautionary tale of how values can collide. Following the 2016 US Presidential election, platforms like Facebook, Google, and Twitter were embroiled in controversy over accusations of political favoritism. Conservative politicians accused the platforms of censorship and bias against conservative content. The representatives of the platforms were unable to effectively defend themselves because their platforms are not transparent, i.e., it is not clear to those outside the companies how their moderation systems work. Moreover, their insistence that the platforms are strictly neutral is, as discussed above, false and disingenuous.

No system of content organization can be neutral. They are all designed systems. The platforms must make choices about what content to allow, what content to prioritize, how to display content, and so much more. These choices are made based on their goals, constraints, priorities, and perspectives. They thereby reflect their values. Rather than trying to pretend that their platforms are neutral, a better response from the companies would have been to acknowledge that their platforms are not neutral (i.e. they reflect values); be explicit that one of their values is impartiality among reasonable political perspectives (e.g. those that do not espouse violence or subjugation of others); and describe the methods by which they try to realize that value in the design and operation of their content moderation systems. Such a response would engage the bias concerns in a reality-based, transparent, and potentially productive way, rather than through specious claims about neutrality that only lead to further accusations.

The online platforms’ commitment to impartiality with respect to political views sounds like a noble position, especially in America where free expression is held up as a fundamental right. However, as suggested above, an unwavering commitment to free expression is complicated by the existence of bullies, trolls, bigots, racists, misogynists, even terrorists, who seek to weaponize speech and the platforms that host it. Online platforms like Twitter, Reddit, and YouTube have become infamous for allowing problematic communities to fester, ultimately harming their broader user base, their platform, and, most importantly, innocent people. The value of free expression can come into tension with other values like human welfare and respect, and responding well to those tensions can be complicated.

Content moderation practices, or the lack thereof (which is also a choice), can have devastating real world consequences. Facebook has pushed aggressively into international markets, in part by subsidizing internet service so that people can access Facebook. However, Facebook was unprepared for the volatile cultural contexts that it entered. Incidents of ethnic violence, and even genocide in Myanmar, were fueled in part by inflammatory posts and deliberate misinformation spread on Facebook. Although Facebook didn’t create these extreme cultural forces, it empowered them with an unfiltered information pipeline directly into a population that lacked robust online media literacy.

Content Moderation “Solutions” That Haven’t Work

As the examples above show, not taking ethics and values sufficiently into account during the creation of these platforms was a serious mistake. It was also a design failure. Treating the problems like bugs and attempting to patch them after-the-fact does nothing to mitigate the real harms that people and society have already borne, to say nothing of the reputational and financial costs that the platforms have suffered due to them.

Platforms have scrambled to implement new moderation policies and systems that attempt to thread the needle between their espoused values and the messy reality of the internet. Unfortunately, these moderation systems have their own problems. Some online platforms have rolled out machine learning algorithms that attempt to preemptively detect and filter hate speech. These systems sound good in theory: they are highly scalable, and they have a veneer of neutrality because they don’t rely on the judgment of human moderators. However, researchers have shown that these algorithms are often biased against minorities (for example, they mistakenly flag benign speech from African Americans). Ironically, these are the very groups that the systems are ostensibly supposed to be protecting.

Another approach to content moderation leverages the labor of tens of thousands of behind-the-scenes human moderators who review content that is flagged by platform users. However, this task is complicated by constant revisions to the platforms’ community guidelines. Furthermore, content moderators are exposed to a never ending stream of the most vile content on the internet, which has caused them to suffer psychological harms. In other words, the design of this system failed to account for the health and welfare of the human moderators.

If you are interested in the human workforce that labors behind-the-scenes of online platforms, we recommend the book “Ghost Work” by Mary Gray.

Doing Content Moderation Right

There is no getting around the fact that content moderation is a hugely challenging task that is fraught with competing values. That said, the solution is not to avoid the problem by falsely claiming “our platform is neutral”. Instead, the solution is to earnestly take on the ethical challenges that the task presents from the very inception of a platform’s design through its implementation within complex and often diverse social and political systems. Some considerations and tensions that aspiring platform engineers must consider with respect to inciteful speech (to take just one example of challenging content):

  • How to define hate speech and other forms of violent speech? This should be done in concert with representatives from impacted groups, so that their values can be clearly articulated and incorporated. Furthermore, this process should be sensitive to historical power imbalances between groups that underlie many forms of intolerance.
  • How to take context into account? This includes dealing with challenging content, like historical quotes and artistic expression, that may seem to violate content guidelines but should not be flagged. In extreme cases, where content cannot be effectively moderated due to a lack of cultural context (for example, Facebook in Myanmar), the platform owner must seriously consider whether it is ethical to open the platform in the impacted areas.
  • What process to use for flagging content? Crowdsourcing, paid human moderators, and machine learning all have their place, if they can be made complementary. However, these systems also need to be tempered with transparent and accountable processes for contesting decisions and requesting review.
  • How to enforce moderation decisions? In addition to censoring content and banning users, there are other options such as adding warning labels, putting content behind a warning blockade, or preventing content from being algorithmically promoted (e.g., in search results or news feeds).

All of these issues require grappling with questions connected to human values. It is not possible to design an online content platform well without taking values into account. But this is not unique or special to them. All technology embodies and expresses values. Good technology design always requires taking values into account in the design process. In the next section, we introduce Value Sensitive Design (VSD) as a framework for thinking about how to design technology in a value-informed way.