A critical reflection on the use of toxicity detection algorithms in proactive content moderation systems

Toxicity detection algorithms, originally designed for reactive content moderation systems, are being deployed into proactive end-user interventions to moderate content. Yet, there has been little critique on the use of these algorithms within this moderation paradigm. We conducted design workshops with four stakeholder groups, asking participants to embed a toxicity detection algorithm into an imagined mobile phone keyboard. This allowed us to critically explore how such algorithms could be used to proactively reduce the sending of toxic content. We found contextual factors such as platform culture and affordances, and scales of abuse, impacting on perceptions of toxicity and effectiveness of the system. We identify different types of end-users across a continuum of intention to send toxic messages, from unaware users, to those that are determined and organised. Finally, we highlight the potential for certain end-user groups to misuse these systems to validate their attacks, to gamify hate, and to manipulate algorithmic models to exacerbate harm.