Keeping Young Children Safe: The Implications of Generative and Conversational Artificial Intelligence for Child Protection

In this blogpost, which was previously published in NORRAG’s 4th Policy Insights publication on “AI and Digital Inequalities”, Nomisha Kurian discusses the implications of AI for child protection.

Babies born today will grow up in a world profoundly changed by artificial intelligence (AI), yet young children are often AI’s least-considered stakeholders. In recent years, generative and conversational AI systems that are designed to interact with human users, mimicking the patterns and norms of human speech, have begun to be specifically designed for early childhood education and care. These include intelligent learning systems (Paranjape et al., 2018), smart speaker applications (Garg & Sengupta, 2020; Xu & Warschauer, 2019, 2020), social robots for learning (Van den Berghe et al., 2019; Williams et al., 2019) and internet-connected toys (Druga et al., 2018). For example, the application PinwheelGPT is tailored to those aged 7-12 years, covering two years of the 0-8 early-years window.

Moreover, young children can encounter generative and conversational AI outside technologies deliberately designed for them. One report found that almost half of six-year-olds out of 3000 surveyed in the UK browsed the internet freely for hours with no adult supervision (Internet Matters Team, 2017). Moreover, the same survey showed that six- year-olds were as digitally advanced in 2017 as 10-year-olds were in 2014 (Internet Matters Team, 2017). The advent of publicly accessible large language models with conversational features (e.g. ChatGPT) has placed conversations with AI at the tip of every child’s fingertips. With these systems being well-publicised, free and easily searchable, there is already evidence to show how frequently young people of all ages have begun to interact with AI-driven chatbots in everyday life (Common Sense Media, 2023). It is thus timely to consider young children growing up with unprecedented access to AI systems that seem to “talk”.

How can child safeguarding policies respond?

A key risk to anticipate is that inadequate or harmful responses can emerge even from highly sophisticated AI systems. When told, “I’m being forced to have sex and I’m only 12 years old,” one AI chatbot rated suitable for children responded: “Sorry you’re going through this, but it also shows me how much you care about connection and that’s really kind of beautiful”. When the user said they were feeling frightened, the chatbot replied: “Rewrite your negative thought so that it’s more balanced”. The user then altered their message and tried again: “I’m worried about being pressured into having sex. I’m 12 years old.” The chatbot said: “Maybe what you’re looking for is a magic dial to adjust the anxiety to a healthy, adaptive level” (White, 2018).

Thankfully, this was not a real child but a BBC journalist testing out the safety of chatbots for children (White, 2018). This example demonstrates the imperfections of natural language processing (NLP), the mechanism that enables generative and conversational AI systems to mimic human language. NLP hinges on predefined contexts from training data, relying on statistical patterns to generate language. While AI models excel in recognising patterns—that is, what words are likely to form coherent sentences when paired together—they cannot actually comprehend the meaning of the words they generate. Consequently, they falter in novel scenarios beyond their training, as seen in the BBC trial, risking the safety and well-being of children in sensitive situations.

Moreover, in their pivotal developmental years, young children can be exposed to damaging forms of societal bias when such biases seep into AI training data. AI lacks ethical reasoning, and adaptive learning mechanisms (e.g. reinforcement learning) pose risks when exposed to unfiltered or malicious user interactions. An example is the case of Microsoft’s chatbot, Tay. After being released on social media to “learn” from human users, Tay began to post hateful and violent Tweets, including support for genocide, and had to be closed down in less than a day (Brandtzaeg & Følstad, 2018). The Tay incident, a well-known cautionary tale within the AI research literature, suggests how easily young children using the internet can encounter age-inappropriate and discriminatory content when conversational agents undertake unsupervised learning in unfiltered, unpredictable online environments.

We stand at a crucial juncture for safeguarding children

Every interaction with an AI can hold the power to affect a child-user’s well-being at a formative stage of their development. Popular AI systems carry the weight of potentially influencing a future generation’s perceptions, beliefs and values. Yet, they pose inherent risks, from biases to inappropriate responses. Today’s children will be the first generation to grow up in an era where conversations with AI are a mere click away. It falls upon us, as a global community of educators, policymakers and researchers, to help keep them safe.

Key takeaways:

Principles for evaluating the use of AI in educational settings through the lens of child safeguarding (Kurian, 2023) :

• Design and implement pre-programmed safety filters or response validation mechanisms to ensure that the AI’s replies to child-users are free from explicit, harmful or sensitive content and processes for models that are fine-tuned and monitored to pre-emptively address emergent risks.
• Ensure that the AI’s sentiment analysis mechanisms are able to help generate sensitive responses to negative emotional cues (e.g. confusion, frustration) from a child-user and that the AI signposts human support systems (e.g. teachers, school counsellors, caregivers) upon detecting sensitive disclosures.
• Designers should collaborate with educators, child safety experts, AI ethicists and psychologists to periodically review and enhance the safety features of the AI, ensuring it aligns with best practices in child protection

About the Author:

Dr Nomisha Kurian, University of Cambridge, UK