10.2: Connecting the Dots- Bias, Environment, Human Labour and Datafication

The issue of power and control in AI runs deep. In the first post in this series, I spoke about bias and discrimination. Because of the way AI models are constructed, they are often biased towards a particular “worldview”, or disenfranchise already marginalised communities. Take, for example, the structure of a Large Language Model like GPT. The huge dataset contains billions of pages scraped from the web, but the vast majority of the text is in the English language. That content is further biased by the way the data is “crawled” and absorbed into the models. In the words of Emily Bender, Timnit Gebru, and the other authors of the now-famous “Stochastic Parrots” article:

In all cases, the voices of people most likely to hew to a hegemonic viewpoint are also more likely to be retained. In the case of US and UK English, this means that white supremacist and misogynistic, ageist, etc. views are overrepresented in the training data, not only exceeding their prevalence in the general population but also setting up models trained on these datasets to further amplify biases and harms.

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

Other research has demonstrated that the biases in Artificial Intelligence can particularly discriminate against young, non-white males; that predictive policing algorithms and AI used in the courts can unfairly target black people; and that even attempting to filter or remove bias can inadvertently compound the issue. Companies like OpenAI have been found to use low-paid human labour in countries like Kenya to manually classify and filter toxic and discriminatory data, in yet another example of a powerful, Western company profiting from the labour of poorer communities.

What all this means, is that powerful AI across a range of applications from language models to facial recognition and the systems we use to collect data in education can not only reflect but actually reinforce harmful stereotypes and biases.

Even the infrastructure of these systems entrenches existing societal inequalities. When I wrote about the environmental impact of Artificial Intelligence I focused on the carbon footprint of training models and the extractive mining processes needed to produce and power the hardware AI is built from. But, as Bender, Gebru, and their colleagues also pointed out, the environmental impact of AI particularly affects countries already suffering the effects of the climate crisis:

These models are being developed at a time when unprecedented environmental changes are being witnessed around the world…It is past time for researchers to prioritize energy efficiency and cost to reduce negative environmental impact and inequitable access to resources — both of which disproportionately affect people who are already in marginalized positions.

On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜

Again, I want to stress the interconnectedness of these problems.

AI systems built on large datasets – whether of text, image, population data, or other – can entrench systemic biases. Those systems are built from technologies which contribute to global environmental issues that disproportionately affect poorer countries and already marginalised communities.

And to bring it back to our field – education – the manner in which all of that data is collected and processed, or the datafication of students, compounds these issues further. In a recent blog post Radhika Gorur, Joyeeta Dey, Moira V. Faul and Nelli Piattoeva comment on the dilemma of “decolonising” data in education. Though the article is about EdTech, the discussion applies to the field of AI as the engine which will drive many of the education technologies already present in classrooms across the world.

The authors argue that we urgently need to scrutinise the philosophies and principles underpinning these education technologies and consider how to promote the ethical use of data, especially in the global south. The collection of data on students, and by extension the use of AI in applications offering “personalised learning“, overlooks the diverse cultural, spiritual, and epistemological realities of different communities.

The article raises critical questions about whether international comparative assessments are suitable for all nations, and why we should challenge the ubiquity and apparent inevitability of EdTech.