The effect of AI and algorithmic bias on you as a student

A robotic finger is about to push down on one side of a set of scales — Image generated by Adobe Firefly

AI and algorithmic bias can affect you as a student. Numerous universities and schools have used plagiarism detection tools for years, and the recent development of AI detection tools has many news outlets abuzz.

But AI and algorithmic bias can affect you in much more insidious and unseen ways. AI and algorithmic bias might be affecting:

What score you get on an essay (Feathers, 2019)
False accusations of academic misconduct from remote AI proctoring (Harwell, 2020)
Your ability to get a job (Chang 2023)

You may even be training AI without your knowledge--or any real ability to opt out ("Turnitin," 2024)!

As AI becomes more integrated into daily life, its impact will continue to grow.

Algorithmic bias

The repeatable and systematic outcomes or errors produced by a computer system or program that disadvantage certain groups or categories. Algorithmic bias can arise from faulty data sets, program design, human biases, or a combination of any of these.

Since 2018, researchers have shown that AI facial detection algorithms discriminate on the basis of race and gender (Boulamwini & Gebru). More recent studies indicate this is still a problem (Dooley et.al, 2022). Worse, some AI companies are actively trying to suppress research on this topic. The CEO of Proctorio, a company that provides AI-powered proctoring for tests, tried to get a peer-reviewed article retracted (Chin, 2020) to cover up the potential that these programs may "discriminate against students based on their bodies and behaviors" (Swauger, 2020, para. 1).

What's the cause of AI and algorithmic bias, and what can be done about it?

Buolamwini, J., & Gebru, T. (2018). Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, 77–91. https://proceedings.mlr.press/v81/buolamwini18a.html

Chang, X. (2023). Gender Bias in Hiring: An Analysis of the Impact of Amazon’s Recruiting Algorithm. Advances in Economics, Management and Political Sciences, 23, 134–140. https://doi.org/10.54254/2754-1169/23/20230367

Chin, M. (2020, October 22). An ed-tech specialist spoke out about remote testing software—And now he’s being sued. The Verge. https://www.theverge.com/2020/10/22/21526792/proctorio-online-test-proctoring-lawsuit-universities-students-coronavirus

Dooley, S., Wei, G. Z., Goldstein, T., & Dickerson, J. (2022). Robustness Disparities in Face Detection. Advances in Neural Information Processing Systems, 35, 38245–38259. https://proceedings.neurips.cc/paper_files/paper/2022/hash/f9faef4e1b4dbbd48ef60056ffe14c90-Abstract-Datasets_and_Benchmarks.html

Feathers, T. (2019, August 20). Flawed Algorithms Are Grading Millions of Students’ Essays. VICE. https://www.vice.com/en/article/flawed-algorithms-are-grading-millions-of-students-essays/

Harwell, D. (2020, April 1). Mass school closures in the wake of the coronavirus are driving a new wave of student surveillance. Washington Post. https://go.gale.com/ps/i.do?p=STND&u=iastu_main&id=GALE|A619199748&v=2.1&it=r&sid=bookmark-STND&asid=a83e8fe0

Swauger, S. (2020, April 2). Our bodies encoded: Algorithmic test proctoring in higher education. Hybrid Pedagogy. https://hybridpedagogy.org/our-bodies-encoded-algorithmic-test-proctoring-in-higher-education/

Turnitin. (2024, July 23). In Wikipedia. https://en.wikipedia.org/w/index.php?title=Turnitin&oldid=1236120068#cite_note-34

AI and algorithmic bias is a product of multiple biases

"Bias in AI systems is often seen as a technical problem, but the NIST report acknowledges that a great deal of AI bias stems from human biases and systemic, institutionalized biases as well" (NIST, 2022, image caption). (Image credit: N. Hanacek. https://www.nist.gov/news-events/news/2022/03/theres-more-ai-bias-biased-data-nist-report-highlights)

People may think that data is impartial and therefore, algorithms or AI must be unbiased. Unfortunately, the opposite is true — algorithms and AI amplify our biases (Schwartz et al., 2022).

This is a complex and nuanced problem, but overall, this problem arises because algorithms and AI don't exist in a vacuum — they exist and are used within societies and specific contexts.

These programs are created by humans operating under different cultures, rules, and norms. Algorithms replicate human tasks, using datasets and rules based on human actions.

Some of these actions were based on bias, and now these biases are embedded in the core programming. New iterations of the program are built on top of this core programming, like an iceberg.

The National Institute of Standards and Technology (NIST) created this iceberg image based on a report (Schwartz et al., 2022) to illustrate how pervasive all of these biases can be within algorithms and how hard it is to remove bias from them.

The statistical and computational biases that we notice are supported by other, hidden biases. Even when humans work to not add more bias to the system, the effects of past bias are still in the dataset, pushing AI bias into view.

Schwartz, R., Vassilev, A., Greene, K., Perine, L., Burt, A., & Hall, P. (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (NIST Special Publication 1270). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.1270

The amplifying effect of AI and algorithmic bias online

So how does all of this affect you as a researcher?

Cognitive biases already make evaluating information sources a challenge, and algorithmic bias can make it even harder. Algorithmic bias can make it harder for you as a researcher to break free of this bias. One of the most insidious problems with algorithmic bias is that you might never know what you're not seeing. For example, a study by Datta et al. (2015) found that Google showed women fewer ads for high-paying jobs.

It also turns out that the bias built into all algorithms offering "personalization" can make the tendency to seek out echo chambers even worse. One study (Shin & Jitkajornwanich, 2024) found TikTok's algorithm pushed people past echo chambers and into radicalization. Shin & Jitkajornwanich (2024) found the increased extremism and polarization were a result of the algorithm's ability to "slot users into specific categories and reinforce their ideas" (p.1020).In other words, amplifying bias all the way into extremism.

Datta, A., Tschantz, M. C., & Datta, A. (2015). Automated Experiments on Ad Privacy Settings: A Tale of Opacity, Choice, and Discrimination. Proceedings on Privacy Enhancing Technologies, 2015(1), 92–112. https://doi.org/10.1515/popets-2015-0007

Shin, D., & Jitkajornwanich, K. (2024). How Algorithms Promote Self-Radicalization: Audit of TikTok’s Algorithm Using a Reverse Engineering Method. Social Science Computer Review, 42(4). https://doi.org/10.1177/08944393231225547

What does this mean for your research?

So what can you do about all this?

Knowing is half the battle! Knowing about the effect cognitive — and other — biases have on your research can help you mitigate them to some degree. Additionally, you can:

Use a search engine that is less likely to use your browsing history to inform what it retrieves for you, such as DuckDuckGo
Opt out of personalization whenever you can
Clear your search history and cookies periodically
Scan through User Agreements so you know if you can opt out of tracking and how to do so
Use SIFT to get a full picture of the academic landscape

Key takeaways

The complex and nuanced problems of AI and algorithmic biases are are due to more than just faulty datasets because human biases are integrated deeply into their core programming.

These biases can affect multiple aspects of your life — including your research. As a researcher, be critically aware of this bias and use SIFT before fully trusting algorithmic outputs.