Critical AI Education in Schools: Gender, Bias and Agency

Artificial intelligence is already part of school life. It appears in search engines, learning apps, translation tools, recommendation systems and chatbots. Young people may use these systems before they understand how they work or how they influence what they see.

This makes AI education urgent. But teaching students how to write a good prompt is not enough. They also need to ask harder questions: Who built this system? What data does it use? Whose experiences are missing? Who benefits from its decisions—and who may be treated unfairly?

This is what critical AI education in schools is about. It connects basic technical understanding with gender, fairness, participation and human rights. It helps young people move from simply using technology to questioning and shaping it.

AI is not neutral

An AI output can look objective because it comes from a machine. But every AI system is shaped by human choices.

People decide which problem the technology should solve. They select or collect data, create labels, define success and choose how a system will be tested. Schools, companies and public institutions then decide where and how it will be used. Bias can enter at every one of these steps.

This means that bias is not only a technical error. It can also reflect unequal structures in society. Historical data may contain past discrimination. A design team may overlook people whose lives differ from their own. A system may work well during a test but fail for groups that were not represented. Even a technically accurate tool can be unfair when it is used in the wrong context or given too much power.

Young people need language for these connections. Otherwise, they may learn to trust an automated answer simply because it is fast, confident or personalised.

What gender has to do with AI

Gender bias can appear in AI in several ways.

An image generator may show mostly men when asked for engineers or technology leaders. A language model may describe women through family roles more often than professional roles. A recruitment tool trained on past hiring decisions may reproduce old patterns. Voice or image systems may perform differently for people who were underrepresented in the data used to build them.

UNESCO has documented regressive gender stereotypes in large language models. Its research also shows that gender bias can overlap with racism and homophobia. This overlap matters. People do not experience gender on its own: skin colour, disability, class, age, language and other parts of identity can change how a digital system sees—or fails to see—them.

This is sometimes called intersectionality. In simple terms, it means that different forms of inequality can connect and intensify each other.

Gender and AI is therefore not a small specialist topic. It is a way to understand a larger question: whose knowledge, bodies and futures are treated as normal when technology is designed?

Why schools are a key place to begin

Schools are one of the few places where many young people can explore AI together before unequal access and confidence become even harder to change.

Some students already experiment with AI at home. Others have limited access, do not know which tools are safe or do not see themselves as “technical.” If AI education only rewards the students who are already confident, it can widen existing gaps.

Critical AI education creates a shared starting point. It gives all learners time to understand what AI can and cannot do. It also makes space for experiences that are often missing from technology debates.

UNESCO’s AI competency framework for students describes young people not only as users, but as responsible co-creators. It combines four areas: a human-centred mindset, AI ethics, basic knowledge about AI, and AI system design. This is important because young people need knowledge, values and creative skills together.

UNICEF’s guidance on AI and children makes the rights perspective equally clear. Child-centred AI should support safety, privacy, non-discrimination, transparency, inclusion and children’s development. Children should also be prepared for a world shaped by AI.

What critical AI education should teach

Critical AI education is not about telling students that all technology is bad. It is about giving them the confidence to examine technology carefully and make informed choices.

A strong learning format should help students:

• Recognise AI in everyday life. Where do apps sort, recommend, predict or generate something?

• Understand the basic logic. AI identifies patterns in data; it does not think or understand like a person.

• Question outputs. An answer can sound convincing and still be wrong, incomplete or biased.

• Investigate fairness. Who is represented in the data? For whom was the system tested? Who may be overlooked?

• Protect privacy. What information should never be entered into an AI tool? Where might that data go?

• Keep humans responsible. Which decisions should never be left to a machine alone?

• Imagine alternatives. How could a tool be redesigned to include more people and support their needs?

These skills belong across the curriculum. Bias in an image generator can be explored in art or media studies. Automated decisions fit into citizenship education. Data, probability and model errors connect to mathematics. Language models invite questions about sources, writing and representation.

Four classroom examples that make bias visible

Complex ideas become easier when students can connect them to everyday situations.

1. The learning app that mistakes speed for ability

A learning app assumes that fast answers mean strong understanding. A student who reads more slowly receives easier tasks and fewer chances to progress.

The class can ask: What does the app measure? What does it ignore? Does speed always show ability? What would a fairer system need to know?

This example shows that a system can produce unequal results even without using gender or other identity data directly. The definition of “success” may already be too narrow.

2. The image generator that repeats stereotypes

Students ask an image tool to show people working in technology, care, leadership or science. They compare who appears, what they look like and which roles are missing.

The aim is not to prove that one prompt explains the whole system. It is to notice patterns, test different wording and discuss why representation matters. Students can then create a better set of images or write rules for a more inclusive tool.

3. The camera that does not recognise every face equally

A photo app repeatedly fails to recognise one student’s face. The class explores possible reasons: Was the system trained and tested with a wide range of skin tones, faces, lighting conditions and assistive devices?

This opens a wider question: Who is treated as the “standard user” during design?

4. The classroom robot that only sees the front row

A robot selects students to answer questions, but its camera mainly captures the front row. The same children are chosen again and again.

No one planned to exclude the other students. Yet the technical limitation creates an unfair outcome. This helps learners see an important point: fairness is about effects, not only intentions.

From correct answers to better questions

Critical learning works best when students are allowed to explore uncertainty. They should hear at the beginning that there are no “wrong” observations, no one must share a private story and changing your mind is part of learning.

Useful activities start with the digital world students already know:

• Which apps, games, search tools or chatbots do you use?

• What does the system recommend, hide, rank or decide?

• What can a person do that a machine cannot?

• When can a machine make a mistake?

• Should this decision be automated at all?

The discussion should not end with identifying a problem. Students can design a fair app, machine or classroom tool and define its rules. Who should it help? Who might it forget? What should it never do? Who should be involved when it is designed or changed?

This shift from critique to co-creation is central. It turns concern into agency.

Digital agency means having a real say

Digital agency is the ability to understand digital systems, make informed choices and take part in shaping them. It is different from being a confident consumer.

A student with digital agency can say, “I know why this recommendation may appear, I can question it, and I can imagine another design.” They also know when to ask for human help or reject an automated decision.

The OECD recommends involving teachers, students and other users as co-designers of educational technology. This is more than a good classroom method. People who live with a system often notice needs and harms that are invisible to a distant design team.

Participation must also be meaningful. Asking students for ideas after all major decisions have been made is not co-creation. Young people need clear information, accessible activities and a real chance to influence the outcome.

What teachers need

Teachers do not need to become software engineers before they can teach critical AI literacy. They need reliable concepts, age-appropriate examples, time to test methods and permission to say, “We do not know yet—let us investigate.”

They also need support from schools and education systems. OECD guidance stresses that teachers should be able to assess AI outputs critically, use technology creatively and retain professional agency. High-stakes decisions need transparency, human support and ways to challenge errors.

For educators, five practical principles can guide the work:

1. Start with lived experience. Use tools and decisions students already encounter.

2. Make power visible. Ask who defines the problem, rules and measure of success.

3. Include gender and diversity throughout. Do not isolate them in one final lesson.

4. Test claims together. Compare outputs, look for patterns and discuss limits.

5. End with participation. Let students redesign, set rules and explain what fairer technology would require.

CTRL+INCLUDE: exploring critical AI education with younger learners

AI EMPOWER has long worked on critical, inclusive AI education. With CTRL+INCLUDE, we are adapting this approach for younger people and learning with schools.

Together with nowa, AIT, Independo and five partner schools, we are developing practical formats that connect AI, algorithms, gender, inclusion and design. Students will explore questions such as “How inclusive is AI?” and “How does an algorithm decide?” They will also move from discussion to making: developing and testing their own ideas for fairer digital tools.

The goal is not to prepare children to fit into a technological future decided by others. It is to help them understand that technology is made by people—and that they have knowledge, experiences and ideas that belong in its design.

Explore the CTRL+INCLUDE project — https://www.ai-empower.org/include

Further reading

• UNESCO: AI competency framework for students — https://www.unesco.org/en/articles/ai-competency-framework-students

• UNESCO: Challenging systematic prejudices—gender bias in large language models — https://unesdoc.unesco.org/ark:/48223/pf0000388971

• UNESCO: Guidance for generative AI in education and research — https://www.unesco.org/en/articles/guidance-generative-ai-education-and-research

• UNICEF: Guidance on AI and children — https://www.unicef.org/innocenti/reports/policy-guidance-ai-children

• OECD: Opportunities and guardrails for equitable AI in education — https://www.oecd.org/en/publications/oecd-digital-education-outlook-2023_c74f03de-en/full-report/opportunities-guidelines-and-guardrails-for-effective-and-equitable-use-of-ai-in-education_2f0862dc.html