Inclusive Design of AI's Explanations: Just for Those Previously Left Out, or for Everyone?
Hamid, M. M., Moussaoui, F., Noa Guevara, J., Anderson, A., Agarwal, P., Dodge, J., & Burnett, M.
ACM Transactions on Interactive Intelligent Systems, 16(1), Article 8 · 2026
Explainable AI (XAI) systems aim to improve users' understanding of AI, but rarely consider inclusivity — so "improving" explanations might not work well for everyone. In a 69-participant controlled experiment, we compared an XAI prototype before and after an inclusivity-driven redesign. Participants who used the redesigned version engaged ~23% more with the AI's explanations and formed significantly better mental models of the AI — with benefits extending to everyone, a "curb-cut" effect.
The problem
In XAI, a mental model is the user's understanding of how an AI system works — and helping users form suitable mental models is the aim of XAI systems. As researchers work to make XAI better, inclusivity asks the question: "better for whom?" Improving explanations for one specific user group may not benefit others unless AI development practices adopt inclusive approaches.
One such approach is recognizing users' diverse problem-solving styles — how people process information, their computer self-efficacy, their attitude toward risk, how they learn technology, and what motivates them to use it. While demographic diversity matters, considering problem-solving diversity generates actionable design directions. This paper investigates whether focusing on users' diverse problem-solving styles can reduce inclusivity issues across an information-rich XAI system, ultimately improving users' mental models.
What we did
A team of AI practitioners ("Team J") used the GenderMag inclusive design method to find and fix inclusivity issues in their XAI prototype. The prototype shows a modified Tic-Tac-Toe game (an "MNK" game on a 9×4 board, 4-in-a-row to win) played by two AI agents that use a convolutional neural network to score every possible move. Three visual explanations reveal the blue agent's decisions: Scores Best-to-Worst (BTW), Scores Through Time (STT), and Scores On-the-Board (OTB).
Team J's GenderMag-driven cognitive walkthroughs produced 15 fixes, of which we implemented 13 — from tooltips with exact win/loss/draw percentages, an interactive legend, and clearer labels and titles, to a game log, game history, and a "Top 5 moves" comparison. Notably, 9 of the 15 fixes targeted the explanations themselves.
We then ran a between-subject controlled experiment with 69 participants: 34 used the "Original" version and 35 used the "Post-GenderMag" (inclusivity-fixed) version. We measured participants' engagement with the explanations, their ability to detect and specify the AI's flaws, their perception of the AI agents' behavior, and their mental model scores — computed with a weighted rubric built in collaboration with the AI agents' creator.
Key results
- Explanations mattered: participants' explanation usage significantly predicted their mental model scores (linear regression, R² = 0.202, F(1,67) = 16.96, p = 0.0001) — each additional instance of drawing on the explanations was associated with a ~2.9-point higher mental model score.
- Post-GenderMag participants were on average ~23% more engaged with the explanations than Original participants (mean 18.8 vs. 15.26 mentions; t(67) = 1.93, p = .0575).
- Post-GenderMag participants formed significantly better mental models than Original participants (mean score 129.06 vs. 109.12; one-tailed t(67) = 1.68, p = .049).
- The groups differed by nearly double or more on three understanding measures: 71% of Post-GenderMag participants (vs. 38%) correctly noticed the agent picks high-scoring moves, 69% (vs. 32%) correctly observed it ignores its opponent's moves, and only 14% (vs. 38%) incorrectly believed the agent was random or illogical.
- Two fixes were directly implicated in these differences: "Exact Values" (hovering reveals each move's exact win/loss/draw percentages) and "Top 5 Moves Comparison" — both added for users with comprehensive information-processing styles and low motivation to tinker with unfamiliar technology.
From the paper
