Source-linked AI summary
MAILS -- Meta AI Literacy Scale: Development and Testing of an AI Literacy Questionnaire Based on Well-Founded Competency Models and Psychological Change- and Meta-Competencies
Astrid Carolus, Martin Koch, Samantha Straka, Marc Erich Latoschik, Carolin Wienrich
TL;DR
As interaction with AI becomes increasingly common, the paper develops an AI-literacy questionnaire grounded in existing literature and supplemented with psychological competencies. Analyses of data from 300 German-speaking adults yielded a factorial instrument distinguishing AI-literacy facets, Create AI, and psychological competencies.
Problem
Increasingly common interaction with AI brings challenges, while meta-competencies are particularly important in work and adult education.
Method
The study analyzed data from 300 German-speaking adults using exploratory and confirmatory factor analyses.
Results
The instrument identifies Use & Apply AI, Know & Understand AI, Detect AI, and AI Ethics under AI Literacy, while Create AI did not load on AI Literacy and correlated with it separately.
Takeaways & Limitations
The paper contributes an AI-literacy measurement instrument grounded in current literature, incorporating important psychological constructs and a valid factorial structure.
Takeaways & Limitations
The questionnaire could not be validated against an already existing validated AI-literacy instrument, so further validation is needed.
Abstract
from arXiv · showhide
The goal of the present paper is to develop and validate a questionnaire to assess AI literacy. In particular, the questionnaire should be deeply grounded in the existing literature on AI literacy, should be modular (i.e., including different facets that can be used independently of each other) to be flexibly applicable in professional life depending on the goals and use cases, and should meet psychological requirements and thus includes further psychological competencies in addition to the typical facets of AIL. We derived 60 items to represent different facets of AI Literacy according to Ng and colleagues conceptualisation of AI literacy and additional 12 items to represent psychological competencies such as problem solving, learning, and emotion regulation in regard to AI. For this purpose, data were collected online from 300 German-speaking adults. The items were tested for factorial structure in confirmatory factor analyses. The result is a measurement instrument that measures AI literacy with the facets Use & apply AI, Understand AI, Detect AI, and AI Ethics and the ability to Create AI as a separate construct, and AI Self-efficacy in learning and problem solving and AI Self-management. This study contributes to the research on AI literacy by providing a measurement instrument relying on profound competency models. In addition, higher-order psychological competencies are included that are particularly important in the context of pervasive change through AI systems.
1 INTRODUCTION
AI literacy is increasingly important as AI enters work and adult education, but existing measurement options are limited, context-bound, or insufficiently validated. The paper therefore develops and factorially validates a cross-contextual scale that also includes psychological meta-competencies.
- AI automation and collaboration are expected across many jobs, making AI literacy relevant to work and adult education.
- AI literacy measurement can support personnel selection, identification of skill and knowledge shortages, and evaluation of improvement interventions.
- Existing AI-literacy measures are few, often context-specific or unvalidated, and do not incorporate psychological meta-competencies.
- The study develops and factorially validates a psychometrically grounded, cross-contextual AI-literacy scale embedded in current literature.
2 THEORETICAL BACKGROUND
The paper situates AI literacy within broader competency-based literacy models and identifies limitations in existing conceptualisations and measures. It responds with a modular instrument grounded in established taxonomies and augmented by AI self-management.
- AI-literacy conceptualisations commonly cover knowing, using, evaluating or creating AI, and ethics, but differ in their competency configurations.
- Most prior conceptualisations omit in-depth AI creation or development, whereas this paper incorporates creation as a distinct component.
- Existing educational instruments often target specific interventions, limiting their suitability for broader use cases and modular assessment.
- Prior instruments frequently fail to differentiate AI-literacy facets or examine factorial structure in large samples.
- AI self-management is framed as important for prolonged and sustainable AI use alongside classical AI-literacy facets.
- The proposed instrument is literature-grounded, modular, psychometrically oriented, and supplemented with psychological competencies for AI self-management.
3 EMPIRICAL STUDY
The study developed self-assessment items from AI-literacy and psychological competency models and tested their factorial structure in German-speaking adults. The modified model showed good fit and separated AI creation from the higher-order AI-literacy factor.
- 3.1 Participants and Measures: Data were collected online from German-speaking adults using German self-assessment measures and additional standardised instruments.
- 3.2 Materials: After item refinement, 68 items remained for analysis, comprising 56 AI-literacy items and all 12 AI-self-management items.
- 3.2 Materials: Participants rated abilities on an 11-point scale from 0 to 10, where higher values indicated more pronounced abilities.
- 3.3 Analysis: The modified confirmatory factor model showed good fit: CFI = .926, TLI = .920, RMSEA = .057, and SRMR = .079.
- 3.3 Analysis: All items and first-level factors loaded significantly, while the higher-order factors were significantly correlated.
- 3.3 Analysis: AI self-efficacy and AI self-competency correlated highly with AI literacy, while Create AI did not load on AI literacy and correlated only with it.
4 DISCUSSION
The study developed a modular AI-literacy questionnaire grounded in existing competency models and expanded with psychological competencies. Factor analyses supported distinct AI-literacy facets, a separate Create AI skill, and differentiated psychological domains, while highlighting validation needs.
- Study aims and approach: The questionnaire was designed to be literature-grounded, modular for professional use, and inclusive of psychological competencies beyond classical AI-literacy facets.The study collected data from 300 German-speaking adults and analysed the items using exploratory and confirmatory factor analyses.
- AI-literacy structure: The final AI Literacy factor comprised Use & Apply AI, Know & understand AI, Detect AI, and AI Ethics.The expected AI-literacy factors loaded on the superordinate AI Literacy factor, while the proposed Evaluate & Create AI factor was not retained.
- AI-literacy structure: Create AI formed a separate, correlated skill rather than an inherent part of AI Literacy, supporting modular measurement alongside AI Literacy.Create AI correlated with AI Literacy at r = .5 but did not load on its superordinate factor.
- AI-literacy structure: Evaluation items loaded with Know & understand AI, suggesting that evaluating AI was more closely associated with knowledge and understanding than with developing AI.The authors interpret precise knowledge and understanding as more important for evaluating AI systems than the ability to develop them, based on participants’ self-assessments.
- AI-literacy structure: Detect AI emerged as an independent competence, indicating that recognizing AI may not depend on knowing and understanding AI.The factor was described as similar to awareness, such as distinguishing smart from non-smart devices.
- Psychological competencies: Psychological competencies separated into AI Self-efficacy for problem solving and learning and AI Self-competency for persuasion literacy and emotion regulation.The two domains were clearly correlated but retained sufficient uniqueness to be treated as distinct constructs; all major constructs also showed high correlations while remaining differentiated.
- Associations and implications: Questionnaire dimensions generally correlated positively with positive AI attitudes and willingness to use technology, although Create AI showed comparatively low relations.AI Literacy and AI Self-competency also negatively correlated with negative AI attitudes, whereas Create AI and AI Self-efficacy did not.
- Implications and limitations: The instrument offers researchers, practitioners, and educators a factorially structured measure, but independent-sample testing and validation against other measures remain necessary.The sample was limited to German-speaking individuals in Germany or Austria, and the model changes made during confirmatory analysis mean the approach should be considered exploratory.
A A STRUCTURE OF THE QUESTIONNAIRE
The questionnaire combines AI-literacy facets with psychological competencies related to using, creating, learning about, and managing AI. Its items cover practical application, understanding, creation, problem solving, learning, persuasion, and emotion regulation.
- The questionnaire measures using and understanding AI through items about operating applications and applying them meaningfully in everyday life.
- Create AI is assessed with items on designing, programming, developing, and selecting tools for new AI applications.
- AI problem-solving and learning competencies capture handling difficult AI-related tasks and keeping up with rapid technological change.
- AI self-competency also includes resisting or recognizing AI influence on everyday decisions and regulating frustration, anxiety, and euphoria during AI interactions.