Large Language Models as Tools for Automated Assessment of Regulatory Strategies in Learning

ABSTRACT: Self-regulated learning (SRL) involves learners’ active regulation of cognition, motivation, affect, and behavior, yet accurately assessing self-regulatory strategies remains a challenge. Traditional self-reports lack contextual sensitivity and often fail to distinguish between knowledge of strategies and their actual use. Qualitative approaches provide more precise and reliable insights but are highly time- and resource-intensive. To address this challenge, researchers increasingly employ large language models (LLMs). This study introduces a novel approach to the automated detection of SRL strategies within open-format qualitative data. A sample of 147 employed adults responded to two workplace learning vignettes with three open questions each. Five LLMs—Claude Opus 4.1, DeepSeek R1, Gemini 2.5 Pro, Gemma-3-27B, and GPT-5 Thinking—and three human raters coded 14 cognitive, metacognitive, motivational, and emotion-regulation strategies. Problem-solving performance was used as a criterion measure to assess the predictive validity of human- and AI-generated strategy codes. LLMs showed excellent intra-rater reliability (ICC = .87–1.00) and moderate human-alignment (ICC = .58–.71). The best-performing models achieved near-human predictive validity at a marginal cost and time investment compared with human coders. Findings demonstrate the feasibility of LLM-based SRL assessment as a cost-effective, theory-driven alternative for educational research.