ChatGPT-Scaffolded Chinese Vocabulary Lessons Show Promise in Small Classroom Study

Generative artificial intelligence has been hailed as a revolution in language education, but rigorous evidence about what it actually teaches—and what it does not—remains scarce. A new exploratory study published in SN Social Sciences offers one of the first detailed looks at how ChatGPT functions as an instructional scaffold in Chinese as a second language, and its results are as intriguing as they are cautiously framed. The research, led by Qingli Lei of the University of Illinois Chicago together with colleagues at Guangdong University of Foreign Studies, Jimei University, and the University of Illinois Chicago, tracked vocabulary learning in two intact undergraduate classes over six weeks and found a striking divergence: the class that used ChatGPT appeared to pull ahead dramatically on meaning- and usage-related vocabulary knowledge, while pronunciation gains looked virtually identical across both groups.

The study’s premise rests on a well-established foundation in second language research. Vocabulary knowledge is multidimensional, encompassing word form, meaning, and use. In Mandarin Chinese, this means learners must simultaneously master pinyin romanization with accurate tone marks, semantic relationships including collocations and cultural connotations, and the syntactic patterns governing how words behave in sentences. Providing individualized, adaptive support across all these dimensions at once is a persistent challenge for classroom teachers, who rarely have the capacity to give every student immediate, personalized feedback. The researchers argued that ChatGPT’s conversational architecture—its ability to answer individualized questions, generate contextualized examples, and respond instantly—might fill precisely this gap.

The theoretical scaffolding for the intervention drew on several complementary frameworks. Scaffolding theory, rooted in Vygotsky’s work and Wood, Bruner, and Ross’s tutoring studies, describes contingent support that fades as learners gain autonomy. Long’s Interaction Hypothesis emphasizes that negotiating meaning through dialogue drives acquisition, while Swain’s Output Hypothesis holds that producing language forces learners to notice gaps and refine their knowledge. Craik and Lockhart’s Depth of Processing framework and Laufer and Hulstijn’s Involvement Load Hypothesis add that elaborately and cognitively processed material is retained more durably. ChatGPT-scaffolded instruction, the team reasoned, could activate all of these mechanisms at once: learners ask questions, negotiate meanings, generate sentences, and evaluate usage in an iterative loop.

Sixteen international undergraduates—seven Thai, eight Indonesian, and one Vietnamese student, aged 20 to 24 and proficient at HSK Levels 4 through 6—took part. They were drawn from two pre-existing Chinese language classes at a university in southeastern China, with eight students in each. One class received traditional teacher-directed vocabulary instruction, including explanation, guided reading, pronunciation correction, repetition, and dictation. The other used ChatGPT 3.5 under teacher guidance as a scaffold throughout each lesson, asking questions, requesting explanations, generating examples, composing phrases and sentences, and exploring contextual usage. Both groups were taught by the same experienced instructor, received identical instructional time of 45 minutes per lesson across 13 sessions, and studied the same 110 previously untaught target words from Lessons 2 through 7 of Boya Chinese Intermediate 1. Students typed Chinese characters using voice-to-text input on their mobile devices, since ChatGPT 3.5 itself offered no speech recognition or spoken output.

Vocabulary knowledge was assessed with researcher-developed pretests and posttests covering all 110 instructed items. Phonetic knowledge was measured through pinyin transcription with tonal accuracy, while a composite semantic-syntactic score averaged true/false meaning judgments against plausible distractor glosses with sentence-completion items requiring grammatical, meaningful word use. Two experienced instructors independently scored all assessments, achieving strong inter-rater agreement with intraclass correlation coefficients of 0.94 for phonetic and 0.91 for composite scores.

The quantitative pattern was suggestive. The ChatGPT-scaffolded class showed an observed mean gain of 80.38 points on the 110-point composite semantic-syntactic measure, compared with 58.13 points in the traditional class—a difference of 22.25 points, with an exact permutation test yielding p = .024. On phonetic knowledge, however, the classes were statistically indistinguishable, gaining 46.38 and 44.88 points respectively. An exact permutation test for phonetic gain returned p = .882. In interviews, six volunteers from the ChatGPT class described increased engagement, comprehensive explanations, rapid responses, and contextualized examples; one student noted feeling more comfortable asking ChatGPT questions without fear of embarrassment. Several volunteers, consistent with the quantitative pattern, said the tool was more helpful for meanings and usage than for pronunciation, and they flagged occasional inaccuracies and the inconvenience of VPN access.

Yet the researchers are unflinching about what these numbers cannot show, and that honesty is arguably the study’s most valuable contribution. Because instructional condition was completely confounded with class membership—one class per condition—no statistical model can separate a treatment effect from a class effect. The class-level indicator and the treatment indicator are perfectly collinear, leaving zero residual degrees of freedom for any significance test of the intervention itself. The students were also not randomly assigned, and the ChatGPT class began ahead on both measures at pretest, including a practically meaningful 13.25-point phonetic advantage. Student-level p values answer only how unusual the observed difference would be under random reallocation of these particular 16 students; they carry no information about whether the instructional approach produced it. Pre-existing differences in composition, prior instruction, peer dynamics, and motivation all remain competing explanations.

Measurement constraints add further caution. The instrument’s internal consistency and dimensional structure were never empirically established, since item-level responses were not retained after consensus scoring, and the semantic-syntactic composite cannot support separate conclusions about semantic versus syntactic development. The ChatGPT class’s posttest mean of 102.75 out of 110—with six of eight students scoring at least 105—signals a ceiling effect that destabilizes standardized effect sizes. Identical items at pretest and posttest may have produced practice effects, particularly for the guessable true/false semantic items, and the qualitative sample comprised only six self-selected volunteers who may have been positively predisposed toward the approach. The results therefore describe immediate performance on 110 instructed items, not retention, transfer, or broader lexical competence.

What the study does offer is a bounded but genuinely useful signal and a set of testable hypotheses. The convergence between the quantitative pattern—larger class-level change on meaning and usage, flat differences in pronunciation—and the interviewees’ independent perception that ChatGPT helped them understand word meanings better than pronunciation is theoretically coherent: text-based ChatGPT 3.5 provided no auditory modeling, so phonetic development plausibly depended on the teacher-led practice both classes received. The authors’ pedagogical implications are deliberately provisional: educators who adopt generative AI should match AI activities to the intended learning task, verify AI output, retain teacher oversight, and teach AI literacy so students can critically evaluate generated explanations. Future research, they argue, needs randomized controlled trials with larger samples, validated instruments with adequate posttest headroom, voice-enabled AI systems capable of real-time pronunciation feedback, systematic qualitative sampling across both conditions, and follow-up measures of long-term retention. In a field saturated with enthusiasm and thin on evidence, this small study models something rarer than a positive result: a template for how to test the AI-education hype honestly.

Subject of Research: An exploratory mixed-methods evaluation of ChatGPT-scaffolded instruction on second language Chinese vocabulary learning in two undergraduate classes.

Article Title: Evaluating an AI-scaffolded intervention for L2 vocabulary learning: affordances, constraints, and pedagogical implications

Article References: Lei, Q., Chen, Y., Chen, Y., Zhang, X., & Park, J. (2026). Evaluating an AI-scaffolded intervention for L2 vocabulary learning: affordances, constraints, and pedagogical implications. SN Social Sciences, 6(10), Article 461. https://doi.org/10.1007/s43545-026-01725-w

Image Credits: AI Generated

DOI: 10.1007/s43545-026-01725-w

Keywords: ChatGPT, second language learning, vocabulary acquisition, Chinese language, instructional scaffolding, generative AI, language education, pinyin tones, quasi-experimental design, semantic-syntactic knowledge, L2 pronunciation, educational technology