LLM Simulation Shows Task Structure Shapes AI-Assisted Argumentation

A study published in Frontiers in Psychology demonstrates that the structural design of academic assignments significantly influences how generative artificial intelligence affects the quality of argumentative work. The research, conducted by Liu Shaobo at Central South University in Changsha, China, utilized a large language model simulation to examine the interaction between specific task formats and different modes of AI usage. The primary finding indicates that responsible collaboration with AI yielded the highest overall rubric scores, but this benefit was most pronounced in general position essays rather than in highly structured tasks that already required detailed source explanation or revision.

The study addressed a growing concern in higher education that generative AI weakens the correspondence between the final text of an assignment and the actual argumentative work involved in producing it. The researchers sought to determine how task structure and explicit AI-use instructions shape evidence use, engagement with objections, and the revision process within the context of university ideological and political education.

The task formats included standard position essays, evidence-integration memoranda, and follow-up-and-revision defense tasks. The AI-use scenarios comprised a simulated no-AI condition, a responsible collaboration condition, and a concealed ghostwriting condition.

Two independent raters coded 148 generated texts using the Textual Response Artifact Scoring Index, or TRASI, to assess the quality of the argumentative work. Additionally, an anonymous questionnaire was administered to collect 268 evaluations of 27 selected texts from a broader pool. The questionnaire specifically targeted how participants distinguished between different task functions and the characteristics associated with AI-generated versus human-thought text.

The results revealed that responsible collaboration produced the highest overall rubric score, with a mean of 3.32. In terms of task format, evidence-integration memoranda achieved a mean score of 3.16, and follow-up-and-revision defenses scored 3.10, both of which exceeded the mean score of 2.63 for position essays. The position essays showed the greatest variation across the three AI-use scenarios, suggesting that without inherent structural constraints, the mode of AI use has a more significant impact on the final quality of the argument.

In contrast, the memorandum and defense tasks showed narrower differences across the AI-use scenarios. The researchers attributed this stability to the fact that these task instructions already required source explanation or response and revision. Because the structural requirements forced the writer to document specific argumentative decisions, the additional contribution of responsible AI collaboration was less variable.

Questionnaire participants distinguished the same task functions identified in the rubric scoring. Memoranda were rated highest for source-claim linkage, indicating that the format effectively prompted writers to connect evidence directly to their arguments. Defenses were rated highest for counterargument response and revision explanation, highlighting their effectiveness in eliciting engagement with opposing views. Open-ended responses from participants associated evidence integration, comparison of reasons, and revision with genuine thinking, while formulaic structure, over-polished expression, and generic abstraction were associated with AI-like writing.

The study concludes that task structure and AI-use instructions are functionally complementary in the context of GenAI-mediated assessment. Structured tasks elicit source explanation or challenge and revision directly, meaning that responsible collaboration adds its largest contribution in general position essays where such structures are absent. The findings suggest that educators can design assessments around the argumentative decisions that writers are required to document, thereby making the intellectual work visible even when AI tools are involved in the production process.

While the study provides valuable insights into the interaction between task design and AI use, it is important to note that the findings are based on an LLM simulation and a specific subset of university ideological and political education. The generalizability of these results to other disciplines or types of AI tools may vary.

Subject of Research: The impact of task structure and AI-use instructions on argumentative work in university assessments

Article Title: Making argumentative work visible in GenAI-mediated assessment: an exploratory LLM simulation and questionnaire study in university ideological and political education

Article References: Liu, S., & Liu, W. (2026). Making argumentative work visible in GenAI-mediated assessment: an exploratory LLM simulation and questionnaire study in university ideological and political education. Frontiers in Psychology, 17, Article 1908308. https://doi.org/10.3389/fpsyg.2026.1908308

Image Credits: AI Generated

DOI: 10.3389/fpsyg.2026.1908308

Keywords: Generative AI, Higher Education, Assessment Design, Argumentation, LLM Simulation, Academic Integrity, Educational Psychology, Task Structure, Making, argumentative, work, visible