Social Science Method Reveals Flaws in How We Measure Code Understanding

For more than four decades, computer scientists have published study after study on how programmers understand code, building an entire subfield of software engineering research on the question of program comprehension. Yet according to a team of researchers at the New Jersey Institute of Technology, that edifice may rest on surprisingly weak ground. In a paper titled On the Reliability of Code Comprehension Proxies, Erfan Arvan, a fourth-year doctoral student in computer science at NJIT, together with assistant professor Martin Kellogg and collaborators at William & Mary, systematically examined the measurement techniques that researchers routinely use as stand-ins for genuine understanding. Their conclusion was blunt: many of the field’s standard proxies for comprehension are unreliable, and the discipline has operated without explicit standard definitions or dependable ways of measuring the very thing it claims to study. The work earned a Distinguished Paper Award at the IEEE/ACM Automated Software Engineering conference, held in Munich this fall, one of the premier venues where researchers present advances in tools and techniques for building software.

The starting point for the investigation was an extensive literature review that Arvan described as eye-opening. After more than forty years of publishing papers on program comprehension, the field, he said, has apparently been built on shaky foundations, lacking explicit standard definitions and reliable measurement methods. That gap matters because program comprehension is not an abstract curiosity. Software engineers and programmers spend a considerable amount of their working time simply trying to understand code, whether written by colleagues years earlier or generated moments ago by an artificial intelligence system. If researchers cannot measure comprehension accurately, they cannot reliably evaluate new programming languages, teaching methods, documentation tools, or development environments designed to make code easier to digest. Every downstream conclusion inherits the weaknesses of the measurement instruments used to produce it.

The NJIT-led team imported a technique from social science to bring order to this uncertainty: the Delphi method. As the researchers noted, the Delphi method is widely used for building expert consensus under conditions of uncertainty in other domains such as medicine and national-security forecasting, but to their knowledge this is its first application to code comprehension research. The procedure is iterative and structured. Over multiple rounds, participants ranked Java code snippets by how difficult they were to comprehend, and the group iteratively resolved disagreements through written feedback and discussion. By converging on a shared expert judgment about which snippets were genuinely hard or easy to understand, the method produced a calibrated reference point, a ground truth of sorts, against which the field’s common shortcut measurements could be tested.

With that consensus baseline in hand, the researchers presented dozens of students with several evaluation methods commonly used in the literature. These included assessments of syntax, timed input/output questions in which participants had to determine what a program would print, and self-evaluations in which participants rated their own understanding of the code. The design allowed the team to compare each proxy against the expert-derived difficulty ranking and ask a deceptively simple question: does this measurement actually track whether a person understands the code? The results separated the proxies sharply. Some widely used instruments proved to be poor indicators of real comprehension, while others performed considerably better, offering the field its first systematic evidence about which shortcuts can be trusted.

The clearest finding concerned what actually signals understanding. Simply asking someone to explain what output a piece of code would produce turned out to be the best indicator of whether they truly understand it, according to the study’s analysis of how experienced developers work in the real world. Measuring how long someone takes to determine that output is the next-best approach, Arvan said. The emphasis here is on the bottom line of a code snippet, what it does, rather than on its surface form. Arvan drew an analogy from linguistics: in everyday life, non-native speakers are judged by how well they get their message across, not by their grammar or spelling. In the same way, comprehension of code should be judged by whether a reader can correctly state the program’s behavior, not by whether they can parse its syntactic details.

That distinction between behavior and syntax carries practical weight for anyone who writes or reviews software. A reader who can recite the rules of a language but cannot say what a loop or a conditional will actually produce has not understood the program in any sense that matters to engineering. Conversely, a developer who can predict the output, and predict it quickly, demonstrates the kind of working mastery that real-world maintenance and debugging demand. By anchoring measurement to output prediction and response time, the NJIT team offers researchers a more defensible instrument than self-reports or syntax checks, both of which can diverge substantially from genuine understanding. The timed variant adds a graded dimension, capturing not just whether comprehension occurred but how effortful it was, which matters when comparing alternative code designs for readability.

The motivation is not purely academic. Arvan pointed to the rise of large language models, which are generating a great deal of code but whose output remains uncertain in many respects, including whether it is reliable, secure, understandable, and verifiable. The team’s idea, he explained, was to see whether they could propose and create a tool or a system that can automatically detect whether a given piece of code is understandable to humans. Such a capability would address a well-documented pain point: because engineers already spend a considerable share of their time trying to understand code, any automated gauge of human readability could steer both human and machine-generated programs toward forms that are easier to comprehend from the start. The researchers’ ultimate goal is to teach programmers to write code that is more easily understood from the beginning, rather than repaired after the fact.

The paper also casts a critical eye on tools that developers use every day. Kellogg observed that many software development environments apply a code quality measurement based on cyclomatic complexity, a concept he described as discredited, which assigns a score based on how many decisions a program can make. This, he argued, is indicative of a wider problem: code is too often evaluated based merely on what automation tools and development environments can prove, instead of on what matters to real-world engineers. The new paper did not examine that particular issue directly, but Arvan and Kellogg hope their findings will motivate the people who build software development tools to modernize such approaches, replacing inherited metrics with measures that have been validated against actual human comprehension.

The team’s agenda for the coming years is ambitious. Arvan said a future direction of their plans is to revisit all of the prior studies that use one of the measurements they tested, and to show those results to the community. The aim is to review the existing literature in light of the new findings so that researchers can see which results can be trusted, which cannot, and which papers should be redone and reconducted so that their conclusions become reliable. In effect, the group is proposing an audit of four decades of program comprehension research, with their validated proxies serving as the standard against which older instruments are judged. Such a re-examination could reshape how the field designs experiments, interprets past findings, and builds on one another’s work.

Collaborators at William & Mary will pursue another forward-looking thread: studying the possibility of replacing human participants in comprehension studies with large language model agents. If LLM agents can stand in for human readers in these experiments, research could scale dramatically, running far more comparisons of code designs and evaluation methods than human-subject studies allow. But that substitution only makes sense if the underlying measurements are sound, which is precisely what the NJIT team has worked to establish. Taken together, the award-winning paper suggests a field in the midst of a methodological reset, one that borrows the consensus-building rigor of the social sciences, elevates output prediction as the gold standard of understanding, and aims ultimately at a future in which both people and the AI systems that assist them produce code that is understandable by design.

Subject of Research: Reliability of measurement methods used in program comprehension research

Article Title: NJIT computing researchers awarded for code evaluation paper

Article References: NJIT computing researchers awarded for code evaluation paper. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: code comprehension, program comprehension, software engineering, Delphi method, cyclomatic complexity, large language models, NJIT, Automated Software Engineering, code quality, measurement reliability, output prediction, developer tools