Chess puzzles reveal how experts secretly judge similarity

What makes two chess puzzles feel similar to a strong player? A new study suggests the answer lies almost entirely in what the puzzles demand, not in how the board looks. The research, published in Cognitive Research: Principles and Implications, used supervised machine learning to reverse-engineer the implicit criteria that experts apply when judging similarity, and found that board appearance plays a negligible role.

Chess tactics offered the researchers a natural laboratory for this question, because a tactical puzzle separates what a problem looks like—the arrangement of pieces on the board—from what it requires, namely the solution. A team led by Matej Guid collected 2866 binary similarity annotations from a competitive chess player, who judged pairs of tactical puzzles. The team then trained interpretable classifiers to recover the decision structure hidden inside those judgments.

The result was a striking dissociation. In an ablation study, dynamic features—properties of the solutions—alone achieved an AUC of 0.901, while static features describing the board’s appearance reached only 0.554. In other words, the expert’s sense of similarity was driven by what the puzzles required, not by how they looked.

The classifiers also revealed the structure of the judgment itself. The expert’s implicit decision was well described by a dominant binary criterion with graded refinement: agreement on whether both solutions end in checkmate was near-necessary for a “similar” verdict, and after that, a graded comparison of the solution sequences decided the rest.

The criteria were not merely one player’s idiosyncrasy. Inter-rater validation with two additional titled players confirmed that the same standards were shared across experts, suggesting the recovered decision rule reflects a common professional intuition rather than a personal quirk.

Supporting a classic account of similarity, difference features alone nearly matched the performance of the full model, consistent with a feature-contrast representation of similarity of the kind proposed by Tversky in 1977.

One step of the pipeline proved especially revealing. After the model made its predictions, the expert re-evaluated disputed labels qualitatively—and reversed 70 percent of them, with every reversal converging with the model’s output. The combination of information retrieval, expert annotation, supervised classification, and qualitative re-evaluation constitutes a reusable pipeline, the authors argue, for reverse-engineering implicit expert criteria in other domains.

For chess training, the practical implication is direct: similarity-search systems should prioritize solution-level features over board appearance. The study is an open-access article published on 21 October 2026, and its authors note that the shared early version is subject to further edits before the final Version of Record.

Subject of Research: How expert chess players judge the similarity of tactical puzzles

Article Title: Reverse-engineering expert similarity judgments: evidence from chess tactics

Article References: Guid, M., Bizjak, M., Guid, T., & Krivec, J. (2026). Reverse-engineering expert similarity judgments: evidence from chess tactics. Cognitive Research: Principles and Implications. https://doi.org/10.1186/s41235-026-00755-5

Image Credits: AI Generated

DOI: 10.1186/s41235-026-00755-5

Keywords: chess, tactics, similarity judgments, expertise, machine learning, interpretable classifiers, AUC, Tversky, cognitive science, puzzle annotation, Reverse-engineering, expert