New Formal Model paper format puts modelling plans under peer review before code is written

Every complex simulation begins life as a set of decisions: which processes to include, which to ignore, which equations and parameter values to trust, and which theoretical lens to apply. In most scientific publishing, those decisions remain invisible until the very end, buried in appendices or compressed into a methods section too thin to reproduce. A new publication format proposed in the Food and Ecological Systems Modelling Journal aims to drag those hidden choices into the light, by asking modellers to publish and peer review their intentions before a single line of code is finalized.

The proposal, published in October 2022, introduces the Formal Model as the first of three sequential publication steps: the Formal Model itself, followed by implementation documentation, and finally evaluation and testing. Until now, the first formal step of model development has typically been folded into a single research article covering development, implementation, and application all at once. That approach works tolerably for small models, the authors argue, but becomes unwieldy as models grow larger, more complicated, and more interconnected, drowning reviewers in detail and leaving crucial design decisions unexamined until after the analysis is complete.

The rationale is straightforward. Under the current system, a model is constructed, calibrated, tested, and applied, and only then does a peer-reviewed publication appear. Reviewers who question modelling decisions at that late stage can only suggest redesigns and reruns after enormous amounts of work have already been invested. The Formal Model moves the first point of peer review to the beginning of the modelling cycle, occupying roughly the first quarter of the first iteration. It is a reasoned statement of intent, deliberately separated from the model implementation in code, allowing reviewers to flag problems and suggest alternatives before resources are committed to a final version.

The format draws inspiration from the preregistration and registered report movements that have reshaped experimental science. Preregistration allows researchers to register their plans and generate a digital object identifier, creating accountability and a living document that distinguishes pre-diction from post-diction. But preregistration alone lacks peer review, meaning no one can catch flaws in the experimental design before the work proceeds. Registered reports solve this by having journals review the plans and agree in principle to publish the results, whatever they turn out to be. Preregistration has been adapted for mathematical and small computational cognitive models in psychology, yet the authors argue it cannot comfortably accommodate complex model design, where a modeller may iteratively implement, calibrate, and then apply a model to a host of analyses before any hypothesis testing begins.

The Formal Model is not the first attempt to tame model documentation. Well-commented code and tools such as Doxygen have long served implementation, but they say nothing about model structure and aims. The Overview, Design concepts, and Details protocol, developed for individual-based models, originally separated description from implementation, though later versions moved toward specifying code details, and the ODdox variant links the two together. Electronic notebooks have been suggested for the development process. In software engineering, the Formal Methods Model offers a precisely defined description of components and relationships for planning development, and the European Food Safety Authority once defined formal models as those in which model variables and parameters are linked into mathematical equations or algorithms, building on prior problem definition and conceptual model steps. The new format absorbs all three of those EFSA steps into a single standard yet flexible structure spanning disciplines.

So what does a Formal Model actually contain? The proposed template mirrors the familiar IMRAD structure of standard papers to make reading predictable, while retaining flexibility for different model types. It opens with an introduction laying out the reasons for creating the model, the supporting theory, and the modelling approach, rather than problems and hypotheses per se. An aim and purpose section serves as the problem formulation. A theoretical framework section declares the discipline and perspective, for example foraging theory or the metabolic theory of ecology, and explains why the chosen approach, such as agent-based methodology, is the best fit. A framing section then makes explicit the external influences knowingly left out of the model, a deliberate strategy to avoid false inclusion and false exclusion errors and to embrace what the authors call a modest approach to modelling that avoids strong claims.

The heart of the document is an overview of processes, followed by detailed descriptions of each component. For every process, the modeller reviews the current state of knowledge, citing the literature, tables, and diagrams used to develop the concepts, and then presents the planned implementation with formal representations such as equations or flow charts. State variables, spatial and temporal scales, and units must be specified, often tabulated with symbols, predetermined values, and sources. An example drawn from the agent-based model ApisRAM describes how every bee consumes resources and generates heat according to its metabolic rate, with temperature increase defined by a simple equation relating heat production to the amount of nectar burned and the heat capacity of the bee. An overview of components and their connections, typically with an overall diagram, serves as a roadmap to the details, and an optional section on model properties and behaviour can explore expected functioning under controlled conditions. A discussion covering strengths, weaknesses, exclusions, and the model’s place in the literature closes the format.

The authors are candid that the format is no silver bullet. It adds another documentation burden on top of normal documentation and user guides, and substantial documents require real commitment; the ApisRAM formal model, though not in the new format, runs to 58 pages. There is a risk that nobody reads it afterwards, and a risk of low adoption that would undermine any long-term improvement. Yet these downsides are likely counterbalanced by the reviewed publication status, the improvements gained through dialogue with reviewers, and the fact that the article will be cited whenever the model is used, increasing visibility and impact. If writing the Formal Model is folded into the design process itself, only a little extra effort should be needed to complete it.

Versioning addresses another drawback: many large models undergo several modelling cycles as new information arrives or new applications are needed, and a static reviewed document could confuse rather than clarify. The published Formal Model is therefore associated with a defined model version, typically 1.0. For smaller changes, authors could publish unreviewed updates as minor versions in journals that accept such formats or on a model website. For major upgrades, a second major version of the Formal Model may contain enough material for a full publication. Accompanying each revision with a version number represents the model’s development over time and can even form the basis for dialogue between modellers and stakeholders when versions are published in journals allowing comments.

Beyond transparency and reproducibility, the format carries a quieter ambition: giving modellers credit for the considerable work of extensive literature review, careful design, and ingenuity that often goes unacknowledged, with documented consequences for rigour and scientific quality. It also opens collaboration to people without modelling skills, since the conceptual groundwork becomes a publishable contribution in its own right. The Food and Ecological Systems Modelling Journal is currently the only journal offering this set of formats, though publication of all three steps is not mandatory, and an unfollowed Formal Model could even invite others to develop the proposed model. The authors hope that, through use, the format will improve and be improved by the modelling community, beginning a journey toward better models through better documentation.

Subject of Research: A proposed Formal Model publication format for peer review of modelling intent and data foundations before model implementation.

Article Title: The Formal Model article format: justifying modelling intent and a critical review of data foundations through publication

Article References: Topping, C. J., Kondrup Marcussen, L., Thomsen, P., & Chetcuti, J. (2022). The Formal Model article format: justifying modelling intent and a critical review of data foundations through publication. Food and Ecological Systems Modelling Journal, 3, Article e91024. https://doi.org/10.3897/fmj.3.91024

Image Credits: AI Generated

DOI: 10.3897/fmj.3.91024

Keywords: Formal Model, model documentation, peer review, preregistration, agent-based modelling, reproducibility, transparency, scientific modelling, Food and Ecological Systems Modelling Journal, model versioning, Formal, Model