Pay Per Beep: Simple Incentive Tweaks Can Boost Experience Sampling Data Without Hurting Quality

Every beep of a smartphone survey is a small negotiation between science and daily life. Experience sampling — the method psychologists use to capture thoughts, moods, and behaviors in real time by pinging participants repeatedly throughout the day — lives or dies on whether people actually answer. A missed beep is not just an empty cell in a spreadsheet; it is a lost snapshot of someone’s lived experience, and enough lost snapshots can quietly erode the foundations of a study. Now a team of researchers at KU Leuven in Belgium has put one of the field’s most practical design questions under the experimental microscope: does the way you pay participants change how much data they give you, and how good that data is?

The study, published in the journal Behavior Research Methods, was led by Milla Pihlajamäki together with Ginette Lafit, Olivia J. Kirtley, Inez Myin-Germeys, and Gudrun Eisele. The team recruited 192 students and randomly assigned 64 of them to each of three incentive conditions. The first group received a fixed payment for taking part, the standard arrangement in most experience sampling research. The second group received the same fixed payment plus personalized feedback, a summary of the data they had contributed, on the theory that seeing one’s own psychological patterns might motivate more careful responding. The third group was paid per beep, earning money incrementally with every completed prompt, a structure that mirrors the piece-rate logic of behavioral economics rather than a flat salary model.

The protocol itself was demanding by design. For fourteen consecutive days, participants received nine prompts — or beeps — per day on their smartphones, producing up to 126 measurement moments per person and roughly 24,000 potential data points across the sample. This intensity is precisely what makes experience sampling so scientifically valuable and so operationally fragile. Unlike a one-off questionnaire, an intensive longitudinal design asks people to interrupt whatever they are doing — in a lecture, at dinner, with friends — to report on their inner states. Compliance, in this context, is not a given; it is an achievement that must be engineered through careful protocol design, and incentives are one of the most powerful levers available.

What makes the study methodologically notable is that the researchers did not stop at counting completed surveys. They systematically distinguished between data quantity and data quality, treating the two as separable outcomes that incentives might influence in different directions. Quantity was straightforward: compliance rates, or the proportion of beeps answered, and how that proportion changed over the two-week study period. Quality required more nuance. The team screened for careless responding — the phenomenon where participants click through items without genuine attention — and examined whether the temporal dynamics of such responses shifted across the study, alongside retrospective measures of participant burden and experience.

Careless responding matters because its consequences are surprisingly severe. As the methodological literature has repeatedly shown, even a modest proportion of inattentive respondents can distort correlations, inflate or deflate reliability estimates, and lead analysts astray in ways that are hard to detect after the fact. In experience sampling data, the problem is compounded by the sheer number of measurement moments and the fatigue that accumulates as the days wear on. A participant might respond attentively on day one and slide into patterned, mechanical answers by day ten. The Leuven team, drawing on their own prior work on the temporal dynamics of careless responding, built this dimension directly into the analysis, using analyses of variance and multilevel linear and logistic regression models to test whether incentive structure affected not only average behavior but its trajectory over time.

The headline finding is refreshingly clean: payment per beep increased compliance rates compared with the other two conditions, and that was essentially where the differences ended. The incremental payment scheme did not make participants answer more carelessly, did not change their reported burden, and did not alter their retrospective experience of the study in measurable ways. Personalized feedback, meanwhile — despite its intuitive appeal and growing popularity in mobile health and self-tracking applications — produced no measurable advantage over a plain fixed payment on any of the outcomes examined. The feedback condition neither boosted compliance nor improved the quality of responses, a null result that carries real practical weight for researchers weighing the added complexity of generating individualized reports against any assumed motivational payoff.

The finding that pay-per-beep worked without degrading quality is worth unpacking, because incentives in psychology have a complicated reputation. Classic motivation research has documented cases where external rewards crowd out intrinsic motivation or encourage gaming of the reward structure. In an experience sampling context, one might have worried that paying per beep would incentivize rapid, low-effort completion — quantity at the expense of quality. Recent related work on game-based rewards in experience sampling found exactly that pattern: gamification increased data quantity but reduced data quality. The Leuven results suggest that a straightforward monetary increment per completed beep does not carry the same risk, at least not in a motivated student population completing a two-week protocol.

The authors’ recommendation follows directly from the evidence: for motivated samples such as students, use payment per beep to boost data quantity. This is not a trivial prescription. Compliance in intensive longitudinal research varies enormously across populations and study designs, with meta-analyses reporting widely different completion rates depending on the sample, the burden of the protocol, and the population under study. Clinical populations, adolescents, and people experiencing acute distress often show lower compliance, and every percentage point of missing data constrains the statistical models that researchers use to extract meaning from these datasets. Multilevel time-series analyses, network models of psychological dynamics, and idiographic prediction approaches all benefit directly from denser, more complete measurement streams.

There are, of course, boundaries to the conclusion. The study was conducted in a student sample, a population that is generally well-motivated, digitally fluent, and accustomed to participating in research. Whether payment per beep would deliver the same benefit — and remain free of quality costs — in harder-to-reach populations, longer protocols, or studies measuring stigmatized behaviors remains an open question. The reward magnitudes involved were also specific to this context; the psychology of incremental incentives plausibly depends on the size of each increment relative to the total payment. And burden, while measured, was assessed retrospectively and through momentary reports within a demanding 14-day design; incentive effects on the lived experience of participation might differ in studies with different rhythms and demands.

Still, the study exemplifies a broader and welcome shift in psychological methods research: treating study design choices — payment schemes, feedback provision, sampling frequency, questionnaire length — as empirical questions rather than conventions inherited from previous studies. The team post-registered their study, documented every deviation transparently, and made all materials and analysis code openly available on the Open Science Framework, with data accessible through a secure checkout system. In a field increasingly aware that methodological decisions ripple through every downstream finding, knowing that a simple per-beep payment can fill in more of the data grid without compromising what fills those cells is exactly the kind of actionable, evidence-based guidance that turns methodological debate into better science.

Subject of Research: Effects of incentive type on data quantity and quality in experience sampling research

Article Title: The effect of incentive type on data quality and quantity in an experience sampling study in a student population

Article References: Pihlajamäki, M., Lafit, G., Kirtley, O. J., Myin-Germeys, I., & Eisele, G. (2026). The effect of incentive type on data quality and quantity in an experience sampling study in a student population. Behavior Research Methods, 58(11), Article 300. https://doi.org/10.3758/s13428-026-03164-0

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03164-0

Keywords: experience sampling, ecological momentary assessment, incentives, compliance, data quality, careless responding, survey methodology, participant payment, personalized feedback, intensive longitudinal data, burden, psychological methods