Muon Colliders Could Unmask Forbidden Top-Quark Decays a Thousand Times Rarer Than Ever Seen

The top quark is the heaviest known elementary particle, and its extraordinary mass makes it a uniquely sensitive probe of physics beyond the Standard Model. Among the rarest processes it could undergo are flavor-changing neutral current (FCNC) decays, in which a top quark transforms into a lighter up-type quark while radiating a Z boson or a photon. In the Standard Model, such transitions are so violently suppressed by the Glashow–Iliopoulos–Maiani mechanism that their predicted branching ratios hover around one part in a hundred trillion for the Z channel and one part in a trillion for the photon channel. Any measurable event of this kind would therefore be an unambiguous beacon of new physics. A new theoretical study published in The European Physical Journal C by A. Senol, B. S. Ozaltin, M. Tekin and H. Denizli of Bolu Abant Izzet Baysal University quantifies just how well a future 10 TeV muon collider could hunt for these forbidden decays, and the answer is striking: roughly one order of magnitude beyond today’s best experimental limits.

The researchers focused on a single, carefully chosen process: the collision of a muon and an antimuon producing a muon neutrino, a positively charged muon, a b-tagged jet and one additional light jet, together with the charge-conjugate channel. This signature arises when an anomalous FCNC vertex produces a single top quark in association with a neutral gauge boson, after which the top decays through its ordinary charged-current channel into a W boson and a b quark, with the W decaying leptonically. The resulting final state, combining a charged lepton, missing transverse energy from the neutrino, a b-tagged jet and a light-flavor jet, offers a distinctive experimental fingerprint that can be efficiently isolated in the exceptionally clean environment that only a lepton collider can provide.

To parametrize possible deviations from the Standard Model without committing to any specific new-physics scenario, the team employed an effective field theory framework. They wrote down the most general effective Lagrangian describing anomalous tqZ and tqγ interactions, which contains vector-type coefficients X, tensor-type coefficients κ for the Z boson and tensor-type coefficients λ for the photon, each with independent left- and right-handed chiral structures. All of these couplings are in general complex numbers, so the analysis tracked both their real and imaginary components. The authors deliberately restricted their study to the anomalous neutral-current FCNC vertices, leaving a full global analysis including dimension-six four-fermion contact operators for future work.

A crucial theoretical insight shaped the entire analysis. At multi-TeV collision energies, the contributions of right-handed tensor couplings are strongly suppressed relative to their left-handed counterparts. The reason lies in the purely left-handed structure of the Standard Model tbW charged-current interaction: a right-handed FCNC vertex produces a top quark that must undergo a chirality flip before it can decay through the W boson, generating a suppression proportional to the ratio of the top-quark mass squared to the collision energy squared. At 10 TeV this suppression becomes decisive. The vector coupling X also contributes far less than the tensor operators across the parameter space. Consequently, the full detector-level analysis concentrated on the left-handed tensor couplings κ and λ, which dominate the sensitivity.

The simulation chain was deliberately realistic. The effective Lagrangian was implemented in FeynRules and exported as a Universal FeynRules Output model, event generation was performed with MadGraph5_aMC@NLO, parton showering and hadronization were handled by Pythia 8.2, and fast detector simulation used Delphes 3.5.6 with a dedicated detector card designed for a 10 TeV muon collider. Jets were reconstructed with the anti-kt algorithm using FastJet. For each signal benchmark and each Standard Model background, two million events were generated to guarantee statistical precision. The backgrounds included irreducible electroweak production of the exact final state, diboson processes such as WW, ZZ, Zγ, WWγ and WWZ, and top-quark pair and single-top production, the latter being the most stubborn contaminants because they naturally contain real b-jets and leptons resembling the signal topology.

Event selection proceeded in two stages. A set of pre-selection requirements, demanding at least one jet and one b-tagged jet, exactly one muon and no electrons, together with kinematic cuts on transverse momenta, pseudorapidities, missing transverse energy and angular separations, already eliminated much of the multiboson background. But cut-based selection alone proved insufficient for optimal separation, so the team turned to a multivariate analysis based on boosted decision trees. The classifier was trained on an extended set of kinematic observables, including transverse momenta, pseudorapidities, angular separations between final-state objects, and reconstructed system-level variables. Using the Toolkit for Multivariate Data Analysis with adaptive boosting, 450 decision trees of maximum depth three, and a learning rate of 0.5, the BDT achieved a clear separation, with signal events populating the positive response region and backgrounds concentrated at negative values.

The single most powerful discriminating variable turned out to be the invariant mass of the reconstructed b-jet and light-jet system, followed by the transverse momenta of the leading jet, the b-tagged jet and the muon, and angular observables capturing the characteristic spatial configuration of signal events. After applying the BDT selection, backgrounds such as ZZ and Zγ were completely eliminated, and the dominant top-quark backgrounds were reduced by more than an order of magnitude in some cases while signal efficiency remained high. In the invariant mass distribution of the muon, b-jet and light-jet system, the signal emerged prominently in the high-mass region between 2 and 10 TeV, a region where the anomalous contributions, growing with energy, naturally concentrate.

The statistical interpretation went beyond the conventional one-dimensional approach. Instead of varying a single coupling at a time, the authors performed simultaneous two-parameter scans of the real and imaginary components of κ and λ, extracting both 95 percent confidence level exclusion contours and 5-sigma discovery contours from profile-likelihood significance estimators. Systematic uncertainties of 0, 5 and 10 percent were considered, covering background normalization, luminosity, lepton identification, b-tagging and jet reconstruction. The resulting contours are approximately elliptical and centered on the origin, with a small rotation of the principal axes revealing a nonzero correlation between the two couplings. Even under the conservative 10 percent systematic assumption, the projected sensitivities remain at the level of a few times ten to the minus three for the effective couplings, and the real and imaginary components yield nearly identical limits, showing that the reach depends mainly on the magnitude of the anomalous interactions rather than their phase.

Translated into branching ratios, the numbers are remarkable. With no systematic uncertainty, the 95 percent confidence level exclusion reach extends to branching ratios as low as 6.77 times ten to the minus seven for t going to qZ and 4.97 times ten to the minus seven for t going to qγ, while the 5-sigma discovery thresholds sit near 1.63 and 1.07 times ten to the minus six respectively. These figures improve upon the current ATLAS and CMS limits, which stand at roughly 6.2 times ten to the minus five for t going to uZ and 0.85 times ten to the minus five for t going to uγ, by approximately one to three orders of magnitude depending on the channel. They also outperform the projected sensitivities of proposed electron-positron facilities such as FCC-ee and CLIC, whose lower collision energies constrain their kinematic reach for single-top FCNC production. The advantage stems from the unique combination of a 10 TeV center-of-mass energy, an integrated luminosity of 10 inverse attobarns, and the clean lepton-collider environment, all of which conspire to amplify sensitivity to dipole-type FCNC operators whose effects grow with energy.

The broader message is one of complementarity and promise. Muons, being about two hundred times heavier than electrons, radiate far less synchrotron radiation, allowing circular machines to reach multi-TeV energies while maintaining high luminosity, a possibility highlighted in the 2020 European Strategy for Particle Physics. At such energies, vector boson fusion dominates total rates, but direct annihilation channels like the one studied here retain distinctive kinematic signatures with invariant masses sharply peaked near the collision energy. The new analysis demonstrates that the process muon-antimuon annihilation into a neutrino, muon, b-jet and light jet is among the most promising windows onto anomalous top-quark FCNC interactions. Because the projected sensitivities remain many orders of magnitude above the vanishing Standard Model expectations, any event observed within the explored parameter space would constitute a clear indication of physics beyond the Standard Model, making future muon colliders one of the most exciting arenas for the next generation of top-quark discoveries.

Subject of Research: Sensitivity of a future 10 TeV muon collider to anomalous top-quark flavor-changing neutral current interactions

Article Title: Sensitivity to top-quark FCNC interactions at future muon colliders

Article References: Senol, A., Ozaltay, B. S., Tekin, M., & Denizli, H. (2026). Sensitivity to top-quark FCNC interactions at future muon colliders. The European Physical Journal C, 86(9), Article 1068. https://doi.org/10.1140/epjc/s10052-026-16357-w

Image Credits: AI Generated

DOI: 10.1140/epjc/s10052-026-16357-w

Keywords: top quark, flavor-changing neutral current, muon collider, effective field theory, beyond the Standard Model, boosted decision trees, Monte Carlo simulation, tqZ coupling, tqgamma coupling, branching ratio, particle physics, collider phenomenology