fMRI is a deceptively tricky technology. It does not measure neural activity directly; instead, it tracks the blood oxygen level-dependent (BOLD) signal, an indirect proxy for where neurons are firing. A single scan yields a four-dimensional dataset, three spatial dimensions stacked through time, containing tens of thousands to hundreds of thousands of voxels, each with its own slowly fluctuating time series. The signal is noisy, contaminated by head motion, breathing, and scanner drift, and it varies enormously from person to person and from scanner to scanner. Traditional statistical methods, such as generalized linear models and static functional connectivity measures, assume linearity or average activity over an entire scan, discarding precisely the dynamic brain reorganization that scientists most want to understand.

Enter the Transformer. Introduced by Vaswani and colleagues in 2017 for natural language processing, the architecture rests on a deceptively simple mechanism: self-attention, which computes relationships between every pair of elements in a sequence, allowing the model to capture long-range dependencies without the fixed local windows of convolutional networks or the sequential bottleneck of recurrent ones. When researchers realized that a brain scan could be chopped into tokens just like a sentence, in time, in space, or both, the field began to reorganize itself around the idea. The review’s authors, led by Xiaomin Yang and Bocheng Wang of the Communication University of Zhejiang, sort this rapidly growing literature into four paradigms: temporal modeling, spatial modeling, joint spatiotemporal modeling, and self-supervised learning.

Temporal models treat fMRI as a multivariate time series. Brain atlases such as AAL or Schaefer collapse the brain into a set of regions of interest, and each moment in the scan becomes a token representing the brain’s global state. Self-attention then builds a matrix of dependencies across all time points, capturing the slow-frequency fluctuations below 0.1 Hertz that carry biologically meaningful dynamics, from transient synchronization between networks to spontaneous microstate shifts at rest. Models like BolT use windowed attention to make long sequences computationally tractable, and have delivered strong results classifying major depressive disorder, while transfer-learning approaches have adapted Vision Transformer weights pretrained on images to fMRI time series, improving detection of mild cognitive impairment and Alzheimer’s disease.

Spatial models, by contrast, ask how brain regions collaborate regardless of when. Here the field has discovered something profound: functional brain networks are fundamentally non-Euclidean. Regions that work together tightly, such as nodes of the default mode network spanning the front and back of the brain, are often physically far apart, which is exactly where convolutional networks with local receptive fields fail. A spatial Transformer’s attention matrix effectively learns a dynamic functional connectivity map from the data itself, without the predefined graph structures that constrain graph neural networks. Landmark systems such as BrainNetTF treat each anatomical region as a token and model the whole-brain topology end to end, while architectures like Brainformer preserve 3D coordinates and VoxelFormer merges semantically similar tokens to tame the computational explosion of voxel-level data.

But the brain is neither purely temporal nor purely spatial. Network neuroscience insists that brain function is a dynamic system in which space and time are deeply intertwined, and forcing the data into a single dimension inevitably loses something. Joint spatiotemporal modeling, the review’s third paradigm, tackles this directly. Factorized approaches such as Twin-Transformer run parallel spatial and temporal branches, while unified 4D models like SwiFT extend the shifted-window mechanism of the Swin Transformer into four dimensions, computing attention locally in space-time while preserving global information flow. Newer systems such as NeuroSTORM abandon predefined atlases entirely, extracting fine-grained, atlas-free features across the whole 4D volume. The payoff is the ability to explicitly model dynamic functional connectivity: to see not just whether a brain is abnormal, but when, where, and how its networks reorganize, a shift the authors describe as moving from black-box classifiers to dynamic process analyzers.

The fourth and arguably most consequential paradigm is self-supervised learning, which attacks fMRI’s most stubborn problem: high-dimensional data, tiny samples. Scanning is expensive and clinical cohorts are often only a few hundred subjects, yet voxel-level feature counts run into the hundreds of thousands, a recipe for overfitting. Self-supervised methods sidestep the need for labels by inventing pretext tasks. Masked autoencoders hide large chunks of the input, up to 75 percent of spatiotemporal patches in some 3D implementations, and force the model to reconstruct the missing signal, compelling it to internalize the constraints of brain anatomy and network organization. Contrastive learning takes a different route, teaching models to recognize that two augmented views of the same subject’s scan are more alike than scans of different people, sharpening sensitivity to individual differences.

These techniques are converging on something bigger: brain foundation models. BrainLM, pretrained on roughly 80,000 scans from the UK Biobank, supports zero-shot inference of brain latent states and few-shot adaptation to clinical prediction tasks such as depression and anxiety. BrainSymphony incorporates structural connectivity priors during pretraining, aligning learned representations with the brain’s true structure-function coupling. Recent work has even demonstrated that scaling laws, the predictable performance gains that transformed language models, hold for brain imaging models, a finding the review highlights as evidence that the field is entering its own era of large-scale pretraining.

The clinical stakes are high. Spatiotemporal graph Transformers have outperformed conventional deep learning in distinguishing Alzheimer’s patients from healthy controls; mixture-of-experts architectures fuse fMRI, structural MRI, and diffusion imaging to separate Parkinson’s disease from its prodromal stage; and multi-atlas dynamic connectivity Transformers are pushing autism classification forward. On the decoding side, Transformer-based models have reconstructed photorealistic images from brain activity and can even generate natural language descriptions of what a person is viewing. The same global modeling capability that makes Transformers good at diagnosis also makes them more robust to the scanner-to-scanner variability that has long plagued multi-site clinical studies, though domain shift remains far from solved.

The review is careful not to oversell. Self-attention scales quadratically with sequence length, making whole-brain 4D modeling ferociously expensive, and window approximations trade away some long-range fidelity. More troubling is interpretability: a lively technical debate over whether attention weights explain anything at all cautions against reading attention maps as maps of neural causality. The statistical associations a model learns are not anatomical connections, and whether pretrained representations truly reflect functional brain organization still demands validation with neuroscience methods. The authors argue the path forward lies in hybrid strategies, embedding structural connectivity, anatomical priors, and causal inference techniques into attention mechanisms, and in building unified data standards so that the next generation of foundation models can generalize across centers. If they succeed, the technology behind chatbots may become the lens through which medicine finally reads the brain in motion.

Subject of Research: Transformer-based machine learning methods for fMRI brain data analysis

Article Title: Transformers for fMRI analysis: from spatiotemporal modeling to brain foundation models

Article References: Yang, X., Li, Y., Zhang, X., & Wang, B. (2026). Transformers for fMRI analysis: from spatiotemporal modeling to brain foundation models. Frontiers in Neuroinformatics, 20, Article 1869687. https://doi.org/10.3389/fninf.2026.1869687

Image Credits: AI Generated

DOI: 10.3389/fninf.2026.1869687

Keywords: fMRI, Transformer, self-attention, brain foundation models, functional connectivity, self-supervised learning, masked autoencoders, neuroimaging, brain decoding, BOLD signal, deep learning, clinical diagnosis

The case study at the heart of the work is the proposed FASER calorimeter, or FASERCAL, an upgrade concept for the FASER experiment at CERN, which sits in the path of the collider’s forward neutrino beam. Its primary interaction volume, a highly granular three-dimensional calorimeter known as 3DCAL, comprises more than 460,000 scintillator voxels arranged in ten longitudinal modules, only a small fraction of which light up in any given event. Downstream, an electromagnetic calorimeter, a hadronic calorimeter and a muon spectrometer complete the detector. The beam contains electron, muon and tau neutrinos, and the analyses target both charged-current and neutral-current interactions. The resulting event images contain dense shower cores, extended secondary tracks, partial containment and deeply ambiguous local configurations. In this regime, the authors argue, machine learning is not an optional enhancement to an otherwise tractable reconstruction problem; it is a mandatory prerequisite for extracting any event-level physics at all.

The team’s approach borrows a page from the masked-autoencoder techniques that have transformed language and vision modelling. During pretraining, 75 percent of the occupied calorimeter patches are hidden, and a lightweight decoder must reconstruct the missing voxel-level occupancy and charge from whatever remains. To succeed, the encoder is forced to infer non-local spatial correlations, recovering broad shower envelopes, elongated track-like structures and coherent energy flow into the masked regions. On top of this reconstruction objective, the researchers added what they call relational voxel-level tasks: predicting, for each visible patch, whether it contains ghost activity with no matched true particle, whether it belongs to background, primary or secondary interaction hierarchy, and whether its deposits are electromagnetic, muonic or hadronic in character. These targets are especially demanding because a single reconstructed voxel can receive contributions from multiple true particles, yielding soft class mixtures rather than clean one-hot assignments.

The architecture itself is tailored to the detector’s heterogeneity. Sparse three-dimensional convolutions convert the 3DCAL and hadronic-calorimeter voxel grids into patch tokens, so that computational cost scales with detector occupancy rather than with the total instrumented volume. Module-aware self-attention blocks then capture local shower patterns within each of the ten detector modules before a Perceiver-IO bottleneck fuses the calorimetric tokens with compact representations of the electromagnetic calorimeter’s energy matrix and the variable-length muon-spectrometer tracks. The full fine-tuned model contains roughly 32 million trainable parameters. Pretraining ran on eight NVIDIA GH200 GPUs, with the masked-reconstruction phase consuming 584 GPU-hours and the relational continuation adding a further 427, an incremental cost of 73 percent for targeted gains in the hardest channels.

The downstream results are striking, and the pattern of improvement is as informative as its magnitude. For the dominant electron-neutrino charged-current channel, the area under the receiver operating characteristic curve rose from 0.968 for a model trained from scratch to 0.985 with full pretraining; for muon neutrinos, from 0.909 to 0.958; and for neutral-current events, from 0.885 to 0.947. But the most consequential gains appeared in the tau-neutrino channels, precisely where dense overlap, secondary activity and partial containment make classification hardest. For hadronic tau decays, the maximum figure of merit, a standard signal-to-background metric, jumped from 1.58 to 4.58. For the electron tau channel it nearly quadrupled, from 0.38 to 1.35. Charmed-quark identification showed the same structure, with the relational objectives disproportionately improving the lower-yield channels that matter most for the detector’s physics reach.

Regression tasks benefited as well. Primary-vertex reconstruction showed the clearest and most uniform effect, with error distributions for the pretrained models shifted towards smaller values and their interquartile ranges tightened across essentially all event categories. Visible energy, missing transverse momentum and jet momentum also improved, though less uniformly, and at the smallest label budgets of a few hundred events pretraining was not consistently advantageous for every continuous target. The authors are careful about such nuances, presenting the relational objective as generally superior across the task suite rather than universally dominant, and noting that masked reconstruction alone accounts for most of the gain in several regression quantities.

Perhaps the most operationally significant finding concerns data efficiency. Labelled events in an energy-frontier neutrino programme are not a cheap by-product: rare channels require dedicated simulation campaigns, systematic variations are costly, and many target quantities depend on expensive truth association. The data-efficiency study varied labelled training budgets from one hundred to one hundred thousand events while keeping validation and test sets fixed. With roughly one thousand labelled events, the fully pretrained encoder reached a flavour-classification macro-AUROC of 0.818, exceeding the 0.807 achieved by a scratch-trained model with ten times more data. Vertex reconstruction showed a similar compression, with the mean error falling from 239 millimetres for scratch training to 100 millimetres with full pretraining at the same budget. That leftward shift of the performance frontier changes not only how efficiently models can be trained but which physics studies are practical to pursue.

The team also probed whether the learned representation carries meaning beyond its original detector. Saliency maps concentrated attribution near the interaction region and along the main downstream shower and track structures rather than spreading it diffusely. Latent-space projections revealed cleaner flavour groupings and smoother visible-energy progressions for the pretrained variants. Detector-subsystem ablations recovered a physically plausible division of labour: removing the 3DCAL collapsed flavour performance and inflated the vertex error from 21 millimetres to nearly a metre, while dropping the hadronic calorimeter most strongly affected hadronic discrimination, the muon spectrometer degraded muonic channels, and removing the electromagnetic calorimeter worsened energy measurements. A coherent energy-scale stress test, shifting all calorimeter responses by up to ten percent at inference time, produced only mild drifts, suggesting the representation is not acutely brittle to global calibration biases.

Transfer beyond the source domain provided further evidence of reusability. On a public fine-grained plastic-scintillator benchmark at giga-electronvolt energies, the pretrained encoder improved classification diagonals for protons, charged pions, muons and electrons over scratch training, surpassing the strongest published baseline for three of the four species. On PILArNet, a public liquid-argon time-projection-chamber dataset representing a far greater technological and energy shift, the tera-electronvolt-scale encoder adapted to particle-level classification and, on the multi-particle task, edged past the best published ensemble results in both accuracy and area under the curve. A matched alternative-generator stress test, evaluating the same checkpoints on NuWro events matched event-by-event to the GENIE training sample, showed stable flavour and kinematic performance while exposing charm tagging as generator sensitive, a boundary the authors interpret as evidence of task-dependent robustness rather than generator independence.

The researchers are explicit about limits: the study is simulation based, the relational targets inherit simulation truth, and a full detector foundation model would require broader coverage, stronger systematic validation and ultimately calibration on experimental data. They reserve the phrase foundation-style for what they have demonstrated: a single reusable encoder, trained with objectives independent of downstream labels, adapted to many physics tasks, effective with few labels and transferable across detector technologies. For energy-frontier neutrino physics, where event complexity makes artificial intelligence a requirement rather than an optional convenience, that is already a substantive result, and it outlines a concrete path towards the general-purpose detector encoders that future experiments may one day take for granted.

Subject of Research: Self-supervised pretraining of sparse vision transformer models for energy-frontier heterogeneous neutrino detector event reconstruction

Article Title: Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pretraining

Article References: Alonso-Monsalve, S., Cufino, F., Kose, U., Mascellani, A., & Rubbia, A. (2026). Towards foundation-style models for energy-frontier heterogeneous neutrino detectors via self-supervised pretraining. Nature Machine Intelligence. https://doi.org/10.1038/s42256-026-01309-6

Image Credits: AI Generated

DOI: 10.1038/s42256-026-01309-6

Keywords: neutrino physics, self-supervised learning, masked autoencoders, machine learning, particle detectors, FASER, CERN, vision transformers, transfer learning, event reconstruction, Nature Machine Intelligence, foundation models

The appeal of 3D data is easy to grasp. A photograph flattens the world onto a grid of pixels, discarding the very depth information that lets humans judge distance, volume and shape. Three-dimensional data, by contrast, carries precise spatial measurements, allowing an algorithm to know not just that a pedestrian is present but exactly how far away they stand and how much space they occupy. The survey identifies five dominant ways of representing this information: point clouds, voxel grids, depth and normal maps, polygonal meshes and neural fields. Each comes with trade-offs in computational cost, storage demands and expressive power, and choosing the right representation is often the first critical decision in any 3D pipeline.

Point clouds deserve particular attention because they dominate real-world sensing. Produced by LiDAR scanners and depth cameras, a point cloud is simply a set of points floating in space, each carrying a 3D coordinate and sometimes color or intensity. The trouble is that these points are irregular and unordered, which makes them fundamentally incompatible with the convolutional neural networks that revolutionized 2D image analysis. The field’s answer is sparse convolution, a clever trick that performs computation only on non-empty regions of a voxelized space. Because most of a scanned scene is empty air, sparse convolution slashes the computational burden dramatically. Its cousin, sub-manifold sparse convolution, goes further by restricting computation to kernel centers that already contain data, preventing the dilation that would otherwise destroy the input’s sparsity, though the dilation property itself can be useful for giving models contextual awareness of their surroundings.

With these foundations in place, the survey turns to the heart of modern 3D research: pre-training. The idea borrows directly from the playbook that made 2D vision so successful. Models pre-trained on ImageNet’s fourteen million labeled images learned general-purpose features that transferred effortlessly to new tasks. But 3D data lacks an equivalent of ImageNet, largely because annotating millions of 3D scans is prohibitively expensive. The solution has been self-supervised learning, in which models generate their own training signals from raw data. The survey organizes these efforts into three families: contrastive methods, masked autoencoder approaches and rendering-based techniques.

Contrastive learning teaches a network to pull similar data points together in an embedding space while pushing dissimilar ones apart, typically through an InfoNCE loss. In the 3D world this takes several forms. PointContrast, a landmark 3D-to-3D method, encourages networks to learn representations that remain stable under different viewpoints and noise, directly boosting downstream detection and segmentation. Other approaches reach across modalities: CrossPoint and SLidR bridge point clouds and rendered images, transferring knowledge from 2D backbones trained on vast image collections into 3D networks. Perhaps most strikingly, methods like PointCLIP and its successor PointCLIPV2 connect 3D perception to language, projecting point clouds into multi-view images and aligning them with the text embeddings of CLIP-style models. This enables zero-shot recognition, where a model classifies objects it has never been trained on simply by matching visual features to textual descriptions.

Masked autoencoder methods take a different route, inspired by the masked image modeling that powered recent breakthroughs in 2D vision. The core idea is to corrupt or hide parts of the input and force the model to reconstruct them, thereby learning the underlying structure of 3D shapes. VoxelMAE converts unstructured points into voxel grids and predicts the occupancy of masked voxels. PointMAE, borrowing a page from BERT, tokenizes point clouds using farthest point sampling and K-nearest-neighbor grouping, then trains with an L2 loss on masked token features; its descendant PointM2AE adds pyramid architectures that capture both fine geometric detail and high-level semantics, achieving state-of-the-art linear classification on the ModelNet40 benchmark. GD-MAE extends the paradigm to convolutional backbones favored in autonomous driving, with a masking strategy designed to prevent knowledge leakage during downsampling.

The third family is the most visually intuitive: rendering-based pre-training. Differentiable rendering makes the process of drawing an image from a 3D model mathematically transparent, so gradients can flow backward from the rendered image to the 3D representation itself. The Ponder framework describes 3D surfaces within a volume and optimizes them against 2D image projections using a NeRF-like structure, while PonderV2 scales the idea to outdoor scenes and can pre-train both 2D and 3D backbones. According to the survey, this rendering-based approach outperforms both contrastive and masked-autoencoder methods, suggesting that grounding 3D learning in the physics of image formation may be the field’s most promising direction.

These pre-trained representations then feed a rich ecosystem of downstream tasks. Classification, the field’s founding problem, began with PointNet’s pioneering use of shared multi-layer perceptrons on raw points and matured through architectures like DGCNN’s EdgeConv, which builds local graphs to learn edge embeddings. Segmentation assigns a category to every point in a scan and has spawned a fierce architectural competition: MinkowskiUNet applies U-Net-style designs to voxel data, Cylinder3D replaces cubic partitions with cylindrical ones to handle LiDAR’s varying point density, SphereFormer exploits the transformer’s global receptive field, and RangeFormer brings powerful 2D-style transformers to range-view representations. Detection, crucial for autonomous vehicles, spans LiDAR-based methods like PointRCNN and PVRCNN, camera-based systems like BEVFormer that follow Tesla’s vision-centric pipeline, and fusion approaches like PointPainting and LoGoNet that marry the precise geometry of LiDAR with the rich texture of images. Tracking, matching and registration round out the toolkit, with methods like GeoTransformer and robust variants of the classic ICP algorithm enabling machines to align separate 3D scans into coherent wholes.

Progress on all these fronts is measured against a demanding set of benchmarks. ModelNet40 and ShapeNet test synthetic shape understanding, ScanNetV2 and S3DIS probe indoor scene parsing, while KITTI, nuScenes, Waymo and ONCE push algorithms to their limits in real-world driving scenarios containing millions of annotated frames. Evaluation relies on metrics such as mean average precision, computed from intersection-over-union thresholds between predicted and ground-truth boxes, and mean intersection-over-union for segmentation, with registration tasks judged by rotation error, translation error and registration recall. The survey’s careful cataloging of these standards provides newcomers with an immediate map of how the field keeps score.

Looking forward, the authors identify several frontiers that could define the next decade. Large language models, brimming with world knowledge distilled from vast text corpora, are only beginning to be connected to 3D perception through systems like Uni3D-LLM. Knowledge transfer from the data-rich 2D and language domains into the data-starved 3D world remains a wide-open opportunity. Synthetic data could relieve the crushing cost of manual annotation, though bridging the gap between simulated and real scenes will demand advances in domain adaptation and neural rendering. Most ambitiously, the field awaits its own foundation model: while 2D vision has been reshaped by systems like Segment Anything, no equivalent exists for 3D, largely because large-scale 3D benchmarks are still lacking. Building one, the survey argues, could dramatically cut the design and deployment costs of countless 3D applications and pave the way toward a genuine era of 3D artificial general intelligence. For a field that measures the world in three dimensions, the trajectory is unmistakably upward.

Subject of Research: Self-supervised pre-training methods and downstream tasks for 3D point cloud perception

Article Title: Advances in 3D pre-training and downstream tasks: a survey

Article References: Hou, Y., Huang, X., Tang, S., He, T., & Ouyang, W. (2024). Advances in 3D pre-training and downstream tasks: a survey. Vicinagearth, 1(1), Article 6. https://doi.org/10.1007/s44336-024-00007-4

Image Credits: AI Generated

DOI: 10.1007/s44336-024-00007-4

Keywords: 3D perception, point clouds, self-supervised learning, pre-training, masked autoencoders, contrastive learning, sparse convolution, LiDAR, autonomous driving, neural rendering, foundation models, benchmarks