Shape-aware transformer boosts multivariate time series classification accuracy

A new artificial intelligence model that reads time series data the way a skilled human analyst might—by looking at both the overall shape of a signal and the raw numbers behind it—has demonstrated record-setting performance across a broad suite of classification benchmarks, including a notoriously difficult task in solar flare prediction. The model, called VSFormer, was developed by a team of computer scientists and solar physicists led by Wenjie Xi of George Mason University, and is described in an open-access paper published in Data Mining and Knowledge Discovery.

Multivariate time series classification, or MTSC, is one of the central problems of modern data mining. It involves assigning a class label to a collection of simultaneously recorded, time-ordered measurements—for example, multiple channels of an electrocardiogram, or dozens of magnetic field features observed on the Sun’s surface. Applications range from human activity recognition and motion sensing to the prediction of destructive space weather events. Over the past decade and a half, the dominant strategy has been to search for discriminative patterns: short, repeated subsequences, sometimes called shapelets, that reliably distinguish one class from another. This idea, first formalized by Ye and Keogh in 2009, has since been absorbed into numerous deep learning architectures that use neural networks to hunt for such patterns automatically.

But that strategy rests on a fragile assumption—that discriminative patterns actually exist in the data. In many real-world scenarios they do not. The clearest example comes from solar flare prediction, one of the case studies highlighted in the new paper. When researchers examined magnetic time series leading up to major flaring events versus quiet periods, they found no visually distinctive shapes separating the two classes. What separated them was the raw magnitude of the measurements: values in flaring samples ranged from roughly 0.08 to 0.12, at least an order of magnitude above the 0.0014 to 0.008 typical of non-flaring samples. Ironically, the very normalization step required by shape-based methods—rescaling subsequences so their shapes can be compared—strips away precisely this classifying information. VSFormer was designed from the ground up to avoid throwing either kind of information away.

The architecture is built as a two-branch Transformer. The first branch, dedicated to shape, uses the well-known STOMP motif discovery algorithm to mine repeated patterns from concatenated sequences belonging to each class and variable. It identifies the top-k motif pairs, selects prototype shapes from them, and then scans every individual time series to locate subsequences resembling those prototypes. Each match, together with its Z-normalized Euclidean distance to the prototype, becomes a “shape token” fed to a Transformer encoder. The second branch, dedicated to value, ignores shape entirely. Instead of treating individual time points as input—which prior work has shown conveys little label-relevant information—it slices each variable into intervals at multiple levels of granularity, from one single interval spanning the whole series up to ten equal-length segments. For every interval it computes three statistics: the mean, the standard deviation, and the slope derived from a linear regression fit. These become “value tokens,” capturing the numerical character of the signal at different temporal scales.

A decision layer then fuses the two branches, and this is where much of the model’s adaptivity lies. The learned shape and value representations are each projected into class probability space, and a balancing factor λ, computed through a linear layer and sigmoid activation from both representations, determines how much each branch contributes to the final classification. If shapes are genuinely discriminative in a dataset, λ shifts toward one and the shape branch dominates; if raw values carry the signal, λ falls toward zero. On the AtrialFibrillation and StandWalkJump datasets from the UEA archive, where discriminative electrocardiographic patterns are known to exist, the model autonomously learned to weight shape more heavily. On the solar flare data, it flipped, leaning on value information instead—an interpretable confirmation that the fusion mechanism responds to the actual structure of the data.

Perhaps the most technically distinctive contribution is the Prior-Enhanced Self-Attention, or PESA, mechanism. The authors argue that conventional self-attention in Transformers, when applied naively to time series, wastes capacity attending to features irrelevant to classification, injecting noise that degrades accuracy. PESA counteracts this by constructing a prior score matrix from class-specific information computed during training. For shape tokens, the prior weight combines two factors: a discriminativeness score for each prototype shape, based on the ratio of mean inter-class to intra-class match distances and amplified exponentially when the shape truly separates classes, and a similarity term that decays exponentially with the distance between a token’s shape and its prototype. For value tokens, the prior is an information-theoretic feature importance, computed as the entropy gain—the reduction in uncertainty about the class label that a given interval statistic provides. Element-wise multiplication of this prior matrix with the standard scaled dot-product attention scores, followed by softmax, steers the model’s attention toward tokens that demonstrably matter for the task.

The paper also revisits a subtle but consequential detail: positional encoding. Existing Transformer approaches for time series normalize variable indices, start timestamps, and end timestamps into the range zero to one. Xi and colleagues point out that this manufactures a spurious ordinal relationship among variables—normalizing three variables to one-third, two-thirds, and one makes the third variable appear numerically three times “larger” than the first, even though variables are discrete and unordered. Their Time Series Information encoding instead represents variable identities in binary form, preserving their discreteness, and augments every token’s encoding with the class-specific prior information described above. Ablation experiments across all thirty datasets showed this encoding beats learnable positional encodings, and that PESA consistently outperforms vanilla self-attention.

The empirical evaluation was exhaustive. VSFormer was tested on all thirty datasets in the UEA multivariate time series classification archive, the field’s standard benchmark collection, against a formidable lineup of competitors: distance-based baselines using Euclidean distance and dynamic time warping, the bag-of-patterns method WEASEL+MUSE, deep learning models including TapNet, ShapeNet, TARNet, RLPAM, SRL and MPTSNet, the widely used random convolutional kernel methods ROCKET and MiniRocket, and state-of-the-art Transformers such as TST, ConvTran, and SVP-T. VSFormer achieved the best average rank of 4.867 and, notably, the lowest standard deviation of ranks at 2.362, indicating unusually stable performance across datasets of wildly different character—a stability the authors attribute to its dual exploitation of shape and value. Rival methods that excel on particular datasets, such as RLPAM and TST, showed rank deviations nearly twice as large. Against the three leading Transformer baselines, VSFormer won or tied on 20, 23, and 23 of the 30 datasets respectively, and Wilcoxon signed-rank tests confirmed statistically significant superiority over ten of the compared methods at the p < 0.05 level.

The solar flare case study may prove the most consequential result for applied science. Using the Space Weather Analytics for Solar Flares dataset, built from 24 magnetic features measured at 12-minute cadence by the Helioseismic and Magnetic Imager aboard NASA’s Solar Dynamics Observatory, with flare events labeled by NOAA’s GOES satellites, the team constructed a carefully sampled, class-balanced problem from four sample types capturing flaring and non-flaring active regions. Because the data lacks discriminative shapes, most prior work has defaulted to timestamp-level features. When benchmarked against TST and SVP-T on both accuracy and area under the curve, VSFormer came out ahead of both. SVP-T, which relies entirely on shape representations, fared worst—a direct illustration of the paper’s central thesis that shape-only methods fail where shapes do not exist, while a model that can fall back on raw values thrives. The interpretable weighting confirmed the mechanism: the value branch dominated the learned decision.

Interpretability runs through the entire design. Because the inputs are subsequences and interval statistics rather than opaque embeddings, the model can highlight exactly which shapes or which intervals drove a classification. On StandWalkJump, visualizations of the highest-weighted shape tokens revealed that the “jumping” class exhibits a sharper peak than the “standing” class—a subtle but genuine difference the model detected. On the solar flare data, the highest-weighted value token corresponded to an interval whose mean value differed by three orders of magnitude between a flaring and non-flaring instance, making the basis for the model’s decision transparent to domain scientists.

The work is not without acknowledged limitations. The preprocessing stage, dominated by exact motif discovery with a computational cost that scales with the square of the number of sequences and the square of series length, can become expensive on very large datasets or extremely long time series, although the authors note it runs offline only once per dataset and that faster approximate motif discovery algorithms could be substituted. Within the trained model itself, attention cost is actually favorable: because shape and value tokens are far fewer than raw time points, the self-attention computation is more efficient than timestamp-level Transformers like TST. As time series data continues to proliferate across medicine, energy, finance, and space weather monitoring, VSFormer’s lesson is likely to resonate broadly: a model that refuses to assume what the data looks like—and can lean on either structure or magnitude as the evidence demands—may simply see more than one that looks for shapes alone.