The case against subword tokenization is technical but consequential. Because a token such as a whole word is treated as an atomic unit, information about the individual characters inside it is largely obscured. That matters little for casual conversation, but it becomes a serious handicap for scientific data, where meaning often lives at the character level: computer code, DNA and protein sequences, chemical formulae and multilingual text with unusual orthographies all depend on fine-grained structure that subword vocabularies flatten. Subword tokenization also introduces subtler pathologies, including a bias towards particular responses depending on how a prompt happens to be segmented, a fixed vocabulary that in practice skews models towards English, and a rigid allocation of compute that spends the same effort on every token regardless of difficulty.

Byte-level models, which treat the UTF-8 encoding of text as their basic units, promise to solve these problems in principle. Every character, in every language, decomposes into the same 256 bytes, so no vocabulary is needed and no character information is lost. Yet despite years of promising research, byte-level models have never been adopted by any leading laboratory. The researchers behind the new work identify a structural reason for this mismatch between theory and practice: byte-level models have almost always been trained from random initialization and compared against subword models also trained from scratch, while the state of the art in subword model training, spanning data curation, architecture and post-training, evolves so quickly that byte-level efforts cannot keep pace.

Their solution is to stop competing on that uneven playing field and instead convert the winners. The team, led by Benjamin Minixhofer of the Allen Institute for AI and the University of Cambridge, together with colleagues at the University of Washington, Imperial College London and LMU Munich, calls the procedure byteification. Starting from fully open source models, including Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base and Llama 3 8B, they produced byte-level counterparts named Bolmo 7B, Bolmo 1B, Bwen 8B and Blama 8B. The total conversion budget was 49.1 billion tokens, less than one percent of a typical pretraining run, yet the resulting models approach the performance of their subword parents and decisively outperform earlier byte-level systems of comparable size.

The architecture underlying the converted models belongs to a family the authors call latent tokenizer language models. A shallow local encoder first builds contextualized representations of each byte. A boundary predictor then decides where to place patch boundaries, grouping bytes into patches. These patch representations are pooled and passed through the deep global transformer, which does most of the computational work, before being depooled back to the byte level by a local decoder that predicts the next byte. Crucially, the global model is inherited intact from the source subword model, so the expensive knowledge embedded in its weights is preserved rather than relearned.

One of the study’s most elegant insights concerns a subtle asymmetry in how boundaries are placed. Subword tokenizers are not causal: they consult future characters when deciding where a token ends. A tokenizer might split the flowerbed after flower in one sentence but keep flowerb together in another, even though the text up to that point is identical, because what comes next differs. Earlier byte-level architectures, which must decide boundaries using only past context during generation, cannot replicate this behaviour, creating an expressivity gap. The team resolved it with a two-part scheme: during prefilling, their boundary predictor peeks one byte into the future, which proved sufficient to match subword behaviour, while during decoding a special boundary symbol fused into an expanded 512-entry byte vocabulary lets the model emit boundary decisions alongside each byte at essentially no extra cost.

The conversion itself proceeds in two stages. In the first, the global model is frozen while the new local components are trained to exactly recover the behaviour of the source model, using distillation losses that match pooled byte representations to subword embeddings and patch likelihoods to subword token likelihoods. The boundary predictor alone reaches over 99 percent accuracy in emulating the original tokenizer. In the second stage, the entire model is trained end to end so it can exploit genuinely byte-level information. Ablations showed that skipping the first stage hurts, particularly for smaller models, but also that the first stage serves a second purpose: it provides a fast feedback loop for testing whether a candidate architecture has enough capacity before committing to full training.

The performance results are striking. Bolmo 7B achieved a 16.5 percent absolute improvement in STEM tasks over BLT 7B, a leading byte-level model trained from scratch, and substantially outperformed its own subword parent on character-understanding benchmarks such as CUTE and the multilingual EXECUTE suite, aided by a small injection of synthetic character-manipulation training data. Bwen 8B, built on Qwen3, statistically significantly outperformed all earlier public byte-level models and came close to, and in some categories surpassed, its source model. The byteified models also showed higher pass@16 rates on code generation at lower pass@1, suggesting they produce more diverse candidate solutions than their subword counterparts, an observation the authors flag as requiring further study.

Two further results point toward practical advantages beyond raw accuracy. Because byte-level models are not locked into a fixed vocabulary, the researchers could raise the average number of bytes per patch to trade a smooth, controlled amount of performance for inference speed, avoiding the softmax bottleneck that eventually causes subword models transferred to larger SuperBPE vocabularies to become Pareto-dominated. And because byteification preserves the ecosystem around the source model, the team could transfer instruction-following abilities from a post-trained Olmo 3 checkpoint into Bolmo via task arithmetic, simply adding the weight difference between the post-trained and base transformer layers, with no additional training at all, lifting Bolmo to parity with the original post-trained model on the IFEval benchmark.

The authors position byteification not as a replacement for training byte-level models from scratch but as a complementary, cheap pathway that removes a long-standing performance barrier. If any state-of-the-art subword model can be converted to the byte level for a few percent of its original training cost, high-performing byte-level architectures can be identified quickly, and the English-centric biases, character-blindness and rigid compute allocation of subword tokenization may finally become optional rather than inevitable. For domains where meaning lives in the characters, from genomics to software engineering, the end-to-end byte-level future that tokenization researchers have long promised may have just moved considerably closer.

Subject of Research: Converting subword-based large language models into byte-level models through a retrofitting method called byteification

Article Title: Retrofitting language models to operate over bytes

Article References: Minixhofer, B., Murray, T., Limisiewicz, T., Korhonen, A., Zettlemoyer, L., Smith, N. A., Ponti, E. M., Soldaini, L., & Hofmann, V. (2026). Retrofitting language models to operate over bytes. Nature. https://doi.org/10.1038/s41586-026-11111-4

Image Credits: AI Generated

DOI: 10.1038/s41586-026-11111-4

Keywords: large language models, byteification, tokenization, byte-level modelling, subword tokenization, latent tokenizer, machine learning, natural language processing, character-level understanding, model distillation, task arithmetic, computational efficiency