How Next-Concept Modeling rethinks AI text generation

One of the research trends I’m constantly following is alternatives to autoregressive language models. Despite being very useful, they have their limitations, and I’m always looking for new architectures that can overcome these limitations. 

This is why I became interested in Next-Concept Modeling (NCM), a new AI architecture that changes how models generate text. Developed by researchers at Stanford University, NCM ditches the next-token prediction scheme and learns to represent entire sequences as hierarchies of discrete codes, then generates those representations from coarse concepts to fine details.

In experiments with models of comparable scale, NCM generated text 10–31 times faster than strong diffusion language models (another alternative to autoregressive LLMs) while achieving comparable or better quality. It also required 4–5 times fewer training-token exposures on two language modeling benchmarks.

Why should you care about NCM as a developer? This new paradigm could reduce inference costs, accelerate training, and improve performance on tasks that require consistency across an entire output. The architecture is still in its early stages, but its results challenge some of the assumptions behind today’s language models.

The autoregressive challenge

Most modern LLMs are designed to generate text one token at a time (aka next-token prediction  or NTP). Given a prompt, the model predicts the next token, appends it to the sequence, and uses the updated context to predict the following token.

This approach has been remarkably successful, but it also has its limitations. Every new token depends on all the previous tokens. This dependency creates complications and creates computational and memory bottlenecks as the sequence grows longer.

Another challenge is global planning and revision. When an autoregressive model generates a token, it cannot go back and change it within the same generation pass. So, if the model makes an incorrect decision early in its reasoning, the tokens that come after it must build on that decision or it has to generate extra tokens to correct its course. Techniques such as chain-of-thought reasoning and self-correction can help models plan and revise their responses, but they require additional computation and make the answer sequence longer.

One alternative to autoregressive LLMs is diffusion language models (DLMs). Instead of producing one token at a time, DLMs start with a sequence of masked or noisy representations and progressively refine multiple positions in parallel. They use the same mechanism employed in image generation models.

However, diffusion models also have limitations. They often require many refinement steps to make sure the response quality is acceptable, and each step processes the entire sequence. Reducing the number of steps can degrade performance. Strong diffusion models also require substantial training budgets.

How Next-Concept Modeling works

NCM approaches these challenges by changing the unit of prediction. Instead of generating its response one token at a time, it generates learned representations at progressively finer levels of detail.

The main idea behind NCM is to represent text as a hierarchy of discrete codes. The earliest levels contain a small number of codes that capture broad information about the text. Later levels contain more codes that add increasingly fine details to the response.

Say you want to write an article. You might start with a general outline that establishes the main ideas and structure at a high level. You then develop the sections and refine individual sentences until the article is complete.

NCM follows a similar coarse-to-fine process (although its intermediate representations are numerical codes rather than human-readable outlines). Note that the term “concept” in NCM refers to the information captured by these learned representations, not necessarily explicit ideas that humans can interpret.

NCM uses a two-stage training process to learn these representations and generate new text.

In the first stage, the researchers train a variational autoencoder model that learns to compress a text sequence into multiple levels of discrete codes and reconstruct the original text from them.

Each level represents information that earlier levels have not fully captured. The initial levels provide a rough representation of the sequence, while subsequent levels progressively correct the remaining differences between the reconstruction and the original text.

During training, the researchers sometimes remove the finer levels and require the decoder to reconstruct the text from the remaining codes. This technique, called “nested dropout,” encourages the earlier levels to learn useful information instead of relying entirely on the finer representations.

Once the autoencoder has learned the hierarchy, its parameters are frozen.

In the second stage, researchers train a separate model, called the “concept transformer,” to predict the discrete codes at different levels. During training, the transformer learns to predict all levels in a single forward pass. The researchers also introduce errors into some of the input codes to make the model more resilient to mistakes during generation.

At inference time, the process changes. The transformer starts by generating the first, coarse level. It then uses the accumulated representation to predict the next level, continuing until the hierarchy is complete.

One thing that distinguishes NCM is that the model predicts all codes within each level in parallel. It still generates levels sequentially, but the number of levels is much smaller than the number of tokens in the final output. Moreover, the early levels contain fewer codes, reducing the computational cost of those steps.

After generating all levels, the autoencoder’s decoder converts the completed representation into text.

This architecture enables NCM to progressively refine its representation of an entire output without committing to individual text tokens from the beginning.

NCM in action

The researchers evaluated NCM on language generation, translation, summarization, and Sudoku. Most experiments used models with around 100 million parameters, compared against autoregressive, diffusion, and flow-based models of similar scale.

One of the main findings was the improvement in generation efficiency.

On the One Billion Word (LM1B) benchmark, NCM achieved better generative perplexity than DUO, one of the strongest diffusion baselines, while generating tokens 21 times faster. On OpenWebText, it achieved comparable quality with a 10-fold speedup.

Hermann Kumbong, co-author of the paper, told TechTalks that the team has also observed speedups over conventional autoregressive models. “NCM is 6x to 10x faster than strong autoregressive baselines and the speedup increases as we scale the sequence length,” he said. 

Generative perplexity measures how plausible generated text appears to a separate language model. The researchers also measured output diversity to ensure that the improvements did not come from repeatedly producing a narrow range of predictable text.

NCM also showed advantages in training efficiency. On LM1B, the autoencoder and concept transformer together processed 13.36 billion training-token positions, compared with 66 billion for the diffusion baselines. On OpenWebText, NCM processed approximately four times fewer tokens.

An interesting example of NCM’s generation process appears in a German-to-English translation experiment.

The researchers decoded the model’s intermediate representations after each of its 11 generation steps. The first output was fragmented and contained repeated words. By the sixth step, it had identified much of the sentence’s meaning but still had missing and repeated terms. In later steps, the model corrected these problems and produced a coherent translation.

The researchers also tested NCM on Sudoku, which is a useful test of whether a model can produce globally consistent outputs. On hard puzzles, NCM solved 86.1% of boards, compared with 81.3% for the strongest competing model. It also required fewer model evaluations.

In one example, the researchers showed how NCM gradually refined a Sudoku board across its hierarchy, progressing from mostly incorrect cells to a completely valid solution. Kumbong expects the approach to provide similar advantages in more complex reasoning tasks, including mathematical problem-solving, coding, and multi-step planning, although the team has yet to test this hypothesis.

One possibility is that the hierarchy could provide an alternative to generating long chains of reasoning tokens. Kumbong describes the coarse-to-fine process as a potential way of “thinking step by step,” allowing the model to refine its internal representations before producing the final answer. Whether this approach can match the capabilities of existing reasoning models remains an open question.

Another finding concerns how developers can spend the computation saved through faster inference.

In the summarization experiments, NCM achieved a ROUGE-L score of 28.0 with standard generation. The researchers then used a technique called “minimum Bayes risk decoding,” which generates multiple candidate summaries and selects the one that has the highest average similarity to the others.

This creates an interesting tradeoff for AI applications. Instead of allocating all the efficiency gains to reducing latency, developers could spend some of the saved inference budget on generating and evaluating multiple responses.

Kumbong also sees potential for NCM in locally deployed AI models, where latency is particularly important. Its parallel generation approach could be useful for applications running on devices with limited computing resources, although the researchers have yet to establish its performance on consumer and edge hardware.

Availability and future directions

The researchers have released the NCM code and evaluation tools on GitHub, along with pretrained checkpoints on Hugging Face. 

However, NCM remains an experimental architecture. Most evaluations involve models with around 100 million parameters. The researchers also tested larger generators up to 725 million parameters and observed improvements in generation quality. Kumbong expects these gains to continue at larger scales and does not currently foresee architectural bottlenecks that would prevent further improvements. “We expect the models to improve as we scale them,” Kumbong said. “We had a preliminary scaling in our paper and expect that as we scale more the quality will improve… I think bigger models should make it easier instead.”

There are other limitations. NCM currently uses a fixed number of hierarchy levels and code lengths, regardless of the complexity of the task. The autoencoder and concept transformer are also trained separately, which can limit how well their objectives align. The researchers report occasional artifacts in generated text.

The team is already working on a second version of the architecture. “We are currently working on a V2 expected to solve many of these tokenization problems,” Kumbong said. 

For now, NCM is not a drop-in replacement for established LLM architectures. However, Kumbong sees opportunities to integrate it into existing systems, including speculative decoding, where a faster model proposes tokens for a larger model to verify. This could provide a path toward practical applications while the researchers work on scaling and improving the architecture.

In the longer term, the team has more ambitious plans. “Our goal is to make NCM the leading architecture for text generation,” Kumbong said.