Skip to content
Back to Understanding models

Understanding models · Chapter 3 of 8

Choosing the next token

A model scores every possible next token, turns the scores into chances and picks one at random; temperature, top-k and top-p change the pick, and base, instruct and reasoning models are trained to write differently.

14 minutes0 of 4 goals done

Goals

  • See the 20 likeliest next tokens for a prompt.To do
  • Move temperature and top-p, and pick again.To do
  • Ask a base model and an instruct model the same question.To do
  • Pass the check.To do

Step 1 of 4: The read

One token at a time

In the last chapter, text became tokens. A model writes the same way: it reads all the text so far, picks one next token, adds it to the text and starts again. A reply of 100 tokens takes 100 of these steps.

Scores become chances

At each step the model gives every token it knows a score, called a logit. SmolLM2-360M, the small open model in this chapter, knows 49,152 tokens, so it gives 49,152 scores. The scores are then turned into chances, also called probabilities, that add up to 100%. A higher score means a larger chance: one point more means about 2.7 times the chance.

A random pick, with settings

Then one token is picked at random, by those chances, like a draw from a bag where likely tokens have more tickets. So the same prompt can get different replies. Three settings change the draw. Temperature divides every score before it becomes a chance: below 1 the likely tokens get even likelier, above 1 the chances even out. Top-k keeps only the k likeliest tokens. Top-p keeps the likeliest tokens until their chances add up to p, so a top-p of 0.9 drops the long tail of unlikely ones. Whatever is kept is scaled back up to 100%.

Base and instruct models

A base model has only learned to continue text, so a question may get more questions. An instruct model was trained further on conversations, written in a chat format that marks who is speaking. It answers, then writes an end token, a special token that means it is done.

Reasoning models

A reasoning model writes its working first, in the same token-by-token way, and then the answer. As of 2026-11-03, closed models such as Claude Opus 5.5 and GPT-6.1 Sol make reasoning a setting, and open ones such as DeepSeek-R1 and Olmo 3 7B Think ship as separate reasoning models.

In the visual

The numbers in the visual were recorded from SmolLM2-360M and its instruct version, both from Hugging Face. If your browser has WebGPU, a way for web pages to use the graphics card, you can also download the instruct model and run it live.

Step 2 of 4: Try it: Next-token sampler

Next-token sampler

Choose a prompt and read the chances, then shape them and pick, one token at a time. Last, ask the base and the instruct model the same question.

Mode:RealReal, recorded: every score, chance and answer comes from running SmolLM2-360M and SmolLM2-360M-Instruct on a computer on 2026-11-03, with Hugging Face's Transformers.js. The picks, temperature, top-k and top-p are computed in this page from those numbers, and the text you step through is the base model's own recorded run. The reasoning models come from the snapshot of 2026-11-03. The live run is real too, after an opt-in 204 MB download.

Step 3 of 4: Check

Check

The settings
Prompt: The capital of France is
Model:  SmolLM2-360M (base)

Next token   Chance
Paris        20.3%
the          19.6%
located      10.5%
situated      4.7%
called        3.7%
… 15 more, then all the rest

Temperature: 1
Top-k:       off
Top-p:       0.5
You pick the next token 100 times with these settings. What do you get?

Step 4 of 4: What you can do now

What you can do now

  • Explain how a model turns scores into chances and picks the next token.
  • Say what temperature, top-k and top-p do to the chances.
  • Tell a base model's output from an instruct model's, and say what a reasoning model adds.