How LLMs Work for Prompt Engineering

Knowing how LLMs work makes prompt engineering much easier. You do not need the maths, just a few ideas: tokens, next-token prediction, context windows and sampling settings.

Tokens

Models read and write tokens, which are chunks of text, often a word or part of a word. As a rough guide, 1 token is about 4 characters or three-quarters of an English word. Prices, speed and length limits are all measured in tokens.

"Prompt engineering is fun!"

Approximate tokens: ["Prompt", " engineering", " is", " fun", "!"]  -> 5 tokens
(Exact splits depend on each model's tokenizer.)

Next-Token Prediction

An LLM generates text one token at a time, each time choosing a likely next token given everything before it. This is why the start of a prompt shapes everything: the model continues in the style, language and format it sees.

Context Window

The context window is the maximum number of tokens the model can consider at once, including your prompt, any documents, the conversation history and its own reply. At the time of writing (2026) windows range from a few thousand tokens on small local models to hundreds of thousands or more on large hosted ones. Content outside the window is simply invisible to the model.

Temperature and Sampling

• Temperature 0 - 0.3: focused and repeatable; good for extraction, classification, code.
• Temperature 0.7 - 1.0: more varied; good for brainstorming and creative writing.
• top_p: limits sampling to the most probable tokens.
• max tokens: caps reply length (the reply may be cut off mid-sentence).

Why Models Hallucinate

Because a model produces plausible text, it can state wrong facts confidently, especially about niche topics, recent events or exact numbers. Prompts reduce this by supplying the facts and allowing "I don't know".

Answer using only the text below. If the answer is not in the text, reply "Not stated."

Text: The library opens at 9am on weekdays.
Question: What time does the library open on Sunday?

Sample response:
Not stated.

Frequently asked questions

How many words is 1000 tokens?

Roughly 750 English words, though it varies by language and tokenizer.

What temperature should I use?

Use a low temperature (0-0.3) for factual or structured tasks and a higher one (0.7-1) for creative tasks.

Does the model remember past conversations?

Not by itself. A chat app resends earlier messages inside the context window; anything not sent is unknown to the model.