Hugging Face Text Generation with Open LLMs

Text generation with Hugging Face lets you run open-weight large language models (LLMs) such as Qwen, Llama, Mistral, Gemma and Phi on your own hardware. In this lesson you will chat with a small instruct model, learn how chat templates work, and tune the sampling settings that control creativity.

Chatting with the Pipeline

The text-generation pipeline accepts a list of chat messages directly. A 0.5B-parameter model is small enough for a laptop CPU, so it is perfect for learning:

from transformers import pipeline

chat = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")
messages = [
    {"role": "system", "content": "You are a concise assistant."},
    {"role": "user", "content": "Explain a GPU in one sentence."},
]
result = chat(messages, max_new_tokens=60)
print(result[0]["generated_text"][-1]["content"])

A GPU is a processor with thousands of small cores that performs many calculations in parallel, which makes it ideal for graphics and machine learning.
(sample output — generated text varies)

generated_text contains the whole conversation, including the new assistant message at the end, so [-1]["content"] picks out the reply.

Chat Templates

Every chat model was trained with its own special format for roles and turns. apply_chat_template builds that exact string for you, so you never have to memorize tokens like <|im_start|>.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
prompt = tok.apply_chat_template(
    [{"role": "user", "content": "Hi!"}],
    tokenize=False, add_generation_prompt=True,
)
print(prompt)

<|im_start|>system
You are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>
<|im_start|>user
Hi!<|im_end|>
<|im_start|>assistant

Using model.generate Directly

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

name = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="auto", device_map="auto")

msgs = [{"role": "user", "content": "Give me three names for a cat."}]
inputs = tok.apply_chat_template(msgs, add_generation_prompt=True,
                                 return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=50, do_sample=True,
                     temperature=0.7, top_p=0.9)
new_tokens = out[0][inputs["input_ids"].shape[1]:]
print(tok.decode(new_tokens, skip_special_tokens=True))

1. Whiskers
2. Luna
3. Biscuit
(sample output — generated text varies)

generate returns the prompt plus the new tokens, so we slice off the prompt length before decoding.

Sampling Settings Explained

• max_new_tokens: hard limit on how many tokens to generate.
• do_sample=False: greedy decoding, always picks the most likely token (deterministic).
• temperature: below 1 makes output more focused, above 1 more random.
• top_p: only sample from the smallest set of tokens whose probability adds up to p.
• repetition_penalty: values like 1.1 discourage loops of repeated phrases.

Streaming Tokens

For a chat-like feel, print tokens as they are produced with TextStreamer:

from transformers import TextStreamer

streamer = TextStreamer(tok, skip_prompt=True)
model.generate(**inputs, max_new_tokens=50, streamer=streamer)

1. Mochi
2. Pepper
3. Shadow
(tokens appear one by one — sample output)

Picking a Model Size

A rough memory guide: each billion parameters needs about 2 GB in bfloat16, so a 7B model needs around 14-16 GB of GPU memory. Quantization (covered later) cuts this to roughly a quarter. Start small, check quality, and only move up if you need to.

Frequently asked questions

How do I run an LLM locally with Hugging Face?

Install transformers and torch, then load an instruct model with pipeline("text-generation", model=...) or AutoModelForCausalLM, and pass chat messages. Small models run on CPU; larger ones need a GPU or quantization.

What is apply_chat_template?

A tokenizer method that formats a list of role/content messages into the exact prompt format the chat model was trained on, including special tokens.

Why does my model repeat itself?

Greedy decoding with small models often loops. Enable sampling with a moderate temperature, add a repetition_penalty, and make sure you use the chat template for instruct models.

What does temperature do in text generation?

It scales the probabilities before sampling. Low values make the output more predictable; high values make it more varied and creative but less reliable.