Hugging Face Datasets Library

The Hugging Face datasets library loads, processes and shares data for machine learning. It reads thousands of datasets straight from the Hub, works with your own CSV, JSON or Parquet files, and stores data in Apache Arrow format so even very large datasets can be processed without loading everything into RAM.

Loading a Dataset from the Hub

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb")
print(ds)

DatasetDict({
    train: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    test: Dataset({
        features: ['text', 'label'],
        num_rows: 25000
    })
    unsupervised: Dataset({
        features: ['text', 'label'],
        num_rows: 50000
    })
})

A DatasetDict holds named splits. Each split is a Dataset, which behaves much like a list of dictionaries.

Inspecting Rows

train = ds["train"]
print(train[0]["label"], train[0]["text"][:60])
print(train.features["label"])

0 I rented I AM CURIOUS-YELLOW from my video store because of
ClassLabel(names=['neg', 'pos'])

Loading Your Own Files

from datasets import load_dataset

ds = load_dataset("csv", data_files={"train": "reviews_train.csv", "test": "reviews_test.csv"})
# or a single file, then split it yourself
ds = load_dataset("json", data_files="tickets.jsonl")["train"].train_test_split(test_size=0.2, seed=42)
print(ds)

DatasetDict({
    train: Dataset({ features: [...], num_rows: 800 })
    test: Dataset({ features: [...], num_rows: 200 })
})
(sample — depends on your file)

filter, map and select

filter keeps rows that match a condition. map applies a function to every row and can add new columns. batched=True passes many rows at once, which is dramatically faster for tokenization.

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")
small = ds["train"].shuffle(seed=0).select(range(2000))
small = small.filter(lambda row: len(row["text"]) > 50)

def tokenize(batch):
    return tok(batch["text"], truncation=True, max_length=256)

tokenized = small.map(tokenize, batched=True, remove_columns=["text"])
print(tokenized.column_names)

['label', 'input_ids', 'attention_mask']

Results of map are cached on disk, so running the same code again is instant.

Streaming Huge Datasets

Some datasets are hundreds of gigabytes. streaming=True reads rows on demand without downloading everything:

stream = load_dataset("HuggingFaceFW/fineweb", split="train", streaming=True)
for row in stream.take(2):
    print(row["text"][:50])

(two short text snippets from the web corpus — sample output)

Useful Dataset Methods

• ds.to_pandas() converts a split to a pandas DataFrame.
• ds.rename_column("old", "new") and ds.remove_columns([...]) tidy columns.
• ds.set_format("torch") returns PyTorch tensors when indexing.
• ds.push_to_hub("user/name") shares your dataset on the Hub.

Frequently asked questions

How do I load a CSV file with Hugging Face datasets?

Call load_dataset("csv", data_files="file.csv"). You can pass a dictionary of files to create several splits at once.

What does batched=True do in datasets map?

It passes a batch of rows (a dictionary of lists) to your function instead of one row at a time, which makes fast tokenizers much more efficient.

Can Hugging Face datasets handle data bigger than RAM?

Yes. Data is stored as memory-mapped Arrow files, and streaming mode can iterate over remote datasets without downloading them fully.