Hugging Face Inference Providers and Endpoints

Hugging Face inference services let you call models over HTTP without managing GPUs yourself. There are two main options: Inference Providers, a pay-as-you-go gateway to many hosted models served by Hugging Face and partner companies, and Inference Endpoints, dedicated machines that run a model of your choice just for you.

Calling a Model with InferenceClient

The InferenceClient class in huggingface_hub uses your HF_TOKEN and routes the request to an available provider. Its chat API mirrors the familiar OpenAI style:

from huggingface_hub import InferenceClient

client = InferenceClient()   # reads HF_TOKEN from the environment
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Name two uses of embeddings."}],
    max_tokens=80,
)
print(resp.choices[0].message.content)
1. Semantic search: finding documents with similar meaning.
2. Recommendations: suggesting items similar to ones a user liked.
(sample output — generated text varies)

Which models are available, and from which providers, changes often. The model page on the Hub shows an "Inference Providers" panel listing current options and prices. You can force a provider with InferenceClient(provider="..."), or let the default automatic routing choose.

Using the OpenAI SDK

Inference Providers also expose an OpenAI-compatible router, so existing OpenAI client code can switch to open models by changing the base URL:

import os
from openai import OpenAI

client = OpenAI(base_url="https://router.huggingface.co/v1", api_key=os.environ["HF_TOKEN"])
r = client.chat.completions.create(
    model="Qwen/Qwen2.5-7B-Instruct",
    messages=[{"role": "user", "content": "Say hi in French."}],
)
print(r.choices[0].message.content)
Bonjour !   (sample output)

Other Tasks

vec = client.feature_extraction("Hello world", model="sentence-transformers/all-MiniLM-L6-v2")
print(vec.shape)
(384,)

InferenceClient also has methods such as text_to_image, automatic_speech_recognition and summarization. Each returns Python objects (NumPy arrays, PIL images, dataclasses) rather than raw JSON.

Inference Endpoints

When you need guaranteed capacity, a private or fine-tuned model, or data that must stay in a specific region, create an Inference Endpoint from the model page or the web dashboard. You pick the cloud, region and GPU, and pay per hour while it runs. Endpoints can scale to zero when idle to save money, and they use serving engines like vLLM or Text Generation Inference under the hood.

Choosing an Option

  • Prototyping or low traffic: Inference Providers, pay per request.
  • Steady production traffic or custom models: Inference Endpoints.
  • Full control, offline or strict privacy: self-host with transformers, vLLM, llama.cpp or Ollama.
  • Check free monthly credits and current pricing on your account billing page.

Frequently asked questions

Is the Hugging Face Inference API free?

Accounts include a small amount of monthly credit for Inference Providers; beyond that you pay per use. PRO and team plans include more credit. Check the billing page for current numbers.

What is the difference between Inference Providers and Inference Endpoints?

Inference Providers is a shared, serverless gateway billed per request. Inference Endpoints are dedicated machines you configure and pay for by the hour.

Can I use the OpenAI Python SDK with Hugging Face models?

Yes. Point base_url at https://router.huggingface.co/v1 and use your Hugging Face token as the API key.