#
Lighteval

🤗 Lighteval is your all-in-one toolkit for evaluating Large Language Models (LLMs) across multiple backends with ease. Dive deep into your model’s performance by saving and exploring detailed, sample-by-sample results to debug and see how your models stack up.

##
Key Features

###
🚀 Multi-Backend Support

Evaluate your models using the most popular and efficient inference backends:

- `eval`: Use inspect-ai as backend to evaluate and inspect your models ! (prefered way)
- `transformers`: Evaluate models on CPU or one or more GPUs using 🤗 Accelerate
- `nanotron`: Evaluate models in distributed settings using ⚡️ Nanotron
- `vllm`: Evaluate models on one or more GPUs using 🚀 VLLM
- `custom`: Evaluate custom models (can be anything)
- `sglang`: Evaluate models using SGLang as backend
- `inference-endpoint`: Evaluate models using Hugging Face’s Inference Endpoints API
- `tgi`: Evaluate models using 🔗 Text Generation Inference running locally
- `litellm`: Evaluate models on any compatible API using LiteLLM
- `inference-providers`: Evaluate models using HuggingFace’s inference providers as backend**: Distributed training and evaluation

###
📊 Comprehensive Evaluation

- Extensive Task Library: 1000s pre-built evaluation tasks
- Custom Task Creation: Build your own evaluation tasks
- Flexible Metrics: Support for custom metrics and scoring
- Detailed Analysis: Sample-by-sample results for deep insights

###
🔧 Easy Customization

Customization at your fingertips: create new tasks, metrics or model tailored to your needs, or browse all our existing tasks and metrics.

###
☁️ Seamless Integration

Seamlessly experiment, benchmark, and store your results on the Hugging Face Hub, S3, or locally.

##
Quick Start

###
Installation

    pip install lighteval

###
Basic Usage

####
Find a task

####
Run your benchmark and push details to the hub

    lighteval eval "hf-inference-providers/openai/gpt-oss-20b" \
        gpqa:diamond \
        --bundle-dir gpt-oss-bundle \
        --repo-id OpenEvals/evals

Resulting Space:

Update on GitHub