LM Evaluation Harness

• You are writing an academic paper and need standardized benchmark scores with consistent few-shot prompts.
• You want to compare multiple models (e.g., Llama-2-7b vs. Mistral-7B) on the same task suite and produce a side-by-side table.
• You are training a model and want to track benchmark scores across checkpoints to catch regressions early.
• You need to validate a fine-tuned model against public leaderboard numbers before a release.

• You need instruction-following or chat quality evaluation — consider AlpacaEval or MT-Bench instead.
• You need broader coverage of fairness, calibration, or efficiency metrics — consider HELM.
• You are evaluating a highly domain-specific task with no existing harness task definition and lack the time to write a custom task.

Covers MMLU (57 subjects), GSM8K, HumanEval, TruthfulQA, HellaSwag, ARC, MBPP, WinoGrande, and many more. All tasks use standardized prompts so results are directly comparable across runs and teams.

Switching from the HuggingFace backend to vLLM cuts evaluation time by 5-10×. A full MMLU run on a 7B model drops from ~2 hours to ~15-20 minutes with tensor-parallel vLLM.

Provide a list of model IDs, run evaluations in batch, and generate a markdown comparison table with per-task scores. Supports any mix of HuggingFace hub models and local checkpoints.

Evaluate checkpoints at regular training steps using fast benchmarks (HellaSwag, GSM8K), save per-step JSON results, and plot learning curves to monitor model improvement over time.

Run 4-bit and 8-bit quantized models via load_in_4bit and load_in_8bit model args, reducing VRAM requirements from ~16 GB to ~8 GB for 7B models without changing the evaluation pipeline.

Supports --batch_size auto for optimal throughput detection, --device selection, CPU offloading via device_map=auto, and multi-GPU tensor parallelism through the vLLM backend.

Run 5-shot MMLU, GSM8K, and HellaSwag evaluations on your model using the same prompts as published papers and the Open LLM Leaderboard, producing directly comparable numbers.

Before releasing a fine-tuned checkpoint, run the standard benchmark suite to confirm quality thresholds are met and generate a results JSON suitable for model card reporting.

Integrate lm_eval calls into your training loop to evaluate lightweight benchmarks every N steps, catch regressions early, and plot accuracy curves across the training run.

Evaluate a list of candidate models (e.g., Llama-2-7b, Mistral-7B, Phi-2) on the same task suite in batch and output a markdown comparison table for team review.