TOOL SIGNAL / ACTIVE

AIPerf

NVIDIA

AI/LLM Inference Open Source Self-hosted
Official site ↗ Source repo ↗
Pricing model
Free; Apache License 2.0.
First released
2025
Last verified
Answer

AIPerf is the benchmarking client in NVIDIA’s ai-dynamo stack: `aiperf profile` points at any OpenAI-compatible endpoint — vLLM, TGI, Ollama, NIM, or a hosted API — and reports time to first token, time to second token, request latency, inter-token latency, per-user and aggregate output token throughput, and sequence lengths as avg/min/max/p99/p90/p50/std, exported to CSV and.

About

AIPerf is the benchmarking client in NVIDIA’s ai-dynamo stack: `aiperf profile` points at any OpenAI-compatible endpoint — vLLM, TGI, Ollama, NIM, or a hosted API — and reports time to first token, time to second token, request latency, inter-token latency, per-user and aggregate output token throughput, and sequence lengths as avg/min/max/p99/p90/p50/std, exported to CSV and JSON alongside a live terminal dashboard (or headless output for CI). Load can be driven by fixed concurrency, request rate, request rate capped by max concurrency, or deterministic trace replay, with constant, Poisson, and gamma arrival patterns, gradual ramping, warmup phases, adaptive SLA-boundary discovery, request cancellation, and multi-URL load balancing. Ten services communicate over ZMQ so a single run can scale across processes, and a plugin system extends endpoints, datasets, transports, and metrics. Datasets include ShareGPT, MMStar for vision models, AIMO math traces, production captures from SageMaker and BurstGPT, and inline or custom formats; optional extras stream results to MLflow, OpenTelemetry, or Weights & Biases.

Best for

  • Token-level latency and throughput benchmarks against LLM inference endpoints
  • Teams that prefer open-source licensing and self-hosting options
  • Groups comfortable scripting in Python, YAML

Not the best fit when

  • Teams unwilling to operate generators, results storage, and scaling

Architecture and concurrency

AIPerf is commonly operated as self-hosted software with scenario authoring in Python, YAML. Concurrency and distribution depend on how you size workers or injectors.

Specs

Scripting
PythonYAML
Protocols
HTTPHTTPSSSE
Operating systems
LinuxmacOSWindows
Tags
aillmpythonclitui

CATALOG LAST VERIFIED · 2026-08-12

Frequently asked

ANSWER-SHAPED
Who is AIPerf for?

AIPerf is best for teams benchmarking LLM and generative AI endpoints on token latency and throughput. It is open source, so teams can self-host and extend it. Vendor: NVIDIA. Deployment model: self-hosted. Free; Apache License 2.0. Confirm details against the official site and a short proof-of-concept in your environment.

How does AIPerf differ from I am speed?

Compared with I am speed, deployment is self-hosted versus hybrid for I am speed; scripting centers on Python, YAML rather than None. AIPerf lists protocols HTTP, HTTPS, SSE; I am speed lists HTTPS, SSE. Status is active versus active. Use the Test Rig to compare full specs side by side before a proof-of-concept.

What scripting language and concurrency model does AIPerf use?

AIPerf uses Python, YAML for scripting or scenario definition. Runtime model: code-first scenarios where virtual users execute scripted behavior under load. OS support: Linux, macOS, Windows. Protocols: HTTP, HTTPS, SSE. Match these to how your team already authors tests and which systems you must drive under load.

What is the cloud or enterprise path for AIPerf?

AIPerf is primarily self-hosted; teams typically run generators on their own machines, CI runners, or private cloud. Plan for generator capacity, network access to targets, and result storage yourself. As open source, commercial support or hosted siblings may exist separately. Check the official site. Pricing note: Free; Apache License 2.0.

What are known limitations of AIPerf?

AIPerf limitations to weigh: self-hosted tools require you to provision, scale, and observe load generators yourself. Status: active. Always validate against your target protocols and scale with a proof-of-concept. Confirm details against the official site and a short proof-of-concept in your environment.