Inference Perf came out of the Kubernetes wg-serving effort to standardise how inference benchmarks and their metrics are reported, so numbers from different model servers can be compared apples-to-apples. A run is described in YAML (or CLI overrides) covering the server, the data source, and a multi-stage load schedule, then executed by a multi-process generator.
About
Inference Perf came out of the Kubernetes wg-serving effort to standardise how inference benchmarks and their metrics are reported, so numbers from different model servers can be compared apples-to-apples. A run is described in YAML (or CLI overrides) covering the server, the data source, and a multi-stage load schedule, then executed by a multi-process generator the project reports as scaling past 10k QPS, with automatic saturation sweeps to find where a deployment breaks. Reported metrics are the inference-specific ones — time to first token, time per output token, inter-token latency, input/output token throughput, and goodput against your own SLO constraints — with charts generated for QPS versus latency and throughput. Workloads come from real datasets such as ShareGPT, CNN DailyMail, Billsum, and VisionArena, from synthetic distributions with exact input/output shapes, or from replayed traces including OpenTelemetry traces for agentic call trees; shared-prefix and multi-turn chat scenarios exercise KV cache reuse. vLLM, SGLang, and TGI are verified with server-side aggregate and time-series metrics scraped alongside the client view, and any OpenAI-compatible endpoint works. It installs from PyPI, ships a container image, and has a manifest set for running benchmarks as Kubernetes jobs.
Best for
Token-level latency and throughput benchmarks against LLM inference endpoints
Teams that prefer open-source licensing and self-hosting options
Groups comfortable scripting in Python, YAML
Not the best fit when
Teams unwilling to operate generators, results storage, and scaling
Architecture and concurrency
Inference Perf is commonly operated as self-hosted software with scenario authoring in Python, YAML. Concurrency and distribution depend on how you size workers or injectors.
Specs
Scripting
PythonYAML
Protocols
HTTPHTTPS
Operating systems
LinuxmacOS
Tags
aillmpythonkubernetescli
CATALOG LAST VERIFIED · 2026-08-12
Frequently asked
ANSWER-SHAPED
Who is Inference Perf for?
Inference Perf is best for teams benchmarking LLM and generative AI endpoints on token latency and throughput. It is open source, so teams can self-host and extend it. Vendor: Kubernetes SIGs. Deployment model: self-hosted. Free; Apache License 2.0. Confirm details against the official site and a short proof-of-concept in your environment.
How does Inference Perf differ from I am speed?
Compared with I am speed, deployment is self-hosted versus hybrid for I am speed; scripting centers on Python, YAML rather than None. Inference Perf lists protocols HTTP, HTTPS; I am speed lists HTTPS, SSE. Status is active versus active. Use the Test Rig to compare full specs side by side before a proof-of-concept.
What scripting language and concurrency model does Inference Perf use?
Inference Perf uses Python, YAML for scripting or scenario definition. Runtime model: code-first scenarios where virtual users execute scripted behavior under load. OS support: Linux, macOS. Protocols: HTTP, HTTPS. Match these to how your team already authors tests and which systems you must drive under load.
What is the cloud or enterprise path for Inference Perf?
Inference Perf is primarily self-hosted; teams typically run generators on their own machines, CI runners, or private cloud. Plan for generator capacity, network access to targets, and result storage yourself. As open source, commercial support or hosted siblings may exist separately. Check the official site. Pricing note: Free; Apache License 2.0.
What are known limitations of Inference Perf?
Inference Perf limitations to weigh: self-hosted tools require you to provision, scale, and observe load generators yourself. Status: active. Always validate against your target protocols and scale with a proof-of-concept. Confirm details against the official site and a short proof-of-concept in your environment.