Inference Perf came out of the Kubernetes wg-serving effort to standardise how inference benchmarks and their metrics are reported, so numbers from different model servers can be compared apples-to-apples. A run is described in YAML (or CLI overrides) covering the server, the data source, and a multi-stage load schedule, then executed by a multi-process generator.
Best for
Token-level latency and throughput benchmarks against LLM inference endpoints
Teams that prefer open-source licensing and self-hosting options
Groups comfortable scripting in Python, YAML
Not the best fit when
Teams unwilling to operate generators, results storage, and scaling
Architecture and concurrency
Inference Perf is commonly operated as self-hosted software with scenario authoring in Python, YAML. Concurrency and distribution depend on how you size workers or injectors.
Inference Perf is best for teams benchmarking LLM and generative AI endpoints on token latency and throughput. It is open source, so teams can self-host and extend it. Vendor: Kubernetes SIGs. Deployment model: self-hosted. Free; Apache License 2.0. Confirm details against the official site and a short proof-of-concept in your environment.
How does Inference Perf differ from I am speed?
Compared with I am speed, deployment is self-hosted versus hybrid for I am speed; scripting centers on Python, YAML rather than None. Inference Perf lists protocols HTTP, HTTPS; I am speed lists HTTPS, SSE. Status is active versus active. Use the Test Rig to compare full specs side by side before a proof-of-concept.
What scripting language and concurrency model does Inference Perf use?
Inference Perf uses Python, YAML for scripting or scenario definition. Runtime model: code-first scenarios where virtual users execute scripted behavior under load. OS support: Linux, macOS. Protocols: HTTP, HTTPS. Match these to how your team already authors tests and which systems you must drive under load.
What is the cloud or enterprise path for Inference Perf?
Inference Perf is primarily self-hosted; teams typically run generators on their own machines, CI runners, or private cloud. Plan for generator capacity, network access to targets, and result storage yourself. As open source, commercial support or hosted siblings may exist separately. Check the official site. Pricing note: Free; Apache License 2.0.
What are known limitations of Inference Perf?
Inference Perf limitations to weigh: self-hosted tools require you to provision, scale, and observe load generators yourself. Status: active. Always validate against your target protocols and scale with a proof-of-concept. Confirm details against the official site and a short proof-of-concept in your environment.