Artificial Analysis open‑sources a tool to time AI agents on your own laptop
AA-AgentPerf-Local replays real agent sessions against llama.cpp, vLLM, LM Studio or Ollama and reports tokens a second. First results cover DGX Spark, RTX 5090, Ryzen AI Halo and MacBook Pro M5 Pro.
Artificial Analysis announced AA-AgentPerf-Local on X, calling it an open-source inference testing tool for local AI models. The benchmarking firm says it replays real agent trajectories on laptop and workstation hardware, and it published a list of serving configurations to help people plan a local agent setup. It said initial results cover NVIDIA's DGX Spark and GeForce RTX 5090, AMD's Ryzen AI Halo and a MacBook Pro M5 Pro.
https://x.com/ArtificialAnlys/status/2105133181536243942
The GitHub README says the tool replays recorded agent conversations against any OpenAI-compatible server. It names llama.cpp, vLLM, SGLang, LM Studio and Ollama. It reports output tokens a second, median time to first token, and 95th-percentile first-token and turn times, saved as JSON. Users can point it at a running server or let it download a pinned model and start one.
Setup is a single command, `uv tool install agentperf-local`, followed by `agentperf-local`, which opens a text interface. The README says it runs on macOS, Linux and Windows under an Apache-2.0 licence, and that full runs need a 65,536-token context at batch size 1. It measures speed only and does not judge the quality of the model's output.
Artificial Analysis said the tool and leaderboard will soon cover more hardware and frameworks. The announcement post as retrieved did not include per-machine numbers, so the paper has not seen the leaderboard figures. The repository had 26 stars when the paper read it, and no outside reviewer has yet tested how well its replayed trajectories predict real agent workloads.