Same GPU. Same model. One optimization enabled.

If you only ran the standard gateway-direct test, you would see identical numbers and have no reason to prefer one configuration over the other. The agent end-to-end tier is what shows the actual difference.

Experiment Design: Three Optimization Paths

I tested three vLLM server configurations (or “arms”) on a single Lambda A10G-24GB using Qwen2.5-3B-Instruct. The goal was to isolate how different infrastructure flags handle agentic workloads.

Prompt Growth in Agentic Loops

Unlike a simple chatbot, an agent maintains a running transcript of every tool call and search result. By the third turn of a conversation, the prompt often exceeds 2,000 tokens.

In a default setup, the GPU reads this entire 2,000-token prompt in one uninterrupted block. While it’s doing that, every other request in the system is forced to wait — a scheduling bottleneck that compounds under concurrency.

This is the same proxy-to-model path from Part 1, now under concurrent load, the growing transcript is what turns a latency problem into a scheduling problem.

Technical Context: The Prefill Phase

LLM inference happens in two stages:

• Prefill: The model reads and processes the prompt. This is computationally expensive.

• Generation: The model writes the response token-by-token.

Chunked prefill forces the system to process that long prompt in pieces, letting other requests run in between. This keeps concurrent requests from stalling behind a long prefill block.

Side note: Direction this points toward, separating prefill and decode onto different hardware entirely — disaggregated prefill/decode. Different phases, different GPU requirements. Worth knowing this exists if you’re thinking about where inference optimization goes next.

Setting Targets: SLOs and Performance Hypotheses

I defined these Service Level Objectives (SLOs) before looking at the data. An SLO written after the fact isn’t a goal — it’s just a description of what happened.

Pre-Experiment Predictions

• Chunked prefill should reduce tail latency: Agents send long prompts but give short answers — exactly the scheduling pattern this flag is designed for.

• Speculative decoding is unlikely to help here: It works best on predictable outputs. Agent tool-call responses are JSON with variable structure — low acceptance rates for the draft model, which adds overhead rather than saving time.

• Prefix caching should activate consistently: Every agent starts with the same ~900-token system prompt. vLLM should reuse that computed result instead of recalculating it every time.

Functional Validation: The Golden Set

Speed is irrelevant if the agent is producing wrong answers. Before running load tests, I ran a “Golden Set” of 10 fixed queries to verify accuracy.

The one failure (query gs-010) was a model-level limitation where the 3B model answered from its own knowledge instead of calling the search tool.

Tiered Benchmarking: Gateway vs. End-to-End

When concurrency increased, the infrastructure differences became visible.

Tier 1: Gateway-Direct

This is what standard benchmarks measure and showing a null result here is the point, not a limitation.

16 concurrent users each sent a single isolated call. p95 is the response time the slowest 5% of requests experience.

Finding: Baseline and Chunked Prefill were identical at 2.4s (p95).

If you stopped here, you would conclude the optimization made no difference. That conclusion would be wrong — gateway-direct measures a single isolated call, not the multi-call sequence an agent actually runs.

Tier 2: Agent End-to-End

4 concurrent users each ran full agent tasks (3–5 sequential model calls).

Finding: Baseline latency reached 15.0s, while Chunked Prefill held at 5.8s.

The “tail” latency the slowest 5% of requests is 3× better with chunked prefill. By forcing the scheduler to share resources in between chunks, long-prompt requests add 9+ seconds of tail latency in the baseline that chunked prefill avoids.

Speculative Decoding and Unit Economics

Rejection Overhead

Speculative decoding came in at 14.0s p95. Slightly better than baseline (15.0s) but 2.4× behind chunked prefill (5.8s), and with a 15% throughput drop at the gateway tier. Because agent responses (JSON and tool calls) are structured but variable, the draft model kept guessing wrong. Each wrong guess forced the main model to redo the work, adding overhead instead of saving time.

Cost Analysis

The cost per task appeared high ($0.036), but this was due to the short 15-minute test window being dominated by server setup time. At a sustained production speed, the cost drops to $0.003 per task easily beating the $0.01 target.

Summary

• Chunked prefill handles the tail: It didn’t make the GPU faster — it made the scheduler fairer.

• Speculative decoding is workload-dependent: It works well for predictable outputs — code completion, templated generation. For variable agent responses, it can be a net negative.

• Benchmark the loop: Measuring the model alone gives you an incomplete picture. To understand agent performance, you must measure the full back-and-forth.

Next up: Part 3 covers the observability layer — the four dashboards that made these results interpretable.

GitHub repo

Most inference benchmarks give you API-level numbers. They don’t show you what’s happening inside the stack when your agent is actually running.

Is the prefix cache being hit? Is a long prompt blocking shorter requests in the scheduler queue? Is the optimization you enabled actually doing anything for your specific workload?

Those questions only have answers if the instrumentation is already in place. I built an end-to-end instrumented inference stack, ran controlled experiments against it, and wrote up what I found.

First in a three-part series on building an observable LLM inference stack, running controlled experiments, and understanding what the numbers actually mean for agentic workloads.

What I Built

The core is an inference gateway—it accepts standard OpenAI-compatible API calls and routes them to different backends: vLLM on a Lambda GPU, Modal’s serverless endpoint, or a local Ollama instance. Every layer emits metrics so you can see exactly what’s happening at each hop.

Figure 1: The full request path — Nginx → FastAPI gateway → SSH tunnel → Lambda vLLM on an A10G GPU. LangSmith captures agent traces. Prometheus collects metrics from the gateway and vLLM engine. Grafana visualizes all of it.

Project Structure

The project is intentionally modular to allow for “pluggable” observability:

gateway/        → Request routing, metrics, logging
agent/          → LangGraph ReAct agent + test runner
ui/             → Chainlit chat interface
observability/  → Prometheus, Grafana, OTel Collector
deployments/    → Lambda and Modal deployment scripts
experiments/    → vLLM server configurations
loadtest/       → Load testing scripts
data/samples/   → Experiment results — committed to the repo

This isn’t a production system. It's a testbed for running controlled experiments — change one variable, run the same tests, read the same dashboards, and understand the tradeoff.

Following a Request Through the Stack

The clearest way to explain this architecture is to trace a single request end-to-end.

01. The user sends a message

The Chainlit UI sends the prompt to the agent. At this exact moment, LangSmith begins recording the session automatically—no code changes required, just an environment variable. Every subsequent tool call and LLM response is captured in this trace.

02. The agent decides what to do

This is a LangGraph ReAct agent. Unlike a standard chatbot, it doesn’t call the LLM once and exit. It follows an iterative loop:

1. Call LLM to select a tool.

2. Execute the tool.

3. Send results back to the LLM.

4. Repeat until a final answer is reached.

The Context Load: One user message typically triggers 3 to 5 separate LLM calls. By the third turn, the context can exceed 2,000 tokens before the model even generates its first word. This is why agentic workloads are an entirely different beast for inference infrastructure compared to simple chat.

03. The gateway receives the request

The FastAPI gateway intercepts a standard POST /v1/chat/completions. Before routing, it performs three critical tasks:

• Header Inspection: Reads the X-Technique header (e.g., baseline, chunked_prefill).

• Telemetry: Records a Prometheus metric tagged with that specific technique.

• Logging: Writes a structured entry for later auditing.

The Design Choice: By using the X-Technique header, a single Grafana dashboard shows all three experiment arms side-by-side without changing a single line of observability code between runs.

04. The backend runs inference

The request is routed to one of three backends depending on the experiment configuration:

• Lambda A10G: Primary engine for metrics.

• Ollama: Local development at zero cost.

• Modal: On-demand serverless scaling.

The inference server handles the heavy lifting: scheduling, memory management, and KV cache orchestration. Its internal metrics are the “heartbeat” of this observability layer.

05. The GPU does the work

Finally, we reach the silicon. This is where tensor operations and token generation happen. The vLLM metrics endpoint allows us to see into the black box—specifically scheduler behavior and cache statistics.

Note: While we track engine metrics, kernel-level profiling (like CUDA stream analysis) is intentionally out of scope for this series.

Modularity by Design

Adding a new backend is a single-file change. The gateway uses a BaseBackend interface, allowing VLLMBackend or SGLangBackend to be implemented independently.

backends:
  vllm-lambda-baseline:
    url: "<http://localhost>:${BASELINE_PORT}"
    gpu: "a10g-24gb"
    server_profile: "baseline"
    spec_decoding: false

The practical reason for this? A controlled experiment requires exactly one variable to change. Swapping a backend shouldn’t touch routing logic, and changing a server flag shouldn’t require a new observability setup.

The Agent Workload

The agent has two tools: web search through Tavily, and a task list stored in a local JSON file.

Figure 2: The Chainlit interface. A response like this comes from 3–4 sequential LLM calls through the gateway — tool selection, tool execution, reading the result, generating the final answer.

What makes this interesting for infrastructure is concurrency. When several users run agent tasks simultaneously, each generates multiple sequential LLM calls with long contexts. This creates the exact conditions where scheduler decisions become visible in the data.

Figure 3: A LangSmith trace — agent → ChatOpenAI → tool decision → manage_tasks → agent → ChatOpenAI. Each row is a separate step with its own timing and token count.

What’s Coming

• Part 2 covers the three-arm experiment: baseline, chunked prefill, and speculative decoding. We’ll walk through why the benchmark tier matters more than the optimization.

• Part 3 walks through the dashboard layers with real experiment data, including the internal vLLM metrics that confirmed our prefix caching hypotheses.

The repo is at github.com/karthikkaiplody/agent-infer-stack. RUNBOOK.md has the full setup instructions.

Part 2 drops next week.