How to Run an LLM Cost Audit: Cut Your AI API Bill Without Hurting Quality

TL;DR: Most LLM bills are inflated by the same five things: calls nobody inventoried, one model used for every task, repeated context that isn't cached, wasted tokens, and no per-feature metering. Fix them in that order. Teams commonly cut a large share of their AI spend this way without users noticing a change in quality, as long as every change is checked against an eval set first.

This is the audit I run when AI spend starts growing faster than usage. I've also packaged it as a Claude Code agent skill you can run on your own repo: LLM Cost Audit skill.

Step 1: Inventory Every Call Site

You can't cut what you haven't found. Search the codebase for every SDK client, raw HTTP call to an inference endpoint and internal wrapper. For each call site, write down:

• Model and max_tokens
• Where the prompt comes from (static, templated, user input, retrieved documents)
• How often it runs: per request, per item in a loop, on a cron, on every keystroke
• Who sees the output: a user, or another step in a pipeline

Flag these right away, because they are where bills explode:

• LLM calls inside loops over collections
• Calls on every page load or keystroke
• Retries without backoff or without a cap

Step 2: Fix Model-Tier Mismatches

The biggest single saving is usually using the smallest model that passes your evals for each task. One DEFAULT_MODEL constant used everywhere is a red flag.

Make the choice explicit per task so it can be changed and tested one task at a time:

// One place that decides which model each task uses.
type Task = "classify" | "extract" | "summarize" | "plan" | "write";

const MODEL_FOR: Record<Task, string> = {
  classify: "small-fast-model",
  extract: "small-fast-model",
  summarize: "mid-model",
  plan: "large-model",
  write: "mid-model", // downgraded from large after it passed the eval set
};

export function modelFor(task: Task) {
  return MODEL_FOR[task];
}

Never downgrade blind. Run the cheaper model on your eval set, compare, and ship only if quality holds. (If you don't have an eval set yet, that's the first thing to build. See What Is an AI Product Engineer? for why.)

Step 3: Cache What Repeats

There are three kinds of caching, and most codebases use none of them.

Prompt caching. Long system prompts, few-shot examples and shared documents that open many requests can use your provider's prompt cache, which bills repeated prefix tokens at a discount. Put the stable parts first and the changing parts last so the prefix actually matches.

Response caching. Deterministic tasks (extraction, classification, anything at temperature 0 on the same input) can be cached in your own store, keyed on a hash of the normalized input:

import { createHash } from "node:crypto";

export async function cachedCall<T>(
  task: string,
  input: string,
  run: () => Promise<T>,
  ttlSeconds = 60 * 60 * 24,
): Promise<T> {
  const key = "llm:" + task + ":" + createHash("sha256").update(input.trim().toLowerCase()).digest("hex");
  const hit = await redis.get(key);
  if (hit) return JSON.parse(hit) as T;

  const result = await run();
  await redis.set(key, JSON.stringify(result), "EX", ttlSeconds);
  return result;
}

Negative caching. If an input reliably fails or gets refused, cache the failure for a short time so you don't pay for the same failure repeatedly.

Step 4: Remove Token Waste

• Send the section, not the document. Retrieve or slice the part the task needs.
• Bound conversation history. Multi-turn flows that resend everything grow without limit. Window or summarize older turns.
• Tighten output formats. Ask for compact JSON with short keys, no prose wrapper, and no reasoning text that nobody reads.
• Set realistic max_tokens and stop sequences so runaway outputs are cut off.
• Repair locally before retrying. If a JSON parse fails on a trailing comma, fix it in code instead of resending the whole context.

Step 5: Meter Spend Per Feature

You can't optimize what you can't attribute. Tag every call with a feature name and record tokens in and out:

export async function meteredCall(feature: string, task: Task, params: CallParams) {
  const started = Date.now();
  const res = await llm.call({ ...params, model: modelFor(task) });

  await metrics.record({
    feature,
    task,
    model: res.model,
    inputTokens: res.usage.inputTokens,
    outputTokens: res.usage.outputTokens,
    cachedTokens: res.usage.cachedTokens ?? 0,
    latencyMs: Date.now() - started,
  });
  return res;
}

With that in place, add:

• Spend alerts per feature and a global daily budget alarm
• Per-user or per-tenant rate limits on anything user-triggered
• A shadow or A/B path to test model downgrades on live traffic before committing

What the Audit Should Produce

The output isn't a rewrite. It's a ranked list:

1. 01Top savings, each with the call site, current pattern, proposed change, estimated share of that call's cost saved, risk, and how quality will be checked
2. 02Quick wins that are safe to ship this week (usually caching and loop fixes)
3. 03Instrumentation gaps to close before the next round

Rough estimates are fine as long as you state the assumptions (volume, average tokens). A ranked list that's roughly right beats a precise spreadsheet nobody acts on.

Related

• How to Ship an AI Feature Safely: rollout flags, guardrails and cost ceilings
• Claude Haiku 5.5: the best small model for subagents
• Redis Caching Strategies for High-Performance APIs
• LLM Cost Audit agent skill

Harsh Rastogi is an AI Product Engineer and Business AI Head at Modelia, building enterprise-grade AI products and agents. He writes about shipping AI to production at harshrastogi.tech.