Traditional cloud spend is broadly predictable. Servers run continuously, storage grows gradually, and the bill moves slowly enough that quarterly review catches problems.

AI workloads break all three assumptions. GPU capacity can cost more per hour than a server costs per month. Inference spend scales with usage rather than with provisioned capacity, which means it scales with success. And experimentation is bursty by nature — that is what makes it experimentation.

The result is that a monthly review cycle is too slow. By the time the invoice arrives, the money is spent.

Where the surprises come from

Idle GPU capacity

The most expensive and most common. A GPU instance is provisioned for training or fine-tuning, the work finishes, and the instance keeps running because nobody explicitly stopped it. We have found instances idle for months. On premium capacity, a single forgotten instance can be a five-figure annual cost by itself.

Inference scaling with success

The pilot cost very little because forty people used it. Then it was rolled out to two thousand people and the cost scaled linearly with adoption, which nobody had modelled because the pilot’s cost was too small to be worth modelling.

Retrieval amplification

Retrieval-augmented systems fetch context before each call. Poorly tuned retrieval returns far more context than the answer needs, and since you pay per token, an inefficient retrieval configuration can multiply cost several times over with no visible effect on quality. This one is almost always invisible without deliberate instrumentation.

Development traffic in production

Test harnesses, evaluation runs and automated regression suites calling production endpoints. Individually trivial, collectively significant, and rarely attributed to anyone because the calls come from CI rather than from a person.

Egress and storage accumulation

Vector stores, embedding caches and training artefacts accumulate. Nobody deletes anything because storage feels cheap and nobody is sure what is still needed. Egress charges on data movement between regions or providers are a recurring surprise.

Five controls that hold

These are the controls we put in place on managed AI environments. They are deliberately boring, which is why they work.

1. Tag enforcement at provisioning

Every resource carries a workload, owner and environment tag, enforced by policy at creation — untagged resources simply cannot be provisioned. This sounds trivial and it is the foundation of everything else. Without it, cost attribution is guesswork and every subsequent control is unenforceable.

Retrofitting tags onto an existing estate is genuinely painful, which is the argument for enforcing it from the first day rather than the first audit.

2. Budgets with hard caps on non-production

Production environments get a budget with alerting. Development and experimentation environments get a budget with a hard stop. Teams get a monthly allocation they can spend as they see fit, and when it is gone the resources stop.

This is less restrictive in practice than it sounds. Teams handle a known allocation well. What they handle badly is unlimited spend with retrospective criticism, which is the default arrangement in most organisations.

3. Anomaly alerting on rate, not total

Monthly budget alerts fire after the money is gone. Rate-based alerting — comparing spend velocity against a rolling baseline — fires within hours.

This is also a security control. A sudden unexplained spend increase is a classic indicator of compromised cloud credentials being used for crypto mining or model abuse. Cost anomaly and security alerting should route to the same desk, because the first symptom is frequently financial rather than technical.

4. Scheduled shutdown by default

Non-production compute stops outside working hours unless explicitly exempted. For most organisations that is roughly a 65 percent reduction in non-production compute hours, for essentially no impact on the work.

Exemptions are fine. They just need to be requested and recorded, which turns an invisible default into a visible decision.

5. Cost per transaction as a tracked metric

The most useful control, and the one most often missing. For every AI workload, track cost per processed item — per document, per ticket, per request.

Total spend rising is ambiguous: it might mean waste, or it might mean the system is being used more, which is what you wanted. Cost per transaction rising is unambiguous: something has become less efficient. It is also the number that makes the business case legible, because it can be compared directly against what a human costs for the same item.

What not to do

• Do not block provisioning behind approval queues. Teams will use a personal account or a corporate card, and you lose visibility entirely. Budgets with hard caps achieve the control without the shadow IT.
• Do not optimise before you measure. Right-sizing against a week of data usually produces the wrong answer. Thirty days of real usage is the minimum for a decision you will not have to reverse.
• Do not treat reserved capacity as free money. Commitments are a good deal for stable workloads and an expensive mistake for experimental ones. AI workloads change shape faster than most commitment terms.
• Do not let cost control become a veto. The point is predictability, not minimisation. A control regime that makes legitimate experimentation difficult will be routed around, and then you have neither cost control nor visibility.

Where to start

If you do one thing, enforce tagging. Everything else depends on being able to attribute spend to a workload and an owner, and it is the control that becomes exponentially more expensive to add later.

If you do two things, add rate-based anomaly alerting. Between them, tagging and rate alerting will surface the overwhelming majority of cost surprises while there is still time to do something about them.