Your LLM spend cap is bypassable: X-Forwarded-For vs CloudFront-Viewer-Address, and why you also need an origin lock

Posted on Oct 26 • Originally published at liftstack.gaperi.ro

If you put a per-user or per-IP free-tier cap in front of a paid call on Lambda behind CloudFront and API Gateway, the obvious key is spoofable, the execute-api URL is a side door, and an in-memory counter overshoots. Here is each bug and its fix.

You run a Lambda behind CloudFront and API Gateway. In front of a paid call — an LLM completion, say — you put a free tier: so many calls per anonymous user per day, then you stop. The counter needs a key. The obvious choices are the caller's IP or a user id the client sends. Both are a mistake, and the mistake is quiet: nothing errors, the bill just grows.

This is three bugs in that design and the fix for each, ending with commands that exercise a live demo. The key function and the origin lock are copied from code running in production; the atomic counter in Fix 3 is newer and runs in the demo.

The naive version keys the counter on X-Forwarded-For, or on an x-user-id the client supplies. It usually looks like this.

// DON'T: the client controls this value. function callerIp ( event ) { const xff = event. headers [ ' x-forwarded-for ' ]?? ''; return xff. split ( ', ' )[ 0 ]. trim (); // left-most entry } Enter fullscreen mode Exit fullscreen mode The spoofable key.

CloudFront is both the reason this fails and the tell. When a viewer request already carries X-Forwarded-For, CloudFront appends the viewer's real IP to the end of the header before it reaches your origin, leaving whatever the client sent to the left. Reading the left-most entry hands you an attacker-chosen value. The right-most entry is no better: API Gateway appends the CloudFront server's address after it. Don't parse this header at all.

So the cap is free to defeat. Rotate the header and every request looks like a fresh IP, so a new bucket is minted on each call.

for i in $( seq 1 100 ); do curl -s -X POST https://your-app.example/api/llm \ -H "X-Forwarded-For: 10.0.0. $i " done # every request looks like a new IP, so the per-IP cap never trips Enter fullscreen mode Exit fullscreen mode A hundred paid calls, one real client, against any API keyed this way.

An x-user-id you let the client send is the same bug without the CloudFront detail: the client picks the key, so the client picks the bucket.

Key on CloudFront-Viewer-Address. CloudFront derives it from the viewer's connection; it holds the IP and the source port, for example 198.51.100.10:46532. When your origin request policy adds it, CloudFront overwrites any value the viewer sent, so the client cannot choose it.

Two conditions attach. First, the header has to reach your Lambda: it is a viewer-location header, which you forward with an origin request policy (it is not available in a cache policy). The managed AllViewerExceptHostHeader policy — the one this site's /api/* behaviour uses — forwards all viewer-location headers and drops Host, which is what API Gateway and Lambda function URL origins need. Confirm the header actually arrives at your handler, and fail closed if it is missing.

Second, strip the port and treat a missing value as no-key, not as a free pass. This is the whole key function from the code behind fiscura.ro.

export function ipKey ( event: APIGatewayProxyEventV2 ): string | null { const viewer = event. headers [ ' cloudfront-viewer-address ' ]; if ( viewer ) return `ip: ${ viewer. replace ( /: \d +$/, '' )} `; // strip the port if ( process. env. NODE_ENV!== ' production ' ) return ' ip:dev '; return null; } Enter fullscreen mode Exit fullscreen mode Fails closed only if you set NODE_ENV=production; Lambda does not set it for you.

Header names arrive lowercased in the HTTP API v2 payload, so read cloudfront-viewer-address, not CloudFront-Viewer-Address.

A per-address key has one more gap: IPv6. CDK turns IPv6 on for CloudFront distributions by default, and one host can hold many addresses (on AWS, one network interface can take a /80 prefix), each with its own bucket and no forged header needed. Key IPv6 viewers on their /64 prefix or wider. The demo below does; the function above does not yet.

There is a second entrance. The execute-api URL API Gateway gives your HTTP API is public by default: anyone who has it can call it directly. You can disable that default endpoint, but then the API answers only on a custom domain name, which is just as reachable from the internet, so turning it off moves the side door rather than closing it. A direct caller skips CloudFront entirely, so it can send any CloudFront-Viewer-Address it wants.

That means Fix 1 only holds for traffic that genuinely came through your distribution. Straight to execute-api, the viewer-address header is caller-controlled again, and you are back to Bug 1.

curl -s -X POST https://332dsafb8d.execute-api.eu-central-1.amazonaws.com/demo/cap \ -H 'CloudFront-Viewer-Address: 203.0.113.99:443' # without an origin check, the origin trusts this Enter fullscreen mode Exit fullscreen mode The side door. This site's API is locked, so here the same command gets a 401 (see Try it).

The Lambda event's sourceIp will not save you here: it is the IP of the immediate TCP connection to API Gateway, which for CloudFront traffic is the edge node, not the viewer. Allow-listing CloudFront's IP ranges admits every distribution, not just yours.

Lock the API to CloudFront with a secret only CloudFront can present. Configure a custom origin header — call it X-Origin-Verify — with a long random value on the API origin. CloudFront adds it to every request it sends there and overwrites any same-named header a viewer sends; a caller going straight to execute-api can send the header too, but does not know the value. AWS documents this pattern for keeping callers from bypassing CloudFront: HTTPS on both legs, and a value you rotate periodically, accepting old and new while the change rolls out.

The Lambda compares the header against the expected secret in constant time and refuses anything else. timingSafeEqual throws if the two buffers differ in length, so compare lengths first.

function viaCloudFront ( event: APIGatewayProxyEventV2 ): boolean { const expected = process. env. ORIGIN_VERIFY; if ( expected === undefined ) return true; if ( expected === '' ) return false; const sent = event. headers?.[ ' x-origin-verify ' ]; if ( typeof sent!== ' string ' ) return false; const a = Buffer. from ( sent, ' utf8 ' ); const b = Buffer. from ( expected, ' utf8 ' ); // Constant-time, so response timing cannot guess the token byte by byte. // timingSafeEqual throws on buffers of different lengths, so a length // mismatch is answered first; that reveals only the length, which is no // secret (the stack fixes it at 40 characters). return a. length === b. length && timingSafeEqual ( a, b ); } Enter fullscreen mode Exit fullscreen mode Empty secret = misconfigured, refuse. Unset = local runs and tests, so it fails open: make sure every deployed stack sets it.

The refusal is 401, not 403, on purpose. A direct execute-api call never passes through CloudFront, so its refusal reaches the caller as sent. But a request that did come through CloudFront can still fail the check (mid-rotation, say), and CloudFront custom error responses apply to the whole distribution, not per behaviour. This site maps 403 to its /404.html page, so a 403 from the API would reach the client as that HTML page, with a 404 status. CloudFront can remap only 400, 403, 404, 405, 414, 416 and 500–504, so the API refuses with 401 and caps with 429, which always pass through. This site leaves 404 alone, so the API's own 404s pass through too.

The header lock is the simplest lever for an HTTP API, which cannot use AWS WAF, resource policies or private endpoints (REST API features); a REST API can also check the header in a WAF rule. Alternatives: CloudFront origin mutual TLS to an HTTP API custom domain that requires it, with the default endpoint disabled; or a Lambda function URL behind CloudFront origin access control (AuthType AWS_IAM, and POST/PUT clients must send the body's SHA-256 in x-amz-content-sha256).

With the right key and the side door shut, the counter itself can still overshoot, in two ways.

First, an in-memory counter. On standard Lambda, each concurrent request gets its own execution environment, which handles one request at a time. A Map in module scope is therefore per container: with N warm containers your effective cap is at least N times what you set, and each count resets whenever Lambda replaces a container, which it does every few hours even under steady traffic.

Second, even a shared store overshoots if you read, then decide, then write. Check-then-act leaves a gap.

// DON'T: two steps with a gap between them. const used = await countFor ( key ); // read if ( used >= CAP ) return capped (); await model. invoke ( prompt ); // paid call await increment ( key ); // write, after the fact Enter fullscreen mode Exit fullscreen mode Under a burst, many requests read "under the cap" before any writes land.

They all pass the check and all make the paid call, so the cap overshoots by however many requests arrive in that gap.

Collapse read, check and increment into a single conditional UpdateItem. DynamoDB applies writes to an item in the order received, and evaluates the condition against the latest version of the item, so concurrent calls queue on the item rather than racing each other.

await ddb. send ( new UpdateItemCommand ({ TableName: table, Key: { k: { S: key } }, UpdateExpression: ' ADD #n:one SET expiresAt = if_not_exists(expiresAt,:ttl) ', ConditionExpression: ' attribute_not_exists(#n) OR #n <:cap ', ExpressionAttributeNames: { ' #n ': ' n ' }, ExpressionAttributeValues: { ':one ': { N: ' 1 ' }, ':cap ': { N: String ( cap ) }, ':ttl ': { N: String ( ttl ) }, }, ReturnValues: ' UPDATED_NEW ', })); // ConditionalCheckFailedException => already at the cap, return 429 Enter fullscreen mode Exit fullscreen mode The expression strings are exactly those in the demo's consume() (api/src/lib/caps.ts).

ADD on a number adds to the current value, or creates the attribute at that value when the item is new; if_not_exists sets the TTL on the first call only. When the count is already at the cap the condition is false, DynamoDB throws ConditionalCheckFailedException, and the handler returns 429. No burst gets past the cap.

Three limits remain. The update is not idempotent: if it lands but the response is lost, the AWS SDK can retry it and count that call twice, so the cap errs toward refusing early. It counts calls, not dollars: bound each call's cost (output tokens, input size), or reserve the worst case before the call and settle after. And it bounds each address, not the total: many real addresses mean many buckets, so keep one global daily ceiling behind it.

The demo stores the address only as an HMAC-SHA256 hash, with a TTL so DynamoDB deletes the row. The reset comes from the UTC date in the key, not from TTL: TTL deletes within a few days of expiry, and an expired row can still be read until then.

The guide page has a live widget backed by POST https://liftstack.gaperi.ro/api/demo/cap: ten calls per viewer IP (IPv6: per /64) per UTC day, keyed on CloudFront-Viewer-Address, refused with 429 after that and reset at 00:00 UTC. Three commands show the fixes.

# 1) a forged X-Forwarded-For is ignored; your real viewer IP is counted curl -s -X POST https://liftstack.gaperi.ro/api/demo/cap -H 'X-Forwarded-For: 203.0.113.99' # 2) a forged CloudFront-Viewer-Address through CloudFront is overwritten curl -s -X POST https://liftstack.gaperi.ro/api/demo/cap -H 'CloudFront-Viewer-Address: 203.0.113.99:443' # 3) straight to execute-api, past CloudFront: the origin lock refuses it curl -s -X POST https://332dsafb8d.execute-api.eu-central-1.amazonaws.com/demo/cap -H 'CloudFront-Viewer-Address: 203.0.113.99:443' Enter fullscreen mode Exit fullscreen mode The first two count your real address; the third never reaches the counter.

The first returns 200 (429 with capped: true after ten calls in a UTC day), with countedAs set to your IP as CloudFront saw it, port stripped (for IPv6, its /64 prefix). Under ignored it echoes the X-Forwarded-For the Lambda received: your forged value, your real IP (appended by CloudFront), then a CloudFront server's address (appended by API Gateway). That is Bug 1, live. The second counts your real IP too: CloudFront replaced the forged header. The third returns 401 {"error":"not_via_cloudfront"}. A 429 without capped is API Gateway's rate limit, not the cap.

A version of this lock runs in production behind fiscura.ro, where it answers 403: a direct call with a forged viewer address is refused, while the identical request routed through the site returns 200.

The live demo with buttons is on the original page: https://liftstack.gaperi.ro/guides/spend-cap-bypass#try-it

The demo stores a keyed hash of your IP address (for IPv6, of its /64) with today’s count, never the address. DynamoDB deletes it after midnight UTC, usually within a day or two.

Disclosure: I build Liftstack, a serverless-SaaS starter kit for AWS. It is in development and not for sale yet. The cap key above is from the code behind fiscura.ro, and the constant-time origin lock from keepvista.com, both of which I run; I am packaging them, with the atomic counter, into the kit.

This article was written with the help of AI and fact-checked before publishing.