The agent works locally and fails in production. Diff the evidence, not the codebase.

Build a paired reproduction

Choose one minimal input that succeeds locally and fails after deployment. Record the same evidence for each run: commit or artifact identifier, runtime and dependency versions, model, region, configuration names, external endpoints, timestamps, correlation IDs, and final observable result.

Redact secret values. Presence, identity, and version are usually enough for the first comparison.

1. Prove the deployed artifact

Verify that production is running the intended commit, generated configuration, migration state, and dependency lock. A green build proves the build process completed; it does not prove the live route uses that artifact or configuration.

2. Test network and TLS from production

Run discovery and health requests from the deployed runtime. Check DNS, IPv4/IPv6 behavior, proxy and egress policy, TLS hostname, redirect chain, firewall, and private-network reachability. Measure latency against every timeout boundary.

3. Verify identity without printing secrets

Confirm the expected environment variables exist, the service account or OAuth client is the intended one, callback URLs match, and permissions are available in the production tenant. Compare token metadata—issuer, audience, scopes, expiry—not raw token contents.

4. Compare runtime constraints

Local processes may have a writable filesystem, long timeouts, persistent memory, implicit credentials, and unrestricted outbound access. Production functions and containers may not. Check memory, CPU, request and background timeouts, concurrency, cold start, filesystem lifetime, payload limits, and region.

5. Trace state across boundaries

Use one correlation ID through model call, tool call, callback, queue, database write, and final response. Look for races, duplicate retries, stale cache, missing awaits, and background work that outlives the request.

6. Verify the user-visible outcome

A 200 response, successful tool handler, or passing unit test is not the final contract. Assert the exact downstream effect: the record changed, message arrived, file exists, scheduled job completed, or user can see the result.

Keep the smallest production smoke

Once repaired, preserve a low-risk smoke that exercises the previously failing path and emits a unique success token. Run it after deploys and dependency, model, auth, or routing changes.