01 / Accepted brief
One workflow. One observable result.
- Starting symptom
- The local MCP tool works, but the deployed agent reports that no support tool is available.
- Bounded workflow
- Classify one synthetic support request, call
create_test_ticket, and return its ticket reference. - Acceptance criterion
- A production-hosted run creates exactly one sandbox ticket containing trace
ARD-SYNTHETIC-20260727, and the agent returns the same reference. - Authorized changes
- Production gateway configuration, MCP route configuration, and one non-destructive smoke test.
Transport routing, protocol initialization, tool discovery, direct sandbox execution, and the agent’s final tool-selection path.
New tools, prompt redesign beyond the failing path, production customer data, UI work, and ongoing monitoring.
02 / Reproduction
The first divergence is at the public MCP route.
The same server package and tool schema are exercised through two paths. The local stdio path initializes and lists the target tool. The production Streamable HTTP path returns a web document before the MCP server receives the request.
| Check | Local | Production | Meaning |
|---|---|---|---|
| Artifact | Match | Match | Same fictional commit and lockfile. |
| DNS + TLS | Not applicable | Pass | Host resolves; certificate is valid. |
POST /mcp | JSON-RPC | 200 text/html | Gateway responds before MCP origin. |
| Origin log | Initialize received | No request | Failure precedes server code and auth. |
| Tool discovery | 3 tools | Blocked | Client never reaches capability negotiation. |
00:00.000 POST https://agent.example.invalid/mcp
00:00.041 HTTP 200
00:00.041 content-type: text/html; charset=utf-8
00:00.042 body-prefix: <!doctype html>
00:00.043 mcp-origin request count: 0
00:00.044 client: invalid JSON-RPC response03 / Root cause
A catch-all web route owns /mcp in production.
The production gateway evaluates its single-page application
fallback before the MCP upstream rule. Any unmatched path,
including POST /mcp, receives the application shell
with status 200. The local stdio path bypasses that gateway,
which explains why the same server appears healthy locally.
A successful HTTP status is not sufficient evidence. The response media type, JSON-RPC exchange, tool execution, and downstream effect must each be verified at their own layer.
04 / Agreed repairs
Three changes, each tied to the accepted outcome.
-
01
Route the MCP endpoint before the application fallback
The explicit
/mcpupstream rule now precedes the catch-all web route and accepts only the methods required by the server. -
02
Preserve the transport contract through the gateway
The gateway forwards the request method,
Content-Type,Accept, and MCP session header, disables response rewriting, and allows the configured streaming duration. -
03
Add a deploy-time production smoke
A credential-scoped sandbox check initializes the server, confirms the target tool schema, calls it once with the unique trace, and verifies the resulting sandbox record.
Protocol checks follow the official MCP Streamable HTTP transport and use the MCP Inspector for direct tool verification.
05 / End-to-end verification
The proof climbs from transport to business outcome.
Transport
POST /mcp reaches the origin and returns an allowed MCP response type.
Protocol
initialize completes with the expected protocol version and server identity.
Discovery
tools/list contains create_test_ticket with the expected input schema.
Direct execution
The tool creates one sandbox ticket carrying trace ARD-SYNTHETIC-20260727.
Agent control
The deployed agent selects the tool for the accepted synthetic request without a manual tool override.
User-visible result
The agent returns the exact ticket reference and the sandbox record is independently readable.
One synthetic request → one tool call → one sandbox ticket → one matching reference.
06 / Rollback
The repair can be reversed without touching data.
- Restore the versioned gateway configuration from the pre-change artifact.
- Redeploy the prior gateway release while leaving the MCP service package unchanged.
- Run the read-only transport probe and confirm traffic again follows the prior route.
- Delete the synthetic sandbox ticket created by the smoke.
The sample change does not alter schemas, production records, authorization policy, or long-lived credentials.
07 / Residual risks and handoff
What is fixed, what is not, and what to watch.
The production MCP route now reaches the intended origin and the accepted agent path passes end to end.
Identity-provider, model-provider, and hosting outages remain outside the repaired route.
Run the smoke after gateway, MCP SDK, model, auth, or client transport changes.
Discovery passes but direct execution or the final sandbox record fails; that is a different layer and a new diagnosis.
Access closeout
- Temporary sandbox credential revoked after verification.
- Synthetic ticket removed after evidence capture.
- Logs retained only in redacted form; raw tokens never enter the packet.
- Customer rotates any credential shared outside its normal access system.
Have a real failing path?
Submit one observable outcome for a fit decision.
There is no charge to submit. Payment is requested only after the workflow boundary, access checklist, and proof target are written down.