Test What Your MCP Client Does When the Server Misbehaves

Testing Stops at the Happy Path

Connecting a product to the rest of a customer's stack used to mean building one integration per app. Over the past year that work moved to the Model Context Protocol (MCP ) and a product that wants its agent to act across the stack now consumes other companies' MCP servers instead of writing an adapter against each API. Notion's Custom Agents connect out to third-party MCP servers. Atlassian ships nine Rovo agents in Jira and Confluence that run on third-party MCP servers, including Figma, Canva, Intercom and Amplitude. It also runs an MCP server of its own, which Claude, Cursor and VS Code connect into. Both companies became MCP clients on the way to shipping an agent.

So did we. Cakewalk builds access management for AI agents and our MCP Gateway sits on both sides of the protocol. Agents connect to it as a server and it connects out as a client to MCP servers we do not own. Both directions of failure land on us.

Building the gateway, we went looking for a way to test that and did not find one. Adoption ran ahead of the means to test it and most of the effort that does exist has gone into helping people write correct servers. What points at clients checks them for compliance, which needs a server that behaves. You can find out whether a server is correct. Nothing tells you what your client does when it is not. A client author cannot manufacture the failure, because no well-behaved server will produce one on request.

So we built a server that gets it wrong on purpose and used it internally. Today we are opening it to anyone building an MCP client. It answers on both protocol revisions now in circulation and it is free to use. Every run leaves a record of how your client handled the failure, which is the question we could never answer about our own.

Failures Do Not Look Like Failures

That question is harder to answer than it sounds, because a failure does not always arrive looking like one. The specification is explicit about why.

MCP puts failures in two different places.

Protocol errors, meaning an unknown tool or a malformed request, arrive in the response's error field.
Tool execution errors, meaning an API failure or a validation problem inside the tool, arrive in the result field instead, flagged with isError: true.

/ json

{ "jsonrpc": "2.0", "id": 3,
  "error": { "code": -32602, "message": "Unknown tool: get_wether" } }

{ "jsonrpc": "2.0", "id": 4,
  "result": { "resultType": "complete", "isError": true,
              "content": [{ "type": "text", "text": "Invalid departure date" }] } }

The second is the one that matters here. The call succeeded and the work inside it did not. Nothing watching status codes sees that and nothing watching for JSON-RPC errors sees it either.

The specification then says what to do with each. Clients MAY pass protocol errors to the language model. Clients SHOULD pass tool execution errors to the model, so it can correct itself and retry.

In practice some server authors have stopped trusting that path. On r/mcp, a team building MCP servers reports that clients render isError "inconsistently enough that we stopped depending on it," so they return plain successful results describing the failure in text instead. That reaches the model reliably and it moves the invisibility one layer down, which is the thread author's own read on it.

So the failure class that hides best is also the one routed by default to the component that will not stop and ask. An agent given a failed result works around it and where the gap was information it needed, it fills the space with something plausible. Nothing in that path requires a person to ever find out.

What This Looks Like From Inside a Client

Finding out is the hard part. On r/mcp , a .NET server maintainer's install audit turned into a thread about client error handling and its sharpest case came from a commenter driving a GUI daemon over HTTP. After his lease expired, every action kept returning a normal success envelope with the failure status sitting inside it. He sent about ten clicks before noticing the screen had never changed, "because nothing about the response said failure."

Ours was louder and still pointed at the wrong layer. In August 2026 we hit two failures on consecutive days while wiring up a third-party MCP server. The first was a genuine authorization break and we fixed it. The second looked identical from the outside and was not. Their server answered a discovery call with a result missing supportedVersions and capabilities, both of which the SDK requires and nothing that reached us said so. Having just fixed a real authorization bug, we spent the morning in the wrong layer.

None of that is unusual. Integrating with software you do not control has always looked like this and the standard answer is to test for the failures before a customer finds them.

You Do Not Get to Write the Failure

Several projects do exactly that. MCP Inspector connects to a server and lets you call its tools by hand. The official conformance suite runs your client against a test server it spins up per scenario, then checks the exchange against the specification and answers pass or fail. Its suites include auth and OAuth flows. MCPJam exercises a server the way real clients do and in early August published 700 compatibility checks across 14 clients and over 50 capabilities, on the stated premise that "having a label that says 'supports MCP' doesn't tell you much anymore". MCP Playground hosts six fixed mocks you can point a client at without signing up. We use two of the four.

Two of those four point at servers. The conformance suite and Playground point where we do and both run a catalog somebody else authored. That is the line worth drawing. Conformance asks whether your client obeys the specification, which requires a server that behaves. We ask what your client does when the server does not and the specification has little to say about that, which is why nothing here comes back pass or fail.

	What it does	Direction
MCP Inspector	Connect to a server, then list and call its tools by hand	Tests your server
Conformance suite	Run your client against a scenario test server, or probe your server as a client. Pass or fail against the specification	Both, CI-oriented
MCPJam	Exercise a server using emulations of real clients, plus a matrix of what works in each one	Tests your server
MCP Playground	Fixed mocks, including deliberate errors, a 401 challenge and the new revision	Tests your client
MCP Test Kitchen	Failures you configure, on the call you choose, plus the server's record	Tests your client

A Server That Misbehaves on Command

Six scenarios ship today. There is one endpoint and one tool called run_configured_test_scenario and what that tool does depends on which scenario you picked in the console. Signing in is what ties your token to that choice.

Scenario	What it does
Baseline (success)	Answers normally, with an optional delay before returning. The control case
HTTP transport error	Returns a configured sequence of status codes across successive requests, so the failure sits below MCP
JSON-RPC error	Returns HTTP 200 with an error object in the body, on the call you name
Tool error (isError)	Returns a valid result carrying isError on the call you name, so the failure sits inside a success
Elicitation approval	Returns an input required result, then completes when your client calls again with the answer. Needs the 2026-07-28 revision
C# SDK v2 backward compatibility	Tests whether your client handles the new revision's multi round-trip requests or falls back to the older path

It will not tell you whether your client is correct, only what your client did. Nor does it test your server or certify anything against the specification. The transport is stateless, so the server cannot bridge state across a live connection and stateful paths stay out of scope.

Take the failure from earlier, the one where every action came back as a normal success envelope. Tool error (isError) reproduces it and the parameter that matters is which call it lands on.

/ json

{ "errorMessage": "Lease expired", "onInvocation": 2 }

A failure on every call gets noticed. The one that costs you arrives after things worked, so the first call returns clean and the second carries isError. No fixed endpoint can express that, which is the difference between this and the other four.

Point your client at the endpoint and run the task you would run against a healthy server.

The server writes down one row per accepted request, carrying the method, the phase, the status, the duration, a summary of the params your client sent, whether it cancelled and the protocol revision it declared.

          method: tools/call  phase: tool_execution  status: 200  duration: 38ms  protocol: 2026-07-28
method: tools/call  phase: tool_execution  status: 200  duration: 41ms  protocol: 2026-07-28

Two rows, both 200, with nothing in either one saying which call failed. The transport layer cannot separate a tool error from a success, so nothing watching status codes could have caught it. Now you have seen that rather than assumed it.

What the record cannot tell you is what your client did next and that is the part you are testing. You know the second call carried isError, because you configured it. If an answer still reached your user, your client passed a failed result upward as content. That is a result-handling fix.

The revision column carries a second reading. A client sends whatever its SDK defaults to, which may differ from what you configured. That is what decides which handshake runs. The server answers on both revisions in circulation, so you can point the same scenario at each and compare. Two scenarios are the exception: elicitation approval and parts of the C# SDK v2 scenario need the newer revision.

Change what you found, then run the same scenario again. The second record is your proof.

How to Run It

Getting to that first record takes three steps. Sign in to the console with Google or GitHub, copy your personal access token, then pick a scenario and its parameters. The rest is client configuration.

/ json

{
  "mcpServers": {
    "mcp-test-kitchen": {
      "url": "https://mcp-test-kitchen.cakewalk.security/mcp",
      "headers": { "Authorization": "Bearer <your-token>" }
    }
  }
}

If you have no client to point at it yet, the console links to a TypeScript test client you run locally with your endpoint and token. The server is open source too, under the MIT license, so you can read exactly what it does before you trust its record.

We are asking you to point a real client at a service we run, so here is what the service keeps. It records each accepted request with its method, timing, status and a summary of its parameters, along with the email you signed in with. Tool arguments are never recorded. It keeps six request headers by allowlist and replaces every other value with [redacted], so your authorization token is never written. It keeps the records for up to seven days, and deletes the oldest sooner when the shared record fills up, so pull anything you want to keep into your own tracker.

Tell Us What to Break Next

Your client is the one end you can always change. Some of the servers it connects to are yours to fix and the rest are not. For those, the specification leaves plenty of room to be unhelpful without being broken. That split is permanent, so the question worth answering is what your client does when the other side gets it wrong.

We built the MCP Test Kitchen to answer that question about our own client and it is yours to use for the same purpose. The code is on GitHub if you want to see how it misbehaves. Tell us what broke and tell us what we should break next. Either reaches us at service@cakewalk.io and the console has a feedback form if you would rather use that.

Happy testing.