QA Skills Library / mcp-server-verification

Verifying an MCP server (half a test is not a test)

An MCP server is a write-capable interface to somebody's live data. "It starts and lists 17 tools" is not verification — it is the first of five stages, and the four that matter are the ones that touch real records. Sohail's ask lands in exactly this shape:

«ال mcp شغال يسطا؟؟؟ خلى كلود يتأكد انه شغال وبيغير البيانات اللى ف الموقع فعلا والايجنت يقدر يستخدمه ويضيف محتوى عالداتا بيز بتاعت موقع الديمو ولا لا؟»

Two questions hide inside that sentence: does it read? and does it write, in a way a human looking at the site can see? Answer both with evidence, and clean up after yourself.

The five stages

1. Enumerate — names and input schemas

tools/list giving a count proves registration, not usability. Pull the schema and classify each tool read vs write; the write set is your test surface.

node scripts/mcp_probe.mjs            # every tool + required props + enums + a WRITE/read tag

scripts/mcp_probe.mjs is a dependency-free stdio JSON-RPC client — spawn the server, initialize, tools/list, optionally --call <tool> '<json>'. Use it instead of hand-writing a client per project. Server output arrives as JSON inside result.content[0].text; parse that string, don't assume the result is an object.

2. Read — one real call, with the numbers quoted back

Call the most load-bearing read tool and write the returned values into the report verbatim (cash = 180000, first customer name). If a read tool errors, stop here; nothing downstream is meaningful.

3. Write — a marked record, through the MCP, never through SQL

Pick a write tool, create one record whose name/title contains a unique marker (فحص-كلود-مستقل, hermes-write-probe-<epoch>). Record the returned id. See scripts/mcp_write_test.mjs for the shape: baseline read → create → re-read by marker → report.

Read the enum from the schema, not from the seed data. A create_customer call was rejected with Invalid enum value. Expected 'active' | 'lead' | 'inactive', received 'prospect' — while the seeded dataset itself contained a customer with status: "prospect". The tool schema and the domain vocabulary had drifted apart. That mismatch is a finding to report, not a reason to give up; re-call with a legal value.

4. Confirm it on the surface the user looks at

This is the stage everyone skips. An API returning the new record proves the API works — not that the site changed. Fetch the public URL and grep for the marker:

curl -s --max-time 60 "https://<public-host>/customers" | grep -c 'MARKER'   # expect ≥ 1
curl -s --max-time 60 "https://<public-host>/tasks"     | grep -c 'MARKER'   # expect ≥ 1

Only a hit here answers "بيغير البيانات اللي في الموقع فعلاً". A public hit also proves the proxy / gateway / tunnel path, not just localhost.

5. Revert — and prove the revert on the same surface

cp <db-file> /tmp/backup/before-write      # BEFORE any write. Non-negotiable on a shared demo DB.
# ... write test ...
# cleanup, then:
curl -s "https://<public-host>/customers" | grep -c 'MARKER'   # expect 0
curl -s "https://<public-host>/" | grep -oE '180,000|110,000|141,000' | sort -u   # invariants intact

Prefer the app's own reset/cleanup endpoint over raw SQL. An app that owns its data will invalidate its own caches; your SQL will not (see the cache pitfall below).

Pitfalls — each one cost a real cycle

bash hermes mcp list # is the server configured and enabled at all? hermes mcp test <server> # does it actually CONNECT? (a 30s hang means no) <server-venv>/bin/python <server>/main.py # run the entrypoint by hand — the real error is here

tool_describe on a name you have seen work before answering "not a deferrable tool" is the tell: the tool is not renamed, the server is not up. The historical names are still recoverable — grep -rhoE 'mcp__[a-z_]+__[a-z_]+' ~/.hermes/logs | sort -u — and that is also how you establish the server once did work, so it is worth repairing rather than routing around. - A dead long-lived session is the usual cause for a user-account (non-token) server, and it names itself. An authenticated server keeps a session/credential on disk; when the same one is consumed from two places, the provider invalidates it rather than sharing it:

AuthKeyDuplicatedError: session in use from another IP

Retrying never fixes this — it needs a new session minted for this box, which usually means an interactive re-login the owner completes (a QR scan; Phase 2h of headless-cli-auth, script ready at scripts/qr-login-by-proxy.py). Until that happens the honest report is "the server is configured but its session is dead" — not "the MCP is broken", and not a silent drop from the verification you were asked to run.

Independent verification by a second agent

The owner will often ask for a different agent to confirm — which is also the honest way to check your own work. Wire it up with a real MCP config and require a machine-readable verdict:

// /tmp/<name>-mcp.json
{ "mcpServers": { "<name>": {
    "command": "node", "args": ["/abs/path/mcp/dist/index.js"],
    "env": { "API_URL": "http://127.0.0.1:3100/api/v1", "API_KEY": "…" } } } }
claude -p "$(cat verify_prompt.txt)" --mcp-config /tmp/<name>-mcp.json \
       --dangerously-skip-permissions < /dev/null > /tmp/verify.out 2>&1

The brief must end with a fixed first-ish line a script can branch on, so the verification itself is auditable:

MCP-VERDICT: WORKS      (or)      MCP-VERDICT: BROKEN

…plus an explicit step ordering: enumerate → read → write → confirm on the live site → run the app's reset → confirm the marker is gone → confirm the acceptance numbers are intact. Telling the verifier to clean up after itself is what stops an "independent verification" from becoming a second contamination of the client's demo. Use templates/mcp-verify-brief.md.

Handing it to a REMOTE consumer (different question, same server)

«هات لينك الـMCP وطريقة وضعه في كلود شات» is not a verification task — it is a hand-off, and it fails in ways verification never touches: the client runs on somebody else's servers, so localhost working means nothing.

Full recipe, the exact curl/check-host commands and the paste-ready hand-off message shape: references/remote-mcp-exposure.md. The read-only projection recipe (FastMCP re-registration of a tool subset, the systemd unit + reverse-proxy block, the 200/404 verification matrix, and the secret-hygiene checks) is in references/read-only-mcp-publishing.md.

Support files

Related skills: ask-claude-when-stuck (running claude -p as a second opinion at all), delegated-work-evaluation (§5 — commissioning an independent reviewer and relaying its fix list), web-visual-qa (browser-side measurement when the write has to appear on a rendered page).

raw Markdown · all skills