The deep-probe study — MCP tool behaviour

Methodology is on the methodology page. This study set out to answer one question: does the Digital Applied MCP reliability study — median 71% pass rate, top decile ≥95%, bottom decile 38%, P50 320ms / P95 1,840ms, schema mismatches causing 38% of failures — reproduce?

Basis for this report: the full behavioural probe battery run once, live, against all 164 read-only tools in a curated corpus (selection rationale in §8 and §11), on 2026-08-23. 129 tools completed a battery and were persisted; 118 of those cleared the 20-trial publish threshold and carry a public behaviour_score, over 5,794 recorded trials. Everything below is measured from that run — see Method for exactly how, and Limitations for what a single run cannot yet say.

The headline: partial replication, in a specific and interesting way

It doesn't replicate cleanly — it replicates in shape, not in number, and the disagreement is itself a finding.

The median tool in this corpus is more reliable than Digital Applied's finding, not less: a typical tool here passes 95% of calls, and the top decile is essentially perfect. But the bottom decile is worse, not better — 0%, meaning the worst tenth of tools in this corpus don't merely underperform, several fail every call. That single fact — a much higher median sitting alongside a much harder floor — is why the pooled, trial-level pass rate (58.8%) reads close to Digital Applied's headline number even though the per-tool median is 24 points higher. Which statistic is "the" pass rate depends entirely on whether a handful of completely broken tools should outvote a much larger population of nearly-perfect ones. Digital Applied's methodology isn't published in enough detail to know which they used; this report publishes both, on purpose, rather than picking the one that reads better.

Latency direction also splits: typical-case latency here is faster (163ms vs 320ms P50), but the tail is at least as heavy (P95 1,975ms vs 1,840ms) and considerably heavier past that (P99 15,006ms — not a figure Digital Applied reported, so no comparison, but worth recording since a 15-second P99 next to a 163ms P50 is itself a notable shape).

Schema mismatches, their largest reported failure driver at 38%, are not the story here — see §4. That's the one dimension where this corpus looks nothing like theirs, and the most likely explanation is corpus composition, not a contradiction of their finding (see Limitations).

1. Score distribution

118 of 129 probed tools cleared MIN_TRIALS_TO_PUBLISH = 20 and carry a public behaviour_score. The other 11 are cost-sensitive tools capped at 5 trials by design (see §5) — correctly unscored, not zeroed.

Mean sits 13.6 points below median — the distribution is left-skewed: most tools cluster near-perfect, and a real minority pull the average down hard rather than the corpus being evenly mediocre. This is the same shape as the pass-rate story in the headline section, restated at the level of the composite score.

2. Component breakdown

Per the published methodology's weights (reliability 35%, schema integrity 25%, failure behaviour 15%, latency 15%, concurrency 10%), computed only over components with evidence — a dropped component reweights the rest rather than being imputed.

3. Reliability, in full

• Per-tool pass rate (the component that feeds the score): median 95.0%, top decile 100%, bottom decile 0%.
• Pooled trial-level pass rate, all 2,415 reliability trials across all 129 attempted tools: 58.8% (1,419 pass / 454 fail / 504 error / 38 timeout).
• The gap between those two numbers is not a bug — see the headline section. A small number of tools failing every call weighs heavily in a pooled count and barely at all in a per-tool median.

4. Schema integrity — the surprise

Digital Applied found schema mismatches caused 38% of failures. Here:

• 110 of 129 tools (85.3%) declare an outputSchema at all — far above the August 2026 census's whole-registry finding of 35.7% (the August 2026 sample). Expected: that census sampled the whole registry; this corpus is Class A only, and open, unauthenticated, well-behaved-enough-to-probe tools are not a random sample of schema hygiene.
• Of tools where conformance was actually checkable (declared a schema and had ≥1 successful call to check it against) — 63 of 118 — median and mean are both effectively 100% (99.6% mean). When a declared schema exists and a call succeeds, the response matches it almost without exception.
• The 63-of-110 gap is the reliability finding again, not a separate one: a tool with near-zero reliability produces few or no successful calls to check schema conformance against, so its schemaIntegrity component drops out (inapplicable, per the reweighting rule) even though it declared a schema. The methodology's stated rule ("dropped if no output schema declared") is technically narrower than the implementation ("dropped if no output schema declared, or no successful call exists to check it against") — a necessary implementation detail, recorded here rather than left as a silent gap between the doc and the code.

Read together: in this corpus, tools don't fail because their schema is wrong. They fail outright (timeout, error, tool-reported failure) at a meaningfully higher rate than Digital Applied found, and the ones that do succeed almost always match their declared shape. That's a real, specific disagreement with the prior study's largest reported failure driver, not a methodological wash.

5. Failure behaviour, timeout, cancellation, determinism

Pooled outcome counts across all attempted tools (129), not just the 118 published:

Timeout and cancellation both show zero recorded failures wherever the probe could reach a verdict at all — most of the volume for both is skipped (inconclusive: the call completed before a cancel could land, or errored for an unrelated reason), which is deliberately not counted against the tool (the ordinal treats missing evidence as benefit-of-the-doubt, not a penalty — trials_n is what surfaces coverage, not the score). Determinism is the one weak spot: just under half of testable determinism trials were inconsistent, on the (smaller) subset of tools flagged as static-result candidates.

6. Concurrency

• Component (pass-rate-at-4 ÷ pass-rate-at-1), 72 tools scored: median 100 — where a tool's baseline reliability is known and concurrency-at-4 could run, concurrency essentially never made things worse.
• Pooled raw outcome, concurrency-at-2 and concurrency-at-4 combined, all 129 tools, 2,360 trials: 1,418 pass / 371 fail / 571 error — 60.1% pooled. Same bimodal shape as reliability: the component score (ratio, per-tool) looks clean because it's normalized against each tool's own baseline; the pooled raw count re-surfaces the same handful of badly broken tools that drag every pooled statistic in this report downward.

7. Category breakdown

Category is a rough, disclosed-as-approximate 10-bucket heuristic, not a rigorous classification, joined in for this report only.

A roughly 60-point median spread from best to worst category (100.0 → 38.5) — a comparably wide category effect to Digital Applied's browser-automation-vs-filesystem finding (47% vs 89%), even though the specific categories don't correspond. dev-infra and communication are the weakest categories measured here; worth a closer read, since n is small per category (6–15) and a handful of broken tools on one server can swing a whole category's median (see §9).

8. Coverage: 164 → 129 → 118

35 tools did not complete a battery, all handled as clean, logged skips — none crashed the run:

No skip was distributed evenly — several came in visible clusters against the same backend (e.g. three coworking-space tools rate-limited back to back, five e-commerce tools failing identically to a single unreachable server), consistent with skips being a property of specific third-party infrastructure at run time rather than random attrition. This report does not attempt to correct for that clustering; it's recorded as observed.

9. A corpus-construction finding worth flagging

Two tools — check_agent_discovery and brand_audit_get_report — appear twice each in the scored set, both times under the same registry_name (com.blackveilsecurity/dns) but two different identity keys: pkg:npm:blackveil-dns and repo:https://github.com/madaburns/bv-mcp. Their scores are close but not identical (38.1 vs 36.5; 36.9 vs 36.6) — almost certainly the same underlying server, published to the registry twice under different metadata completeness, canonicalized differently, and independently selected into the curated corpus because selection dedupes by identity key, not registry_name.

This affects 4 of 129 tools (~3%) — not large enough to move any headline number — but it's a real limitation of the identity key as currently derived, not a scoring bug, and it's recorded here rather than smoothed over.

10. Method

Probe battery: reliability, schema conformance, malformed input, timeout honour, cancellation, determinism (static-result candidates only), concurrency at 2 and 4, latency — all against a fresh session per tool, read-only tools only. Side-effect classification is manual, never inferred from readOnlyHint.

Trial budget: ~45–52 calls per tool (20 shared reliability/schema/latency trials, ~2–4 malformed, 2 timeout, 2 cancellation, 4 determinism where applicable, 10 concurrency-at-2, 10 concurrency-at-4). Cost-sensitive tools (detected from description text — payment, booking, or similarly consequential language) are capped at 5 reliability-only trials, which is why 11 tools in this run never reach the 20-trial publish floor — that's the cap working as designed, not a shortfall.

Scoring (pure functions, no I/O): behaviour = Σ(component × weight) ÷ Σ(weights of scored components). Latency is normalized against a per-category P95 baseline computed from this run's own data (never a global scale — browser automation is legitimately slower than a filesystem read). Below 20 trials, no score publishes; trials_n is always stored regardless.

Orchestration: two passes. Pass 1 runs every tool's battery and persists each tool's trials immediately — a crash mid-run costs only the tool in flight. Pass 2 computes cross-tool category latency baselines (which need every tool probed first) and persists scores. A 10-minute ceiling on the whole per-tool battery guards against an orphaned promise from inside the MCP SDK's own reconnection logic that a per-request timeout does not catch; this run hit that ceiling 5 times, each time recovering cleanly and moving to the next tool (§8).

This run: single invocation, 2026-08-23, 164 eligible tools, ~2h20m wall clock. The methodology version is stamped on every trial and every score.

11. Reproducibility

Every scoring function is pure and unit-tested independent of live data. Re-running against the live corpus will not reproduce byte-identical numbers — third-party servers change, and this report's own §8 shows measurable run-to-run variance from transient failures alone — but will reproduce the method.

The curated corpus was 200 tools, Class A only, selected deterministically: a per-category cap for diversity, a per-server cap so no single server dominates, and no popularity signal in the selection at all. 164 of those were classified read-only and eligible for the battery; 36 were mutating and never probed.

12. What this report could not determine

• This is one run, not a time series. Score history is recorded, but one day of data says nothing about drift, stability, or whether a tool's score today is representative of its score next week. It becomes meaningful with repetition, not on day one.
• Not a strict replication of Digital Applied. Different corpus construction (Class A only, curated for category diversity, drawn from a 9.8%-coverage census sample — the August 2026 sample), different trial counts, different probe composition, and their methodology is not published in enough detail to match precisely. The comparison in this report is directional, not a controlled replication — read the headline section as "here's how the shape compares," not "here's the same experiment rerun."
• Category taxonomy is a rough heuristic (§7's 10-bucket approximation). Small per-category n (6–15) means a category median can be dominated by one or two badly broken tools on a single server — not evidence about the category as a whole at ecosystem scale.
• Skip clustering (§8) is not corrected for. If a category happens to lean on a server that got rate-limited, that category's apparent coverage and score are affected by an operational event, not a property of the tools themselves. Not disentangled here.
• The identity-key collision in §9 likely affects other servers in the wider registry beyond the one instance found by inspection here; this report did not run a systematic scan for it.
• No unannounced divergence check yet. The spot-check for divergence between announced and unannounced probing is not built. Every result in this report is announced; whether any operator serves different behavior to a known Vouch prober is not yet measurable.