Methodology v0.2.0
How Vouch measures a tool
Vouch scores individual MCP tools — not servers — on measured behaviour, from real invocation trials. This page is the methodology: what’s measured, how it’s scored, and what it deliberately refuses to do. It is versioned; this version is stamped on every score it produced, and a later version never silently rewrites history.
The probe battery
| Probe | Question | Tier |
|---|---|---|
| Handshake | Initialises and responds to tools/list? | Shallow — corpus-wide |
| Schema declaration | Are input/output schemas present and well-formed? | Shallow. Most tools declare no outputSchema — a finding, never a failure |
| Sample call | Does the tool's one heuristically-safe sample invocation actually succeed? | Shallow — one tool per server only (the census probe calls at most one), scored where attempted |
| Schema conformance | Does a valid call return a response matching the declared output schema? | Deep; only scored where an output schema exists, else n/a with reweighting |
| Malformed input | Fails gracefully rather than hanging? | Deep |
| Timeout honour | Declared timeout respected? | Deep |
| Cancellation | In-flight cancel actually stops work? | Deep |
| Determinism | Repeated identical calls consistent? | Deep, static-result tools only |
| Concurrency | Pass rate at 1, 2, 4 concurrent | Capped at 4; higher only with explicit operator consent |
| Latency | P50 and P95 over N trials | Normalised against a per-category baseline |
| Auth enforcement | If auth is declared, is it enforced? | Shallow where testable |
Side-effect classification is manual, not inferred. Reading tool names, descriptions, or a self-reported readOnlyHint annotation is circular — that’s exactly the signal this project distrusts. Mutating tools are skipped and marked untested; read-only classification for the deep corpus is done by hand, with reasoning recorded per tool.
The behaviour score
| Component | Weight | Normalisation |
|---|---|---|
| Reliability | 35% | Pass rate, direct |
| Schema integrity | 25% | % validating; dropped if no output schema declared |
| Failure behaviour | 15% | Ordinal: graceful + timeout + cancel all correct = 100, each missing −33 |
| Latency | 15% | Log-scaled against the category P95 baseline, clamped 0–100 |
| Concurrency | 10% | Pass rate at 4 concurrent ÷ pass rate at 1 |
behaviour = Σ(component × weight) ÷ Σ(weights of scored components)
A component that doesn’t apply (no output schema declared, no latency baseline yet, concurrency not run) drops out of both the numerator and the denominator — the rest reweights to fill the gap. It is never treated as a 0.
Latency is normalised per category, not globally. Browser automation is legitimately slower than a filesystem read; a single global scale would encode “browser tools are bad” as a measurement rather than state a fact about the category.
Below 20 trials, no behaviour score publishes. The tool’s state is insufficient_data, and trials_n is shown alongside every score regardless of whether it cleared the threshold — never a blank, never a zero.
When a server leaves the registry, its tools are marked, not deleted. They drop out of search, rankings, and the corpus counts, but every report card stays reachable with a no longer in the registry banner. A published score is never recomputed or removed after the fact — it reflects what was measured while the server was listed.
Ranking within tiers
A tool is measured at exactly one tier at a time: shallow (manifest, schema, reachability, declared-vs-enforced auth) or deep (the full battery above). Ranking happens within a tier — deep-tier results rank against deep-tier results, and a shallow-tier tool is never pushed down a list for being shallow. It’s shown in its own band with its state stated plainly.
This exists because the alternative is a paid-badge model arriving through the back door: if a more-measured tool ranked higher simply for having more data behind it, then anything that bought its way to more measurement — verification, in Vouch’s design — would be buying rank rather than buying accuracy. Verification grants access to be measured; the trials decide the rest.
The adoption score
Server-level, not tool-level, until real-usage telemetry is available:
| Component | Weight | Normalisation |
|---|---|---|
| Dependents | 40% | log₁₀(count+1), percentile-ranked across the corpus |
| Downloads (30 days) | 30% | log₁₀(count+1), percentile-ranked |
| Maintenance | 30% | Days since last release: ≤30d = 100, linear decay to 0 at 365d |
Behaviour and adoption are never blended into one number. A widely-used, poorly-behaved tool does not get to launder its popularity into an appearance of quality — the two axes are published separately, always.
Because adoption data only exists at package/server granularity, every tool in a server currently carries the same adoption figure. It is labelled as server-level context wherever it appears — equal visual weight to a tool-level number, unequal epistemic status, stated rather than smoothed over.
Five prohibitions
- Never blend the axes. Behaviour and adoption are published separately and never averaged.
- Never score what wasn’t measured. Untested and declined are visible states, not zeros; trials_n is always shown.
- Never use a competitor’s verdict as ground truth. Every label traces to a Vouch trial or a primary source.
- Never let payment or verification move a public number. Verification grants access to be measured; the trials decide the rest.
- Version everything. The methodology version is stamped on every score; changes are logged and dated; historical scores are never silently rewritten.
Publishing it
Public, versioned in git, linked from every number this methodology produced. The underlying corpus analyses are published in full at /research.
What’s published today: the per-trial evidence behind every behaviour score is queryable per tool via the public API. The adoption axis described above is not currently computed — report cards omit it rather than show a placeholder. Owner verification (DNS-TXT or repo-file ownership proof) is live; it makes a server eligible for the deep battery but does not by itself change any published number.
Agent traffic instrumentation
Separately from anything above: Vouch records who calls its agent-facing surfaces — AI crawlers reading this website, and direct calls to the MCP server — and how confidently it knows who they are. This is traffic analytics, not measured behaviour. It is never blended into a score, and it never will be: query volume measures Vouch’s own traffic, not a tool’s quality, and folding “which tools get asked about most” into any score would be exactly the popularity-laundering prohibition 1 forbids, arriving through a side door.
Two things stated explicitly: the raw text of a vouch_find query is kept, length-capped — by construction these strings are tool and package names, not personal data, and what agents search for (and don’t find) is a real product input. Caller IPs are never stored — only a salted hash, enough to count distinct callers without keeping an identifier.
The human-vs-agent split isn’t published yet — not enough classified traffic has accumulated to put a number above noise.
Changelog
- v0.2.0 (2026-08-24) — added the shallow-tier score (schema integrity + sample-call outcome), computed from signal the census probe already collects, and the opt-in real-usage telemetry pipeline (k-anonymised, reported separately, never blended). No change to the deep behaviour score’s formula or weights.
- v0.1.0 (2026-08-23) — first published methodology. Behaviour score formula, component weights, and the 20-trial publish threshold as described above; produced the first deep-probe corpus (129 tools).
See it in action
A page that explains how a score is computed is more useful with one actually being computed. Three real report cards:
Back to the search, the full corpus research, or — if you operate a server this describes — what this means for you and how to opt out.