robots_txt_check
shallowdev.workers.adamhaleworks.tools/web-research · Verify this server
Crawl preflight: whether each user agent may fetch each URL under the site's robots.txt (RFC 9309, longest match, '*' fallback), with crawl-delay, the sitemaps robots.txt lists and whether the site publishes /llms.txt. Default agents: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Applebot-Extended, Bytespider and *. Paid only: up to 5 URLs per call for $0.001 USDC. Paid calls made without a payment return isError with the x402 v2 PaymentRequired object (amount in USDC atomic units, network, payTo, HTTP endpoint); retry with the signed PaymentPayload in params._meta["x402/payment"]. Invalid input and calls where every item fails are never charged. Fetches follow robots.txt, use the User-Agent AdamHaleTools/1.0 and refuse private or internal addresses.
1 trials · measured today
robots_txt_check scores 100.0/100 on Vouch's measured behaviour index, from 1 real invocation trials against dev.workers.adamhaleworks.tools/web-research, measured 20 Oct 2026 under methodology v0.2.0. Every measured component scored 100.
Component breakdown
| Component | Weight | Value |
|---|---|---|
| Reliability | 35% | not applicable |
| Schema integrity | 25% | 100.0 |
| Failure behaviour | 15% | not applicable |
| Latency | 15% | not applicable |
| Concurrency | 10% | not applicable |
Tool details
- Transport
- remote
- Credential class
- self-provisionable
- Input schema
- not declared
- Output schema
- not declared
- Side-effect classification
- unclassified
Score history
| Day | Score | Tier | Methodology |
|---|---|---|---|
| 2026-10-20 | 100.0 | shallow | v0.2.0 |
Probe evidence
| Probe | Outcomes |
|---|---|
| schema_integrity | pass: 1 |
Raw request/response logs are not archived yet — the outcome counts above are drawn directly from every recorded trial.
Embed this score
Available for every tool, scored or not — not a verification perk. Always links back to this page.
[](https://vouch.tools/tools/49fd708d-cb66-43e9-92e0-4a79a4fc4e08)