dev.workers.mike-tusa.robotsgate/robotsgate

name:dev.workers.mike-tusa.robotsgate/robotsgate

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

transport:: remote
credential class:: open

Owner verification

Not yet verified. Verifying proves you control this server and is free, permanently — it never changes a published score.

Start verification →

Tools

check_site
Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, which registry AI crawlers are allowed or blocked, plus validation errors and warnings. Read-only: one GET of /robots.txt with a RobotsGate user-agent. Same engine as GET /api/check.
shallow
generate_robots
Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps and custom rules. Returns the file text, a per-crawler summary and notes. Read-only: it only returns text and changes nothing. Same engine as POST /api/generate.
shallow
list_crawlers
Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a link to the operator's documentation, plus unverified entries and notes. Read-only, no network access. Same data as GET /api/crawlers.
shallow
validate_robots
Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same engine as POST /api/validate. Up to 50,000 characters over MCP (the whole JSON-RPC request must also fit in 64 KB); for larger files use POST /api/validate or check_site.
shallow

Embed this server’s score

Tool count and median score across every tool in this server’s corpus — honest in a way a single cherry-picked tool’s badge wouldn’t be.

[![Vouch score](https://vouch.tools/api/servers/b2afb6df-f99d-465c-a911-5e485c5beb4e/badge.svg)](https://vouch.tools/servers/b2afb6df-f99d-465c-a911-5e485c5beb4e)