intel_robots

ai.tunnelmind/data · Verify this server

Retrieves the target domain's `robots.txt` file and parses it for AI crawler disallow rules. Specifically detects policies for known AI crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, etc.) and returns a structured summary of the crawling policy. Use this tool when: - You need to know whether a domain has opted out of AI training data collection. - You want to check if a specific AI crawler is blocked before citing the domain. - You are building a dataset of AI-accessible vs AI-blocked domains. Do NOT use this tool when: - You want training opt-out signals beyond robots.txt (TDM reservation, noai meta) — use `intel_optout` instead. - You want the full technology stack — use `intel_stack` instead. - You need tracker database data — use `get_domain` instead. Inputs: - `domain` (query, required): Domain to probe. Returns: - `robots_txt_found`: false if the domain returned 404 or the file is empty. - `ai_crawlers_blocked`: list of AI crawler user-agent names that are disallowed. - `all_blocked`: true if `User-agent: *` with `Disallow: /` is present. - `raw`: first 4096 characters of the robots.txt file. Cost: - Free. No API key required. Latency: - Typical: 1-2s, p99: 6s.