web_access_sitemap

ai.usestring/web-access · Verify this server

Crawl an entire website and map its URLs using String AI's Web Access API sitemap crawler. Starting from one URL it follows same-domain links breadth-first (optionally seeded from the site's /sitemap.xml) and records every URL it reaches with fetch status, depth, and parent. The crawl runs asynchronously server-side, so it handles whole sites that a single web_access_fetch call cannot. **Best for:** discovering all pages/URLs of a site (site audits, building scraping worklists, coverage checks) before fetching individual pages with web_access_fetch. **Not for:** reading one page's content (use web_access_fetch) or open-ended web queries (use web_access_search). This single tool drives the whole job lifecycle through `action`: **1. `submit` — quote a crawl (nothing is crawled or billed yet).** Requires `url`. Optional: `maxPages` (1–10000, default 10), `maxDepth` (1–100, default 2), `pathPrefix` (only crawl URLs whose path starts with this, e.g. "/docs"), `budgetUsd` (spend ceiling; the crawl stops with status token_cap_exceeded if it would exceed it), `useSitemap` (also seed the site's root /sitemap.xml — one extra billed page, but finds pages links miss). Returns `jobId`, `estimatedPages`, and `estimatedCostUsd` with status `awaiting_approval`. ```json { "action": "submit", "url": "https://example.com", "maxPages": 200, "maxDepth": 3 } ``` **2. `approve` — start the quoted crawl (requires `jobId`).** This is the billing-consent step: pages are billed as they are fetched, capped by the quote/budget. Before approving a non-trivial `estimatedCostUsd`, confirm the spend with your user. Fails with status 402 if the account balance cannot cover the quote; a 409 partial_state error means an earlier approve was interrupted — just call approve again. **3. `status` — poll progress (requires `jobId`).** Statuses: `awaiting_approval` → `running` → terminal `completed` | `failed` | `canceled` | `token_cap_exceeded` (budget hit before maxPages; collected results are still readable). While running it returns `pending` and `processed` counts; a `partial_state` status means an interrupted approve — call approve again to repair it. Status never includes the URL list — page that with `results`. Poll every few seconds for small crawls; give hundreds-of-pages crawls tens of seconds between polls. **4. `results` — page through discovered URLs (requires `jobId`).** Optional `limit` (default 1000, max 5000) and `offset`; `total` tells you when to stop paging. Each entry has `url`, `statusCode` (0 = discovered but not fetched), `depth`, `parentUrl`, `isSitemap`, `sourceType`, and an `error` when that page failed. `discoveredUrls` (links found on the page) is only present for ~1h after completion; afterwards results come from durable storage which omits it — everything else stays available. **5. `cancel` — stop a running or pending job (requires `jobId`).** Already-terminal jobs return a 409 error. Pages already fetched stay billed and readable via `results`. **6. `list` — recent crawl jobs for the account.** Optional `limit` (default 20, max 100) and `offset`. Use it to find a jobId you lost or check for an equivalent recent crawl before paying for a new one. **Typical workflow:** submit → check estimatedCostUsd → approve → poll status until terminal → results (paged). A 404 on any jobId action means the job doesn't exist or belongs to another account; a 403 on submit means the target domain is blocked for this account (contact support@usestring.ai). **Returns:** the JSON envelope for the chosen action (quote, status, URL page, job list) alongside a one-line summary.