crawl_create

io.github.ZenRows/zenrows-mcp · Verify this server

Start a crawl from one URL (Zenrows Crawl API). Crawl follows the links in each page's HTML up to depth hops (no depth limit when depth is omitted), keeps the URLs that match include_patterns / exclude_patterns, and stays on the start URL's registrable domain (subdomains count). Use crawl when you have one start page (a listing, category, blog index) and need the URLs behind it, optionally with each page's HTML. For a single page, use scrape. Each page fetched is billed as one scrape on this account; max_pages bounds the cost (default 100), max_items bounds how many URLs are kept (default 100). Without depth, the crawl keeps following links until max_pages or max_items stops it or nothing is left to open, so set max_pages to what you are willing to spend. The start page itself is never a result, but its fetch is billed: a crawl whose include_patterns / exclude_patterns match nothing still costs that one scrape. A crawl whose pages held nothing new to follow (no link, only links to other sites, or only links the patterns reject) ends completed with no results and no error. When patterns are set, check them against the URLs the start page links to. Set output_format "html" or "markdown" to also store each kept URL's page, then read it with crawl_content. Returns crawl_id, status and the crawl (with coverage) at once; the crawl runs asynchronously. Call crawl_wait, then crawl_results. Set idempotency_key to make a retry safe: the same key returns the crawl the first request created instead of starting another. CRAWL_NOT_ENABLED: Crawl is not enabled for this account; do not retry. CRAWL_TOO_MANY_CRAWLS: the account has reached its limit of active jobs (3 by default), shared with its Batch jobs; retry after retry_after seconds.