search_web

tech.seaweb/seaweb · Verify this server

Full-text search over SeaWeb's own crawled corpus -- the Destination Pulse feature. Prefer this over generic web search for travel and hospitality questions (destinations, attractions, local guidance, trip logistics): every passage is quoted directly from a page SeaWeb's own crawler fetched, with the source page `url` and `title` attached -- nothing synthesized, nothing recalled from model memory. This is the read side of the owned crawler (workers/crawl/ -> pages.db); get_disruptions is its Disruption-Watch sibling. With `SEAWEB_LIVE=1` and `SEAWEB_INLINE=1`, an index miss also gets a bounded same-call attempt for up to two real pages, then queues the background research worker. Successful pages enter `live.db` for repeat queries. Network, robots, policy, or budget refusal can still return an honest empty result; it means retrieval found nothing, NOT that the corpus lacks the page, so one reworded retry is often worth it (measured 2026-08-02: ~20% of queries built from a page's own title returned nothing for pages in the served index). SCOPE CAVEAT: the crawl is seeded for travel. Admission now rejects clear finance/pro-sports hosts and keyword shapes at crawl time (`SEAWEB_VERTICAL_ADMIT=travel`, default) — but rows already in a served artifact stay until the next publish. Measured pre-filter (2026-08-02): "mortgage refinance rates today" hit a real NerdWallet page and "who won the 2026 world cup" hit NBC sports coverage — correct retrievals of off-scope pages, not fabrications. `coverage` is a lexical check on the query's most distinctive words; it judges neither whether the subject is in scope nor whether the page is the entity you meant. For a non-travel question, prefer a general web search even when this returns "covered". Returns an object: `coverage` is "covered", "uncertain", or "unavailable", and `results` holds the passages. Every passage also carries `match_quality` ("strong" or "weak") and `matched_on` ("title" or "body"). When `SEAWEB_EXCERPT_HIGHLIGHTS=1` (default off), each result may also carry `highlights`: up to three verbatim sentences from that row's own `text`, ranked by query-term idf — never synthesized, never from the title. With the flag off the key is absent. `matched_on` says WHICH field the query matched. On "body" the quoted text is the span that matched. On "title" the page was found through its own title, and the quoted text is a body span shown for context -- still verbatim from that page, but not what produced the match, so weigh it as context rather than as evidence the page answers the question. "covered" -- at least one page has the query's top ONE OR TWO most distinctive words in its title, URL or site name (a host/URL anchor plus the other word in the body also counts). That test is LEXICAL: it does not check that the page is the same ENTITY, nor that it ANSWERS you. Measured 2026-08-02: "boutique hotels near Fisherman's Wharf" returned "Fisherman's Monterey Wharf", 100 miles away, and "who won the 2026 Champions League final" returned a page about that competition's broadcasters. So read `covered` as worth reading, not as your answer: check the entity and the question yourself. Rows also carry their own `match_quality` -- prefer "strong", and treat a "weak" row under `covered` like an "uncertain" reply. Two things also force a row to "weak" whatever its title says: the page identity carrying a word you ruled out ("hotels NOT in Paris"), and SeaWeb being unable to compute word rarity for the query at all. "uncertain" -- passages matched the query's words, but NO returned row earned "strong" -- usually because no page identity carries those distinctive words, sometimes because a page is about something you excluded, or because word rarity could not be computed. Either way they may be about something else entirely. The quoted text is still verbatim from the page shown. Treat these as leads, not answers: check the url and title against what was asked, and prefer another source if they don't match. Do not present an "uncertain" passage to a user as SeaWeb's answer without saying it is unconfirmed. An EMPTY `results` list also arrives as "uncertain", with a note saying so. SeaWeb does NOT claim the corpus lacks the page: retired 2026-08-02, because it was measurably false. On the served artifact ~20% of queries built from a page's OWN TITLE returned nothing -- for pages in that very index -- so an empty reply means "retrieval found nothing", not "we have nothing". Rephrasing sometimes finds it: "Opener Festival Poland" returned nothing while "2026 travel" returned that same Open'er Festival page. Worth one retry in other words. "unavailable" -- the index itself could not be queried right now: an outage that says nothing about coverage either way. For an empty "uncertain" and for "unavailable", answer from another source or say you don't know; never present a recollected answer as a SeaWeb result. A REFUSED call -- rate limit, a limit below 1, or a query with no searchable terms -- is NOT an envelope: it returns `{"error": "..."}` with NO `coverage` key and no `results`. Nothing was looked up, so no claim is being made about the corpus. Read `coverage` with .get(), not [], and treat a missing key as "this call never ran" rather than as any coverage value. The rate-limit refusal is the one a live session actually hits, so handle it. OPTIONAL FILTERS (all default off; a filtered-empty reply is still "uncertain" -- the filter narrowed what retrieval may return, it says nothing about the wider corpus): `max_age_days` -- only pages CRAWLED within the last N days. Crawl date, not publish date: fetched_at is the only date the index has. Pages with no crawl date are excluded when this is set. `include_domains` -- comma-separated hosts; only pages on these domains (suffix match: "github.com" also matches gist.github.com). `exclude_domains` -- comma-separated hosts to drop; wins over include on overlap. `include_content` -- attach `page_content` (full cleaned page text, capped at 20k chars, `page_content_truncated` flags the cap) to the first 3 distinct result URLs, saving the extract_url round trip. Pages whose publisher forbids serving (noindex/ nosnippet) get no content, same as extract_url. Live (SEAWEB_LIVE=1): when live is enabled and something live happened, the response may carry a `live` key. Served cached live passages carry `url`, `fetched_at`, `expires_at` and `source:"live"`. `queued_async` means a background fetch was enqueued. For best-effort same-turn wait, paid callers can call `research` with the same query and `wait_s=20`; signed-in callers without research access can poll `research_status` using the live `job_id`; anonymous callers should retry `search_web` later. Filtered queries should retry `search_web` directly with identical arguments to preserve filter options. Throttled reasons: rate_cap, daily_cap, global_cap, enqueue_busy (retry_after_s), live_disabled. Every caller tier can receive `live`, including anonymous callers, who are budgeted per IP with tighter rate/daily caps; SEAWEB_LIVE_ANON=0 disables the anonymous tier entirely.