test_skill_trigger

io.agent4/agent4-tenant · Verify this server

Reality-check whether your prompts actually trigger tool calls (dry-run) — run this after writing/changing a skill instead of counting corpses in production. Replays your messages N times against the **production** system-prompt assembly, tool schemas and this tenant's actual model routing, capturing only the model's tool-call decision: **tool side effects are NOT executed**, no session is stored. Tokens count toward the tenant quota (messages≤5, samples≤5, at most 25 calls per invocation — pick test messages carefully). Two modes for the skill's two battlefields: - loaded=false (default): first turn, skill not loaded — tests whether the trigger in description works; - loaded=true: simulates post-load_skill — tests the quality of instructions (incl. few-shot examples). Returns per-message hit counts plus claimed_without_call (the model said "noted" WITHOUT calling the tool — the worst failure, fix first). Cover edge cases in your test messages: numbers with spaces, buried in long questions, corrections, email-only. The loop: create_skill → check warnings (static lint) → test_skill_trigger (dynamic reality check) → adjust description / add examples → re-test until the hit rate holds.