Kaminari Ad

Rule testing API

Preview unsaved detection rules against stored scans — one rule at a time, or up to 20 rules across 50 scans in a single call.

Before you save a detection rule, you can run it against scans you already have and see what it would tag. Nothing is saved and no scan is changed.

Both endpoints need the custom_rules.manage permission and accept the same rule fields as creating a rule: rule_type, config, target (default page) and, for AI rules, name — the label the AI prompt quotes.

One rule, one scan

POST /api/v1/custom-rules/test with the rule fields plus scan_id returns matched, the tags the rule produced and elapsed_ms. For an AI rule, llm_failed is true when the model returned no verdict.

Many rules, many scans

POST /api/v1/custom-rules/test-batch tests up to 20 rules against up to 50 scans in one call:

{
  "rules": [
    {
      "id": "8d5c1d1e-2f5e-4f6a-9a51-6a9e0f0c3b21",
      "rule_type": "stopword_content",
      "config": { "contains": ["casino bonus"] },
      "target": "page"
    }
  ],
  "scan_ids": ["0f1c...", "5a7e..."]
}

id is optional and only echoed back as rule_id, so you can match results to rules on your side; it does not look up a saved rule. The most common use is one rule against many scans. Each scan is loaded once, and every rule aimed at it is evaluated together.

The reply is a flat list of cells, one per rule and scan, ordered by rule and then by the order of scan_ids, plus counts:

{
  "results": [
    {
      "index": 0,
      "rule_id": "8d5c1d1e-2f5e-4f6a-9a51-6a9e0f0c3b21",
      "scan_id": "0f1c...",
      "matched": true,
      "tags": [{ "tag_slug": "test", "detail": "casino bonus" }],
      "elapsed_ms": 412,
      "error": null,
      "error_code": null
    }
  ],
  "summary": { "total": 2, "matched": 1, "failed": 1, "deadline_exceeded": 1 }
}

index is the rule's position in rules.

Cells that did not run

A problem with one cell never fails the whole call. Such a cell has matched: false, a human-readable error, and an error_code you can branch on:

error_codeMeaning
deadline_exceededThe batch ran out of time before this cell finished. Send it again.
timed_outThe rule hit its own time limit on this scan.
llm_failedThe AI model returned no verdict.
content_prohibitedThe AI provider refused to analyse this scan's content.
scan_not_foundThe scan does not exist or belongs to another organization.
invalid_ruleThe rule could not be read when it was run.
failedThe cell could not be evaluated for another reason.

Time limit and partial results

The batch always answers within its request deadline, about 110 seconds. Regex, stopword, blacklist and combo rules take well under a second per scan. An AI rule takes a few seconds per scan. One AI rule on 50 scans normally finishes well within the limit.

When the deadline arrives first, the cells still running come back as deadline_exceeded and summary.deadline_exceeded counts them. The other cells are final. To finish the job, send just the deadline_exceeded rule and scan pairs in a new call. Many AI rules over many scans are better split into several calls from the start.

Errors for the whole call

  • 422 — over 20 rules or 50 scans, an empty list, or an invalid rule (the message names it, e.g. rules[2]: ...). Nothing has run.
  • 429 with code checking.rule_test_batch_in_progress — your organization already has a batch running. Retry when it finishes.
  • 403 — the credential lacks custom_rules.manage.
  • 504 with code checking.rule_test_batch_timed_out — the scans themselves could not be read before the deadline, so no cell ran. Retry later.

Limits that carry over from a single test

regexp_request_body rules only find matches on scans from the last day, and regexp_request_url results on stored scans are best-effort. See Matching sub-resource contents for why.

← All documentation