Behavioral bisect

culprit bisect searches only commits that changed the configured prompt file. Git remains the source of history, while the existing eval runner and cache classify each midpoint.

Worked five-commit example

Consider a repository where prompts/guard.txt changed five times:

Commit	Prompt behavior	Expected classification
`6673a3d`	Answers with `SAFE`	good
`7d35dfd`	Answers with `SAFE`	good
`1665e6f`	Answers with `UNSAFE`	first bad commit
`e31e8be`	Still answers with `UNSAFE`	bad
`74f0664`	Still answers with `UNSAFE`	bad

The eval set contains a deterministic assertion:

cases:
  - id: safe-answer
    input: Return the required token.
    assertions:
      - type: exact
        value: SAFE

Start the search:

culprit bisect start \
  --good 6673a3d \
  --bad 74f0664 \
  --file prompts/guard.txt \
  --eval-set evals/safety.yaml

Output captured from the five-commit example using local Ollama with llama3.1:latest:

Estimated provider calls: 3

Culprit found
Commit:  1665e6fe3575434234ae485fb7b0af9290f4cfdd
Author:  Demo User <demo@example.com>
Date:    2026-09-05T10:00:00+05:30
Message: break required response
Flipped cases: safe-answer

Behavioral change summary
The safe-answer score fell from 1.00 to 0.00 and its behavior changed from SAFE to UNSAFE.

The estimate is ceil(log2(file-changing commits)) * number of cases. Above max_calls_without_confirmation, culprit asks once before starting; use --yes only for deliberate automation.

Automated and manual modes

When every case has at least one exact, contains, or regex assertion, classification is automatic. A commit is bad when a case that passed at the good boundary fails.

When cases contain only llm_judge assertions, culprit prints the midpoint scores and asks whether that commit is good or bad. This matches interactive git bisect behavior without silently trusting a judge score.

Continue after interruption

Progress is saved to the git-ignored .culprit/bisect-state.json. If the process is interrupted, run:

culprit bisect continue

The command resumes the saved range. If interruption happened during a provider call, any completed evaluations are loaded from .culprit/cache.sqlite3, preventing repeat spend.