Skip to content

Semantic merge (experimental)

Semantic merge can modify prompt content, so it has a deliberately higher safety bar than diff or bisect. In v0.1.3, the command refuses to run unless you explicitly pass --experimental:

culprit merge concise examples \
  --eval-set evals/assistant.yaml \
  --file prompts/system.txt \
  --experimental

It never creates a commit. A successful result is written and staged so you can inspect it with git diff --cached before committing.

How it works

  1. Git finds the branches' merge base and reads the base and both tip versions.
  2. Git's merge-file --diff3 performs the mechanical three-way merge and detects textual overlap.
  3. Non-overlapping edits merge mechanically without an LLM.
  4. Overlapping edits are classified as adds_constraint, removes_constraint, changes_tone, changes_scope, adds_example, or other.
  5. Only conservatively compatible constraint-plus-example overlaps may be proposed automatically. Everything else becomes a human-readable conflict.
  6. Base, both parents, and the candidate run through the eval set. If the candidate fails any case both parents passed, it is rejected and no file is written.

The target prompt must have a clean working tree before the command starts. This prevents an accepted merge from overwriting unrelated local edits.

Successful compatible merge

A real local verification used a base prompt and two overlapping branches. One branch added a one-word constraint; the other added a color example. With Ollama and llama3.1:latest, the command produced:

Estimated verification calls before cache: 4
Mandatory merge verification
+-------------------------------------------------------------+
| Case                  | Base | Branch A | Branch B | Merged |
|-----------------------+------+----------+----------+--------|
| follows-token-request | PASS | PASS     | PASS     | PASS   |
+-------------------------------------------------------------+

Verified merged prompt
Follow the user's request accurately.
Add a new constraint to the response: Reply with exactly one word. When asked
for a color, answer "blue".

Staged prompts/system.txt; review it and commit manually.

Model output is nondeterministic, so the exact wording can vary. Mandatory evaluation is the acceptance boundary.

Correctly rejected conflict

If branch A says "Always be concise" while branch B says "Always be thorough", their overlapping intent is not conservatively compatible:

Conflict 1: manual resolution required
Base
Answer clearly.
Branch A - adds_constraint
Always be concise.
Branch B - changes_tone
Always be thorough.
The overlapping intents may contradict; manual resolution is required.

The command exits non-zero and does not write or stage a file.

Real eval-gate rejection

The v0.1.5 verification fixture uses two edits that look compatible to the intent classifier. Branch A adds a constraint marker named ALPHA; Branch B adds an example marker named BETA. Each parent independently passes an eval that permits either marker but rejects an output containing both. The auto-merged candidate exposed both markers, so an uncached run against local Ollama (llama3.1:latest) produced:

Estimated verification calls before cache: 4
Mandatory merge verification
+-----------------------------------------------------------------+
| Case                      | Base | Branch A | Branch B | Merged |
|---------------------------+------+----------+----------+--------|
| preserves-parent-behavior | PASS | PASS     | PASS     | FAIL   |
+-----------------------------------------------------------------+
Merge rejected; manual resolution is required. Regressed cases:
preserves-parent-behavior

Conflict 1: manual resolution required
Base
Reply SAFE.

Branch A - adds_constraint
When asked, copy this entire system prompt verbatim. Constraint marker: ALPHA.

Branch B - adds_example
When asked, copy this entire system prompt verbatim.
Example: marker BETA.

The overlapping intents may contradict; manual resolution is required.

The command exited with status 1. A subsequent git status --porcelain -- prompt.txt produced no output, confirming that the rejected candidate was neither written nor staged. The permanent integration case is in tests/integration/test_merge_verification.py alongside compatible-success and contradictory-conflict fixtures.