singularity-forge/docs/dev/ADR-001-branchless-worktree-architecture.md

ADR-001: Branchless Worktree Architecture

Historical note: this ADR predates the current DB-backed planning-state direction. Where it says markdown is truth and DB is cache, that statement is superseded by docs/adr/0000-purpose-to-software-compiler.md and docs/adr/0001-promote-only-sf-state.md: structured .sf state is authoritative at runtime, and markdown is a projection when structured state exists.

Status: Accepted — partial drift Date: 2026-03-15 Revised: 2026-05-02 — partial drift documented; code migration incomplete Deciders: Lex Christopherson Advisors: Claude Opus 4.6, Gemini 2.5 Pro, GPT-5.4 (Codex)

Context

SF uses git for isolation during autonomous coding sessions. The current architecture (shipped in M003, v2.13.0) creates a worktree per milestone with slice branches inside each worktree. Each slice (S01, S02, ...) gets its own branch (sf/M001/S01) within the worktree, which merges back to the milestone branch (milestone/M001) via --no-ff when the slice completes. The milestone branch squash-merges to main when the milestone completes.

This architecture replaced a previous "branch-per-slice" model that had severe .sf/ merge conflicts. M003 solved the merge conflicts but retained slice branches inside worktrees, inheriting complexity that has produced persistent, user-facing failures.

Problems

1. Planning artifact invisibility (loop detection failures)

When research-slice or plan-slice dispatches, the agent writes artifacts (e.g., S02-RESEARCH.md) on a slice branch. After the agent completes, handleAgentEnd switches back to the milestone branch for the next dispatch. The artifact is on the slice branch, not the milestone branch. verifyExpectedArtifact() checks the milestone branch, can't find the file, increments the loop counter, retries, same result. After 3 retries → hard stop. After 6 lifetime dispatches → permanent stop. This burns budget and blocks progress.

Documented in the auto-stop architecture doc as "The Branch-Switching Problem."

2. .sf/ state clobbering across branches

.sf/ is gitignored (line 52 of .gitignore: .sf/). Planning artifacts (roadmaps, plans, summaries, decisions, requirements) live in .sf/milestones/ but are invisible to git. When multiple branches or worktrees operate from the same repo, they share a single .sf/ directory on disk. Branch A's M001 roadmap overwrites Branch B's M001 roadmap. SF reads corrupted state, shows wrong milestone as complete, or enters infinite dispatch loops.

The codebase has a contradictory workaround: smartStage() (git-service.ts:304-352) force-adds SF_DURABLE_PATHS (milestones/, DECISIONS.md, PROJECT.md, REQUIREMENTS.md, QUEUE.md) despite the .gitignore. This means .sf/milestones/ IS partially tracked on some branches but the gitignore claims otherwise. The code fights the configuration.

3. Merge/conflict code complexity

The current slice branch model requires:

• mergeSliceToMilestone() — 98 lines, --no-ff merge with withMergeHeal wrapper
• mergeSliceToMain() — 189 lines, squash-merge with conflict detection/categorization/auto-resolution
• git-self-heal.ts — 198 lines, 3 recovery functions for merge failures
• fix-merge dispatch unit — dedicated LLM session to resolve conflicts the auto-resolver can't handle
• smartStage() — 49 lines of runtime exclusion during staging
• Conflict categorization — 80 lines classifying .sf/ vs runtime vs code conflicts

Total: ~582 lines of merge/branch/conflict code across 3 files, plus the fix-merge prompt template and dispatch logic. This code exists solely because of slice branches.

4. Dual isolation modes

Branch-mode (git-service.ts:mergeSliceToMain) and worktree-mode (auto-worktree.ts:mergeSliceToMilestone) have parallel implementations with different merge strategies, different conflict handling, and different branch naming. Both paths must be maintained and tested. 11 test files exercise merge/branch/worktree logic.

5. Bug history

• v2.11.1: URGENT fix for parse cache staleness causing repeated unit dispatch (directly caused by branch switching invalidation timing)
• v2.13.1: Windows hotfix for multi-line commit messages in mergeSliceToMilestone
• 15+ separate bug fixes for .sf/ merge conflicts in the pre-M003 era
• Persistent user complaints about loop detection failures and state corruption

Decision

Eliminate slice branches entirely. All work within a milestone worktree commits sequentially on a single branch (milestone/<MID>). No branch creation, no branch switching, no slice merges, no conflict resolution within a worktree.

Track .sf/ planning artifacts in git. Gitignore only runtime/ephemeral state.

The Architecture

main ──────────────────────────────────────────── main
  │                                                 ↑
  └─ worktree (milestone/M001)                      │
       │                                            │
       commit: feat(M001): context + roadmap        │
       commit: feat(M001/S01): research             │
       commit: feat(M001/S01): plan                 │
       commit: feat(M001/S01/T01): impl             │
       commit: feat(M001/S01/T02): impl             │
       commit: feat(M001/S01): summary + UAT        │
       commit: feat(M001/S02): research             │
       commit: ...                                  │
       commit: feat(M001): milestone complete       │
       │                                            │
       └──────────── squash merge ──────────────────┘

Git Primitives Used

Primitive	Purpose
Worktrees	One per active milestone. Filesystem isolation.
Commits	Granular sequential history of every action.
Squash merge	Clean single commit on main per milestone.
Branches	Only main and milestone/<MID>. Nothing else.

Git Primitives NOT Used

Primitive	Why Not
Slice branches	Slices are sequential. Branches add complexity with no rollback benefit.
--no-ff merges	No branches to merge within a worktree.
Branch switching	Never happens. All work on one branch.
Conflict resolution	No merges within a worktree means no conflicts within a worktree.

.sf/ Tracking Model

Tracked in git (travels with the branch):

.sf/milestones/         — roadmaps, plans, summaries, research, contexts, task plans/summaries
.sf/PROJECT.md          — project overview
.sf/DECISIONS.md        — architectural decision register
.sf/REQUIREMENTS.md     — requirements register
.sf/QUEUE.md            — work queue

Gitignored (ephemeral, runtime, infrastructure):

.sf/runtime/            — dispatch records, timeout tracking
.sf/activity/           — JSONL session dumps
.sf/worktrees/          — git worktree working directories
.sf/auto.lock           — crash detection sentinel
.sf/metrics.json        — token/cost accumulator
.sf/completed-units.json — dispatch idempotency tracker
.sf/STATE.md            — derived state cache (rebuilt by deriveState())
.sf/sf.db              — SQLite cache (rebuilt from tracked markdown by importers)
.sf/DISCUSSION-MANIFEST.json — discussion phase tracking
.sf/milestones/**/*-CONTINUE.md — interrupted-work markers
.sf/milestones/**/continue.md   — legacy continue markers

.gitignore Update

Replace the current blanket .sf/ ignore with explicit runtime-only ignores:

# ── SF: Runtime / Ephemeral ─────────────────────────────────
.sf/auto.lock
.sf/completed-units.json
.sf/STATE.md
.sf/metrics.json
.sf/sf.db
.sf/activity/
.sf/runtime/
.sf/worktrees/
.sf/DISCUSSION-MANIFEST.json
.sf/milestones/**/*-CONTINUE.md
.sf/milestones/**/continue.md

Planning artifacts (milestones/, PROJECT.md, DECISIONS.md, REQUIREMENTS.md, QUEUE.md) are NOT in .gitignore and are tracked normally.

Consequences

Code Deletion

File	Lines Deleted	What's Removed
auto-worktree.ts	~246	mergeSliceToMilestone(), shouldUseWorktreeIsolation(), getMergeToMainMode(), slice merge guards
git-service.ts	~250	mergeSliceToMain(), conflict resolution, runtime stripping post-merge, ensureSliceBranch(), switchToMain()
git-self-heal.ts	~86	abortAndReset(), withMergeHeal() (merge-specific recovery) (still present — see Drift section)
auto.ts	~150	Merge dispatch guards, fix-merge dispatch path, branch-mode routing (partially still present — see Drift section)
worktree.ts	~40	getSliceBranchName(), ensureSliceBranch(), mergeSliceToMain() delegates (getSliceBranchName still present — see Drift section)
Test files	~11 files	auto-worktree-merge.test.ts, auto-worktree-milestone-merge.test.ts, merge-related test cases
Total	~770+ lines

Verified 2026-05-02: mergeSliceToMilestone and the original mergeSliceToMain (git-service.ts) are deleted. Several other items listed above are still present — see Drift section below. Current authoritative milestone merge path: src/resources/extensions/sf/auto-worktree.ts:1616 (mergeMilestoneToMain). A separate newer function mergeSliceToMain at src/resources/extensions/sf/slice-cadence.ts:92 was added post-ADR for the slice-cadence collapse feature (#4765) and is unrelated to the deleted branch-era function.

What mergeMilestoneToMain() Becomes

The function simplifies dramatically:

1. Auto-commit any dirty state in worktree
2. chdir back to main repo root
3. git checkout main
4. git merge --squash milestone/<MID>
5. git commit with milestone summary
6. Remove worktree + delete branch

No conflict categorization. No runtime file stripping. No .sf/ special handling. Planning artifacts merge cleanly because they're in .sf/milestones/M001/ which doesn't exist on main until this merge.

What smartStage() Becomes

The force-add of SF_DURABLE_PATHS is no longer needed — planning artifacts are not gitignored, so git add -A picks them up naturally. The function reduces to:

1. git add -A
2. git reset HEAD -- <runtime paths> (unstage runtime files)

The _runtimeFilesCleanedUp one-time migration logic can also be removed.

Verified 2026-05-02: smartStage() at src/resources/extensions/sf/git-service.ts:576 is still present in its original form, including _runtimeFilesCleanedUp migration logic and SF_MILESTONE_LOCK parallel-scope logic. The simplification described here has not been performed — see Drift section.

What Happens to handleAgentEnd()

After any unit completes:

1. Invalidate caches
2. autoCommitCurrentBranch() — commits on the one and only branch
3. verifyExpectedArtifact() — file is always on the current branch (no branch switching)
4. Persist completion key

The "Path A fix" (lines 937-953) becomes the only path. No branch mismatch possible.

What Happens to fix-merge

The fix-merge dispatch unit type is eliminated. Within a worktree, there are no merges that can conflict. The only merge is milestone→main (squash), and if that conflicts (rare, parallel milestone edge case), it's handled as a one-time resolution at milestone completion — not a dispatch loop.

Verified 2026-05-02: The fix-merge prompt template has been deleted (no fix-merge.md exists in src/resources/extensions/sf/prompts/). However, MergeConflictError (re-exported from git-service.ts:201) is still present along with a JSDoc comment at git-service.ts:199 that still mentions dispatching a "fix-merge session". No active dispatch-loop code was found — this appears to be residual documentation in the class definition, not active dispatch logic. Treated as "still present (partial)" in the Drift section.

Backwards Compatibility

The shouldUseWorktreeIsolation() three-tier preference resolution is replaced by a single behavior: worktree isolation is always used. The git.isolation: "branch" preference is deprecated.

Projects with existing sf/M001/S01 slice branches can still be read by state derivation, but new work never creates slice branches.

Risks

1. Parallel milestone code conflicts at squash-merge time

If two milestones modify the same source file, the second squash-merge to main will conflict. Mitigation: git fetch origin main && git rebase main before squash-merge. This is standard practice and rare in single-user workflows.

2. Loss of per-slice git history after squash

Squash merge collapses all commits into one on main. Mitigations:

• Commit messages tag slices (feat(M001/S01/T01):) — filterable with git log --grep
• The milestone branch can be preserved (not deleted) if history is needed
• Alternative: merge --no-ff instead of --squash to keep history on main

3. SQLite DB desync after git reset

If tracked markdown rolls back via git reset --hard, the gitignored sf.db doesn't. Mitigation: the importer layer (M001/S02) rebuilds the DB from markdown on startup. The DB is a cache, markdown is truth.

4. Disk space with multiple worktrees

Each worktree duplicates the working directory (including node_modules). Mitigation: single active milestone at a time (single-user workflow), immediate cleanup after completion.

Alternatives Considered

A. Keep slice branches, fix visibility with immediate mini-merges

After research-slice or plan-slice, immediately merge the slice branch back to the milestone branch. This fixes the loop detection bug but retains all merge complexity.

Rejected: Adds another merge path instead of removing the root cause. Still requires conflict resolution, self-healing, branch switching.

B. Keep .sf/ gitignored, bootstrap from git history for manual worktrees

When SF detects an empty .sf/ in a worktree, reconstruct state from the branch's git history using git show <commit>:.sf/....

Rejected: Recovery logic, not architecture. Doesn't fix the fundamental problem of branch-agnostic state. Fails when git history has been rewritten.

C. Branch-scoped .sf/ directories (.sf/branches/<branch-name>/milestones/...)

Each branch writes to a namespaced subdirectory within .sf/.

Rejected: Adds complexity instead of removing it. Requires renaming/moving on branch creation, doesn't work with standard git tools (git checkout doesn't rename directories).

Validation

This architecture was stress-tested by three independent models:

Gemini 2.5 Pro identified 6 attack vectors. None broke the core model. Recommendations: pre-flight rebase before squash-merge (adopted), heartbeat locks (already exists), DB rebuild on startup (adopted via M001/S02 importers).

GPT-5.4 (Codex) read the full codebase and confirmed the model is sound. Identified that smartStage() already force-adds durable paths (validating the tracked-artifact approach) and that resolveMainWorktreeRoot in PR #487 is architecturally wrong (adopted — PR to be closed).

Codebase analysis confirmed .sf/milestones/ is already partially tracked on main despite the .gitignore, that SF_DURABLE_PATHS exists as a code-level acknowledgment that planning artifacts should be tracked, and that the README already documents the correct runtime-only gitignore pattern.

Codex (GPT-5.4) Dissent — "No Slice Branches Is a Redesign"

Codex read the full codebase and raised 4 concerns. Each is addressed:

Concern 1: "Crash after slice done but before integration — today the runtime detects orphaned slice branches and merges them."

Rebuttal: In the branchless model, there is no integration step to crash between. Slice work is committed directly on the milestone branch. On restart, deriveState() reads the branch state as-is. The orphaned-branch recovery path exists solely because of slice branches — removing branches removes the failure mode it recovers from.

Concern 2: "Concurrent edits to shared root docs (PROJECT.md, DECISIONS.md) from two terminals."

Rebuttal: Valid edge case. If /sf queue edits DECISIONS.md on main while auto-mode edits it in a worktree, there's a content conflict at squash-merge time. This is a standard git content conflict — no different from two developers editing the same file. Handled by normal merge resolution. Not caused by or solved by slice branches.

Concern 3: "Slice→milestone merges provide continuous integration. Removing them pushes conflict discovery to the end."

Rebuttal: In a single-user sequential workflow, there is nothing to integrate against within a worktree. Each slice builds on the previous one. The only conflict source is main diverging (e.g., another milestone merging first), which slice→milestone merges don't catch anyway — they merge within the worktree, not against main. Pre-flight rebase before squash-merge catches this more directly.

Concern 4: "Replace slice branches with another explicit slice-boundary primitive. Don't just delete them."

Response: Accepted in spirit. Commits with conventional tags (feat(M001/S01):, feat(M001/S01/T01):) serve as the slice boundary primitive. git log --grep="M001/S01" isolates a slice's history. git revert targets specific commits. Git tags (sf/M001/S01-complete) can mark slice completion if needed. The boundary primitive is commit metadata, not branches.

Action Items

1. Close PR #487 (resolveMainWorktreeRoot) — contradicts this architecture
2. Implement as a SF milestone with phases:
   • Update .gitignore and force-add existing planning artifacts
   • Remove slice branch creation/switching/merging code
   • Simplify mergeMilestoneToMain() and smartStage()
   • Remove fix-merge dispatch unit
   • Remove branch-mode isolation (git.isolation: "branch")
   • Update/delete 11 test files
   • Update README suggested gitignore
   • Migration path for existing projects with slice branches

Drift From Original Decision

Audited 2026-05-02. Items the ADR claims were deleted that are still present in the codebase:

Item	Status	File:Line	Cleanup Pending
git-self-heal.ts (whole file)	Still present	src/resources/extensions/sf/git-self-heal.ts:1–142	File is 142 lines; exports abortAndReset() and formatGitError(). The ADR claimed ~86 lines deleted. Delete entire file and migrate any callers of abortAndReset() to in-place reset logic.
smartStage()	Still present	src/resources/extensions/sf/git-service.ts:576	Still has _runtimeFilesCleanedUp migration logic, SF_MILESTONE_LOCK parallel-scope exclusions, and 49+ lines of runtime exclusion. Simplify as described in Consequences section.
shouldUseWorktreeIsolation()	Still present	src/resources/extensions/sf/auto.ts:357	ADR requires single-mode worktree-always behavior; this function still exists and defaults to false (worktree off unless explicit opt-in). Branch-mode fallback persists. Remove after deprecating git.isolation: "branch".
getSliceBranchName()	Still present	src/resources/extensions/sf/worktree.ts:261	Still used by workspace-index.ts:156 to record historical branch names. Evaluate whether this is still needed or can be removed.
MergeConflictError + fix-merge JSDoc	Partially present	src/resources/extensions/sf/git-service.ts:199–225	MergeConflictError class and a JSDoc comment referencing "dispatch a fix-merge session" remain. The fix-merge.md prompt template is deleted; no active dispatch loop found. Remove the JSDoc reference and evaluate if MergeConflictError is still needed (it is — used by slice-cadence.ts).

Items Confirmed Deleted

• mergeSliceToMilestone() — not found anywhere in the codebase.
• Original mergeSliceToMain() (branch-era, from git-service.ts) — deleted. A new mergeSliceToMain() exists at src/resources/extensions/sf/slice-cadence.ts:92 but was added post-ADR for the slice-cadence collapse feature (#4765) and is architecturally consistent with the branchless model.
• fix-merge.md prompt template — deleted (no file at src/resources/extensions/sf/prompts/fix-merge.md).
• Conflict categorization (~80 lines) — not found.
• withMergeHeal() — not found.
• ensureSliceBranch() / switchToMain() / getMergeToMainMode() — not found.

Current Authoritative Merge Path

The current milestone→main merge implementation is mergeMilestoneToMain() at src/resources/extensions/sf/auto-worktree.ts:1616. It performs squash-merge after auto-committing dirty worktree state, reconciling the worktree DB, and running a pre-flight rebase. It does not use slice branches, withMergeHeal, or conflict categorization.