singularity-forge/ARCHITECTURE.md

# Architecture

## Purpose

Singularity Forge (SF) is the product. It runs long-horizon coding work through the Unified Operation Kernel (UOK): milestones → slices → tasks. Each dispatch unit runs a fresh AI context, writes its output to disk, then terminates. UOK owns lifecycle, recovery, and the DB-backed run ledger; runtime files under `.sf/runtime/` are projections for query, UI, and compatibility. A deterministic controller (not an LLM) reads canonical state and decides what to dispatch next. Core changes follow purpose-driven TDD: purpose and consumer first, then failing tests, then implementation. The user is the end-gate — autonomous mode delivers work to human review, it does not merge to production unattended.

## Codemap

| Path | Purpose |
|------|---------|
| `src/loader.ts` | Entry point — initializes resources, registers extension |
| `src/headless.ts` | Non-interactive (headless) mode driver — exit codes 0/1/10/11/12 |
| `src/headless-events.ts` | Transcript event parsing and notification routing |
| `src/extension-registry.ts` | Registers SF as a coding-agent extension |
| `src/resources/extensions/sf/` | All SF extension source (TypeScript) |
| `src/resources/extensions/sf/auto/` | Autonomous workflow orchestrator (UOK lifecycle, dispatch, planning) |
| `src/resources/extensions/sf/bootstrap/` | Context injection, system prompt assembly |
| `src/resources/extensions/sf/prompts/` | Prompt templates (`.md`, loaded by `prompt-loader.ts`) |
| `src/resources/extensions/sf/tests/` | Unit and integration tests |
| `dist/resources/extensions/sf/` | Compiled JS (rebuilt by `npm run copy-resources`) |
| `~/.sf/agent/extensions/sf/` | Installed copy (synced from dist on startup) |
| `docs/` | Durable product, design, plan, reliability, and security context |
| `harness/` | Specs (behavior contracts), evals (model-output tests), graders |

## State layout (`.sf/`)

`.sf/` can be a **symlink** (external state, `~/.sf/projects/<hash>/`) or a **local directory** (tracking-enabled per ADR-001).

**Tracked in git** (travel with the branch, per ADR-001):
```
.sf/milestones/     — roadmaps, plans, summaries, task plans
.sf/PROJECT.md      — project overview
.sf/DECISIONS.md    — architectural decisions register
.sf/REQUIREMENTS.md — requirements register
.sf/QUEUE.md        — work queue / backlog
.sf/KNOWLEDGE.md    — project-specific rules for agents
```

**Gitignored** (runtime/ephemeral — managed by `ensureGitInfoExclude()` in `.git/info/exclude`):
```
.sf/activity/       — JSONL session dumps
.sf/audit/          — audit trail entries
.sf/exec/           — in-flight execution state
.sf/forensics/      — crash forensics
.sf/journal/        — SF journal entries
.sf/model-benchmarks/ — model benchmark results
.sf/parallel/       — parallel dispatch coordination
.sf/reports/        — generated reports
.sf/runtime/        — dispatch records, timeout tracking
.sf/worktrees/      — git worktree working directories
.sf/auto.lock       — crash detection sentinel
.sf/metrics.json    — token/cost accumulator
.sf/sf.db*          — SQLite canonical structured state, priority order, validation/gate state, and UOK ledgers
.sf/STATE.md        — derived state cache
.sf/notifications.jsonl, .sf/routing-history.json, .sf/self-feedback.jsonl, .sf/repo-meta.json
```

The symlink case uses a blanket `.sf` gitignore pattern (git cannot traverse symlinks). The directory case uses granular patterns so planning artifacts remain trackable.

## Key flows

**Autonomous dispatch loop** (`src/resources/extensions/sf/auto/`):
1. UOK reconciles the DB-backed ledger, projections, and runtime diagnostics into a typed state snapshot
2. Controller selects the next dispatch unit (research, plan, implement, verify, etc.) from canonical state
3. A fresh agent context is started with the task plan injected via `system-context.ts`
4. Agent writes artifacts to disk, commits, exits
5. UOK records completion/recovery, updates projections, and repeats until milestone completes or a gate fails

**System context assembly** (`bootstrap/system-context.ts`):
`PREFERENCES.md` → `KNOWLEDGE.md` → `ARCHITECTURE.md` → `CODEBASE.md` → code intelligence → memories → worktree/VCS blocks

**Write gate** (`bootstrap/write-gate.ts`):
All file writes in autonomous mode pass through a gate. Protected files (CLAUDE.md, CODEBASE.md, certain spec files) require explicit override.

## UOK Dispatch State Machine (Five-Phase Loop)

UOK orchestrates work through a deterministic five-phase state machine:

```
PhaseDiscuss → PhasePlan → PhaseExecute → PhaseMerge → PhaseComplete
     ↓            ↓            ↓            ↓            ↓
  (discuss)    (plan)      (execute)    (merge)     (finalize)
     ↓            ↓            ↓            ↓            ↓
  gates       gates        gates       gates      validation
     ↓            ↓            ↓            ↓            ↓
  (continue or remediate)
```

**Phase details:**

| Phase | Purpose | Exit Conditions | Failure Path |
|-------|---------|-----------------|--------------|
| **PhaseDiscuss** | Gather project context, requirements, scope | Gates pass (discussion-close gate) | Loop back for more context or escalate |
| **PhasePlan** | Create milestone/slice plans with success criteria | Gates pass (planning-approval gate) | Add remediation slices or replan |
| **PhaseExecute** | Implement tasks through the dispatch sequence | Gates pass (code-quality, test gates) | Isolate failed task, add recovery slices |
| **PhaseMerge** | Integrate slices, run end-to-end tests, merge branches | Gates pass (integration gate) | Add integration-fix slices, retry |
| **PhaseComplete** | Final validation, audit trail, summary, gate completion | Validation passes (acceptance gate) | Add remediation milestone or escalate |

**Error recovery:**

- If a gate fails, UOK records the verdict and routes through phase-specific handlers
- Failed gates can trigger automatic remediation slices (new plan → execute loop)
- Stuck-loop detection: if the same unit repeats without progress after N attempts, invoke recovery protocol (timeout, manual review, or skip)
- Crash recovery: `.sf/auto.lock` sentinel + `sf.db` WAL enables recovery from agent crash mid-phase

## Gate Verdict Semantics

Every gate runs in parallel and returns one of three verdicts:

| Verdict | Meaning | Next Action |
|---------|---------|-------------|
| **passed** | Gate question answerable; no concern blocking this phase | Proceed to next phase |
| **failed** | Gate question answerable; concern blocks phase progression | Record failure, optionally add remediation slice(s) |
| **omitted** | Gate question not applicable to this unit (e.g., no auth work → auth gate omitted) | Proceed (gate doesn't apply) |

**Critical rule:** `omitted` must have a one-line reason (e.g., "no auth surface"). Unexplained omitted verdicts are treated as failures and re-dispatched with explicit instruction to pick `passed` or `failed`.

## Outcome Learning for Model Selection

UOK tracks model success/failure per task-type using Bayesian updating:

```
P(model_i succeeds | task_type) = (successes + prior) / (total_trials + prior_weight)
```

**Mechanism:**

- After each task completes, UOK logs: `{ model, task_type, succeeded: bool, latency_ms, tokens }`
- Model scores updated dynamically; different models get different confidence per phase/task
- Prior weights prevent early abandonment (new models get benefit of the doubt)
- Used by `benchmark-selector.ts` to route future similar tasks to higher-scoring models

**Current limitation:** Learning updates episodically (per-task), not continuously; recovery paths don't feed learning back.

## Self-Evolution Mechanisms

### Self-Report Collection
Agents and gates file `sf_self_report` with anomalies during dispatch:
- Example: "validation-reviewer prompt lacks explicit rubric for criterion vs. implementation gap"
- Reports captured in `upstream-feedback.jsonl` and `.sf/SELF-FEEDBACK.md` (when dogfooding)
- **Status:** Collection works; triage pipeline incomplete (reports not automatically processed into fixes)

### Knowledge Compounding
`KNOWLEDGE.md` stores judgment-log entries from completed slices:
- Format: `[when] [what] [why] [confidence]` (e.g., "Python 3.12 incompatible with X library — avoid for now")
- Persists across milestones; can be injected into future dispatch prompts
- **Status:** Storage works; injection not automatic (requires manual configuration)

### Gate-Based Pattern Detection
Gates can detect and report repeated failure patterns (e.g., "same requirement-validation failure in S01 and S03")
- **Status:** Logic exists per gate; no automatic aggregation across gates

## Invariants

- UOK and the dispatch controller are pure TypeScript — no LLM decisions in the dispatch loop itself.
- Each dispatch unit runs in a fresh context — no cross-turn state accumulation.
- Planning artifacts are tracked in git; runtime artifacts are never committed.
- DB-backed state is the only executable truth for migrated milestones: planning hierarchy, `sequence` priority/order, validation assessments, gate runs, quality gates, UOK runtime policy, and outcome ledgers all come from SQLite. Markdown/JSON projections are human views, exports, diagnostics, or explicit recovery/import inputs; normal runtime does not fall back to them when `.sf/sf.db` exists and opens.
- `SF_RUNTIME_PATTERNS` in `gitignore.ts` is the canonical source of truth for runtime paths. `git-service.ts` (`RUNTIME_EXCLUSION_PATHS`) and `worktree-manager.ts` (`SKIP_*` arrays) must stay synchronized with it.
- The user is the end-gate. SF delivers for review, not to production.