singularity-forge/docs/dev/ADR-021-versioned-documents-and-upgrade-path.md

ADR-021: Versioned Documents and Upgrade Path

Status: Proposed Date: 2026-05-02 Deciders: Mikael Hugo

Context

SF ships a fixed set of scaffold templates via src/resources/extensions/sf/agentic-docs-scaffold.ts (the SCAFFOLD_FILES array). On project bootstrap (and on every subsequent SF auto-start), the scaffolder calls ensureAgenticDocsScaffold(basePath) which performs skip-and-create: if a target file is missing it is written, otherwise it is left alone.

This is correct for protecting user content but wrong for everything else:

1. No drift signal. When SF adds a new template (e.g. .sf/harness/AGENTS.md, ARCHITECTURE.md) or improves an existing one (e.g. tightens RELIABILITY.md), existing projects never notice. Only newly-bootstrapped projects benefit.
2. No refresh command. A user who hears "SF has a better template now" has no way to pull it.
3. Pending content is frozen. A file that is still the verbatim scaffold stub — neither the user nor any agent has touched it — is treated the same as a fully customized doc. SF refuses to refresh it because it cannot tell the two apart.
4. Only one file is versioned. PREFERENCES.md carries last_synced_with_sf in its frontmatter and gets a silent re-stamp via preferences-template-upgrade.ts on drift. Every other scaffold file is anonymous.

The user directive: everything needs versioning, with upgrade paths for documents that are not completed, perhaps with background agents.

This ADR generalises the preferences-template-upgrade.ts pattern to all scaffold-managed documents and defines a structured upgrade pipeline that can distinguish pending, editing, and completed content.

Decision

Adopt a per-document state model with explicit version markers, a project-local manifest, drift detection, and an upgrade command. Existing files are not stamped; they are migrated by content-hash match against an archive of past template versions.

1. Document states

Every scaffold-managed file is in one of three states:

State	Definition	SF action on drift
pending	Content equals (or hashes to) a known scaffold template version. Neither user nor agent has customised it.	Silent re-write to current template. Update marker.
editing	Marker present and stamped, content has drifted from the stamped template. Customisation in progress.	Do not overwrite. Write <file>.proposed with new template + diff. Optionally dispatch background merge agent.
completed	Marker absent, OR marker explicitly says state=completed.	Never modified by SF.

Detection

For Markdown files, the marker is an HTML comment on the first line:

<!-- sf-doc: version=2.75.2 template=RELIABILITY.md state=pending hash=sha256:abc... -->

Fields:

• version — SF semver that wrote/last-stamped the file.
• template — logical template id (matches an entry in SCAFFOLD_FILES).
• state — pending | editing | completed. Optional; default inferred from hash comparison.
• hash — sha256 of the file body after the marker line at stamp time. Used to confirm pending (current body still matches stamped hash) vs editing (hash mismatch ⇒ drift since stamp).

For frontmatter files (PREFERENCES.md), reuse the existing last_synced_with_sf field and add sf_template_state and sf_template_hash. The frontmatter path is the prior art described in preferences-template-upgrade.ts — extend, do not duplicate.

State derivation:

Marker present?	Body hash matches stamped hash?	Body hash matches some known template version?	State
yes	yes	n/a	pending
yes	no	n/a	editing
yes (state=completed)	n/a	n/a	completed
no	n/a	yes (legacy match via archive)	promote to pending, stamp
no	n/a	no	completed (untouched)

2. Universal versioning — what gets versioned

Apply markers to every entry in SCAFFOLD_FILES. Categories:

Category	Files (examples)	Marker mechanism
Markdown docs	AGENTS.md, ARCHITECTURE.md, docs/RELIABILITY.md, docs/SECURITY.md, docs/DESIGN.md, docs/QUALITY_SCORE.md, docs/RECORDS_KEEPER.md, all */AGENTS.md	HTML comment on line 1
Frontmatter docs	.sf/PREFERENCES.md	Frontmatter fields: last_synced_with_sf, sf_template_state, sf_template_hash (extends prior art in preferences-template-upgrade.ts)
Templates / specs	docs/design-docs/ADR-TEMPLATE.md, .sf/harness/specs/bootstrap.md, .sf/harness/AGENTS.md, .sf/harness/specs/AGENTS.md, .sf/harness/evals/AGENTS.md, .sf/harness/graders/AGENTS.md	HTML comment on line 1
Reference slot text files	docs/references/*-llms.txt	HTML comment on line 1 (Markdown comments are valid in plain text consumed by LLMs)
.siftignore and similar non-Markdown configs	.siftignore	Skip versioning. Sibling file .sf/scaffold-manifest.json records the applied version. (Rationale: hash-based legacy match is sufficient; markers in dotfiles fight tooling.)

User-content files SF must never stamp

These files are user-curated by intent (per ADR-001) and must not appear in SCAFFOLD_FILES nor be touched by the upgrade path:

• .sf/PROJECT.md, .sf/DECISIONS.md, .sf/REQUIREMENTS.md, .sf/QUEUE.md
• .sf/milestones/**/* (roadmaps, plans, summaries, slice artifacts)
• Any user-written exec plan, design doc, or product spec authored after scaffold

Verification: a unit test asserts that none of the above paths appear in SCAFFOLD_FILES and that detectScaffoldDrift ignores them.

3. Scaffold manifest (.sf/scaffold-manifest.json)

Per-project record of what scaffold versions have been applied. Lives under .sf/ (gitignored runtime; per ADR-001 not durable user content).

{
  "schemaVersion": 1,
  "applied": [
    {
      "path": "AGENTS.md",
      "template": "AGENTS.md",
      "version": "2.75.2",
      "appliedAt": "2026-04-30T12:00:00Z",
      "stateAtApply": "pending",
      "contentHash": "sha256:abc..."
    },
    {
      "path": "docs/RELIABILITY.md",
      "template": "docs/RELIABILITY.md",
      "version": "2.74.0",
      "appliedAt": "2026-03-12T08:11:02Z",
      "stateAtApply": "pending",
      "contentHash": "sha256:def..."
    }
  ]
}

Writers: agentic-docs-scaffold.ts, /sf scaffold sync, migrateLegacyScaffold. Readers: detectScaffoldDrift, doctor check. Failure mode: a corrupt manifest is rebuilt by re-walking files and reading their markers — the manifest is a cache, the marker is the source of truth.

4. Drift detection

New module exposes:

function detectScaffoldDrift(basePath: string): ScaffoldDriftReport

Returns five buckets:

Bucket	Condition	Action by /sf scaffold sync
missing	Template in SCAFFOLD_FILES but file does not exist on disk	Write the file, stamp current version
upgradable	File exists, marker present, state=pending, stamped version older than current SF	Silent re-write to current template, restamp
editing-drift	File exists, marker present, body hash ≠ stamped hash, template content has changed since stamped version	Do not overwrite. Write <file>.proposed. Optionally dispatch merge agent (#6).
untracked	File exists, no marker, predates this ADR	Run legacy hash match (#7). Promote to pending if matched, else leave alone.
customized	Marker says state=completed, OR marker absent and no legacy hash match	Skip. SF never modifies.

The report is structured (one entry per file) and can be rendered as a table for CLI output.

5. Upgrade paths

Per drift case:

• missing → write current template, stamp marker. No conflict.
• upgradable (pending) → silent re-render. Same pattern as upgradePreferencesFileIfDrifted in preferences-template-upgrade.ts: replace body, preserve any out-of-band content where the file has separable regions (frontmatter for PREFERENCES.md; for plain Markdown there is no preservation — the file is the template).
• editing-drift → produce <file>.proposed with:
  • The new template content
  • A leading <!-- sf-proposed: ... --> block describing source version, target version, and a unified diff against the stamped template version, not the current file. The agent or user can then merge intentionally.
  • The original <file> is untouched.
• untracked → see migration (#7). Either promote to pending and stamp, or leave fully alone.
• customized → never touch. Period.

6. Background-agent integration (stretch)

When editing-drift items exist and the project enables auto-merge (a new preference flag, default off), the sync command may dispatch a scaffold-keeper background agent. This agent:

1. Reads the stamped template version (from the SCAFFOLD_VERSION_ARCHIVE).
2. Reads the current template version.
3. Reads the current customised file.
4. Produces a 3-way merge: keep user customisations, apply non-conflicting upstream improvements.
5. Writes the result to <file>.proposed.
6. Posts an approval_request notification through the existing structured-notification model so the user (or a supervisor) can review and accept.

This ADR does not specify the agent prompt, scheduling policy, or grading rubric. Those belong to a follow-up ADR; the architecture here merely enables the integration by guaranteeing a clean three-input merge surface (stamped version, current version, current file).

7. Migration strategy for existing projects

Existing projects have no markers. Migration runs once on first sync after this ADR ships:

function migrateLegacyScaffold(basePath: string): MigrationReport

For each entry in SCAFFOLD_FILES:

1. If file does not exist → missing, handled by sync normally.
2. Read file, compute body sha256.
3. Look up the hash in a new constant SCAFFOLD_VERSION_ARCHIVE: Record<TemplateId, Array<{ version, hash }>> that records every template body shipped by every prior SF version.
4. If matched → file is verbatim from a known prior version. Promote to pending, stamp marker with version=<matched>, then let the normal upgradable path bring it forward.
5. If no match → user has touched it (or it predates SF tracking entirely). Leave the file alone, do not stamp. Treat as customized.

The archive starts empty for the version of SF that ships this ADR (no prior versions had hashes for these templates). As SF evolves, every release that changes a template body adds an entry mapping the previous body hash to its SF version. Within a few releases, the archive covers the long tail of real-world projects.

Migration is idempotent and non-destructive: the only filesystem change for unmatched files is none.

8. Doctor integration

A new check in doc-checker.ts (or a sibling module to keep concerns separate):

function checkScaffoldFreshness(basePath: string): DoctorFinding

Reports:

• Count by bucket: missing, upgradable, editing-drift, untracked.
• Severity: warning (non-fatal). Never blocks dispatch.
• Guidance: Run /sf scaffold sync to refresh ${n} pending docs.
• If editing-drift > 0: Run /sf scaffold sync --include-editing to merge customised docs.

Doctor finding integrates with the existing report rendering in doc-checker.ts style — same status/note shape so consumers don't fork.

9. Automatic operation — primary mode

This is the headline behaviour. Manual command operation (#10) exists only as an escape hatch and for dry-run inspection.

The upgrade pipeline runs automatically in three places:

1. On every SF startup that already calls ensureAgenticDocsScaffold (auto-start.ts, guided-flow.ts, init-wizard.ts). The sync runs synchronously in the cheap path:

   • missing items → write immediately (already current behaviour).
   • upgradable items (pending state, hash matches) → silent re-render immediately. No notification — no human attention needed.
   • untracked items → migration tries the legacy hash match in-process; if matched, promote to pending and re-render in the same pass.
   • editing-drift items → defer to background agent (point 3).

2. After every milestone completion (auto-post-unit.ts, auto.ts:stopAuto). The state of the codebase has just changed meaningfully — a good time to re-derive code-dependent docs (see "Code-as-fact verification" below).

3. Asynchronously via the existing subagent extension for editing-drift items and for code-as-fact re-derivation. The sync emits a structured notification with kind: "approval_request" only when the agent has produced a proposed change that needs review. Silent runs produce no notification.

The user never has to know any of this is happening for the common cases. The only signal is: on a fresh sf invocation, the scaffold catches up. If the agent finds something controversial, an approval-request notification surfaces with a <file>.proposed artifact ready for review.

Code-as-fact verification

Static template refresh handles the easy case. The harder, more valuable case: the document has drifted not because the template changed, but because the code has evolved past what the document claims. Examples:

• ARCHITECTURE.md says the codebase has 3 modules; the code now has 6.
• RELIABILITY.md says exit codes are 0/1/10; the code now emits 0/1/10/11/12.
• SECURITY.md lists the write-gate's protected paths; the code added 2 more.

The scaffold-keeper background agent runs the records-keeper skill (already in src/resources/extensions/sf/skills/records-keeper/SKILL.md) which already specifies "prefer source and tests for implemented behavior". That skill is the agent's contract.

Code is the fact. Documents are projections of code at a moment in time. When the projection drifts, the agent re-derives by reading source. When the agent can re-derive non-controversially, it does so silently. When the change is large enough to warrant review, it surfaces a .proposed artifact through the structured notification model.

This generalises records-keeper from a skill an agent can run to a skill the system runs automatically on a regular cadence. The records-keeper attempt was the right idea; this ADR is what makes it autonomous.

Cadence

• Synchronous on startup: only the cheap path (hash comparisons, missing files, pending refreshes). Bounded by file count, runs in milliseconds.
• Asynchronous after milestone completion: dispatches a scaffold-keeper subagent. The subagent runs the records-keeper procedure on the docs whose source dependencies have changed since last sync.
• Optional cron via the existing /loop system: a daily background run for projects that don't close milestones often.

Failure modes are non-fatal

The pipeline is designed to fail open: any error reading a marker, computing a hash, dispatching an agent, or writing a .proposed file results in the file being left alone, a debug log line, and SF continuing normally. The user-facing contract is "SF doesn't break my docs." Drift detection may under-fire; it must never over-fire.

10. Manual command — /sf scaffold sync (escape hatch)

For dry-run inspection, forced refresh, and scoped operations:

/sf scaffold sync [--dry-run] [--include-editing] [--only=<glob>]

Flag	Behaviour
(default)	Force the same operation that would run automatically. Useful when the user wants to refresh on demand without waiting for the next startup or milestone close.
--dry-run	Print the drift report and the planned actions. Make no filesystem changes. The primary diagnostic mode.
--include-editing	Synchronously merge editing-drift items inline (vs. the default async-via-subagent). Used when the user wants a definitive answer right now.
--only=<glob>	Restrict the sync to a path glob. Useful for tightly scoped refreshes (--only=harness/**) or for re-deriving a specific code-dependent doc (--only=docs/RELIABILITY.md).

Exit code: 0 if no errors. Non-zero only if filesystem writes failed for reasons unrelated to drift (permission, disk full).

Implementation phases

Phase	Scope
A	Stamp markers on all SCAFFOLD_FILES writes. Maintain .sf/scaffold-manifest.json. Extend PREFERENCES.md frontmatter with sf_template_state and sf_template_hash. Existing agentic-docs-scaffold.ts callsites unchanged externally.
B	Implement detectScaffoldDrift, migrateLegacyScaffold, and the initial SCAFFOLD_VERSION_ARCHIVE (empty).
C	Automatic synchronous sync: extend ensureAgenticDocsScaffold to apply missing + upgradable + legacy-migrated items in the same pass. No new command surface required for this; the existing callsites get the upgrade behaviour for free. checkScaffoldFreshness doctor finding for visibility.
D	Automatic asynchronous sync via existing infrastructure: after milestone completion (auto-post-unit.ts, auto.ts:stopAuto), dispatch a scaffold-keeper subagent (via the existing subagent extension) that runs the records-keeper skill against drifted docs. Code-as-fact verification: agent reads source and re-derives content. Surfaces results as kind: "approval_request" notifications using the structured-notification model from ADR-019/020.
E (escape hatch)	/sf scaffold sync command for dry-run inspection, forced refresh, and scoped operations (--only=<glob>, --include-editing).

Each phase is independently shippable and testable. Phase A alone unlocks the architectural property: SF can tell pending from completed on every project from now on. Phase C is what the user experiences as "automatic" for the simple cases; Phase D is what makes records-keeper autonomous for the code-derived cases.

Consequences

Becomes possible

• Continuous template evolution: SF can iterate scaffold content freely knowing pending docs auto-upgrade.
• Visible signal for project staleness via doctor.
• Clean separation of SF-managed vs user-owned content per file, not per directory.
• Foundation for background-agent merges of customised docs.
• Future templates (new harness specs, new ADR types) propagate without manual project-by-project edits.

Becomes harder

• Scaffold manager grows from "skip-if-exists" to a state machine. Test matrix grows accordingly.
• Every template change must consider whether to bump the SF version archive entry for the previous body. Forgotten archive entries cause legacy files to be classified customized when they should be pending.
• Marker format becomes load-bearing: changing it is itself a versioning problem (handled by versioning the marker schema in schemaVersion).

Failure modes

Failure	Behaviour
Corrupt scaffold-manifest.json	Rebuilt by re-walking files; markers are source of truth.
Marker hash mismatch with stamped content (e.g. user hand-edited the marker)	File classified editing. SF will not overwrite. User can fix by running sync with --include-editing after reviewing the proposed file.
SCAFFOLD_VERSION_ARCHIVE missing an entry for a real prior version	Affected files classified customized and left alone. Recoverable by adding the archive entry in a later SF release; sync will then promote on next run.
Read-only filesystem	Each writer is wrapped in try/catch (same pattern as upgradePreferencesFileIfDrifted). Sync degrades to dry-run output.
Concurrent sync runs	First writer wins; second sees no drift on second pass. Manifest writes are atomic via temp+rename.

Alternatives Considered

Alternative	Why rejected
Status quo (skip-and-create only)	No visibility, no refresh path, anonymous templates. The whole point of this ADR.
Aggressive refresh on every run (overwrite all scaffold files)	Destroys user customisations. Non-starter.
Git-based detection (compare repo HEAD against SF init commit)	Requires clean git state at sync time, breaks under merges/rebases, conflates user commits with SF state, fragile across worktrees (per ADR-001).
Per-file .<file>.sf-meta sidecar instead of inline marker	Doubles file count, easy to delete, harder to cite in human review. Marker travels with the file it describes.
Single global SF_VERSION stamp on the manifest only	Cannot distinguish pending vs editing per file; a single global re-stamp would either skip everything (current behaviour) or overwrite everything (rejected). Per-file state is the minimum useful granularity.
LLM-based "is this still pending?" classifier	Non-deterministic, expensive, unnecessary. Hash equality is the right primitive.

Migration

See #7. One-shot, idempotent, non-destructive. No project action required; first sync after upgrade classifies everything correctly.

Validation

# Once Phase A lands:
node --experimental-strip-types --test \
  src/resources/extensions/sf/tests/scaffold-versioning.test.ts

# Once Phase C lands:
sf scaffold sync --dry-run
# Expected output: structured drift report with bucket counts.

References

• ADR-001 — Branchless Worktree Architecture (.sf/ durable vs runtime boundary; informs which files are SF-managed vs user-curated).
• ADR-018 — Repo-native Harness Evolution (template kits and harness contracts that this versioning system will manage).
• ADR-019 — Workspace VM Convergence (informs where scaffold sync runs in the execution layer).
• src/resources/extensions/sf/preferences-template-upgrade.ts — prior art for frontmatter drift detection and silent re-stamping; the pattern this ADR generalises to all scaffold-managed files.
• src/resources/extensions/sf/agentic-docs-scaffold.ts — current scaffold list and skip-if-exists logic, to be extended in Phase A.
• src/resources/extensions/sf/doc-checker.ts — existing scaffold-content checker, sibling to the new freshness check in Phase C.