singularity-forge/rust-engine/README.md

SF Native Engine

Rust N-API addon providing high-performance native modules for SF.

Architecture

JS (packages/native) -> N-API -> Rust crates

rust-engine/crates/
├── engine/  (N-API bindings, cdylib — 20+ modules)
├── grep/    (ripgrep internals, pure Rust lib)
└── ast/     (ast-grep structural search)

Inspired by Oh My Pi's pi-natives, adapted for SF's Node.js runtime.

Prerequisites

• Rust (stable, 1.70+): https://rustup.rs
• Node.js (24.0.0+)

Build

# Release build (optimized)
npm run build:native

# Debug build (fast compile, no optimizations)
npm run build:native:dev

The build script compiles the Rust code and copies the .node shared library to rust-engine/addon/.

Test

# Rust unit tests
cd native && cargo test

# Node.js integration tests
npm run test:native

Modules

ast

Structural code search via ast-grep. Provides pattern-based code matching that understands language syntax, enabling searches like "find all functions that return a Promise" rather than raw regex.

clipboard

Native clipboard access for reading and writing system clipboard contents.

diff

Fuzzy text matching and unified diff generation. Provides efficient comparison of text content with configurable matching thresholds.

fd

Fuzzy file path discovery. Locates files by partial name matching across the project tree.

fs_cache

Filesystem caching layer. Caches file metadata and contents to reduce redundant I/O during repeated operations.

git

Libgit2-backed git read operations. Provides fast, direct access to repository status, diffs, blame, and log without shelling out to the git CLI.

glob / glob_util

Gitignore-aware file discovery. Walks directory trees while respecting .gitignore rules, returning matching paths for a given glob pattern.

grep

Ripgrep-backed regex search using the grep-regex, grep-searcher, and grep-matcher crates.

Functions:

• search(content, options) — Search in-memory Buffer/Uint8Array content
• grep(options) — Search files on disk with glob filtering and .gitignore support

TypeScript usage:

import { grep, searchContent } from "@sf/native";

// Search files
const result = grep({
  pattern: "TODO",
  path: "./src",
  glob: "*.ts",
  ignoreCase: true,
  maxCount: 100,
});

// Search content
const contentResult = searchContent(Buffer.from(fileContent), {
  pattern: "function\\s+\\w+",
  contextBefore: 2,
  contextAfter: 2,
});

sf_parser

SF file parsing and frontmatter extraction. Reads .sf files and extracts structured metadata from YAML frontmatter blocks.

highlight

Syntect-based syntax highlighting. Tokenizes source code and produces highlighted output for terminal or HTML rendering.

html

HTML-to-Markdown conversion. Transforms HTML content into clean Markdown, useful for importing web content into SF notes and documents.

image

Image decoding, encoding, and resizing. Supports common formats (PNG, JPEG, WebP) and provides efficient thumbnail generation.

json_parse

JSON parsing utilities. Provides streaming and fault-tolerant JSON parsing for large or partially valid payloads.

ps

Cross-platform process tree management. Lists, inspects, and terminates process trees by PID, used for managing spawned subprocesses.

stream_process

Streaming process I/O. Spawns child processes with non-blocking, streamed access to stdout and stderr for real-time output handling.

task

Task-related native operations. Provides low-level primitives for task scheduling and execution within the native layer.

text

ANSI-aware text measurement and wrapping. Correctly measures visible width of strings containing ANSI escape codes and wraps text to terminal column widths.

truncate

Text truncation utilities. Truncates strings to a target length while preserving ANSI sequences and respecting grapheme boundaries.

ttsr

Tool-triggered system rules. Evaluates and applies system-level rules that activate in response to specific tool invocations.

xxhash

xxHash hashing. Provides fast, non-cryptographic hashing via the xxHash algorithm for content deduplication and cache keying.

Adding New Modules

1. Create a new crate in rust-engine/crates/ (pure Rust library)
2. Add N-API bindings in rust-engine/crates/engine/src/
3. Add TypeScript wrapper in package./rust-engine/src/
4. Add the crate to engine/Cargo.toml dependencies