FairGarden Docs

Architecture

This page explains the overall architecture of @fairgarden/docs, including data flows, key design decisions, and how the various pieces fit together.

Overview

The package is built around one core principle: move locally-derived computation to build time. The cache only updates whenever those files change.

┌─────────────────────────────────────────────────────────────────┐
│                           BUILD TIME                            │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │  TypeScript  │    │    Source    │    │   Markdown   │       │
│  │   Sources    │    │    Files     │    │    Files     │       │
│  └──────┬───────┘    └──────┬───────┘    └──────┬───────┘       │
│         │                   │                   │               │
│         ▼                   ▼                   ▼               │
│  ┌──────────────┐    ┌──────────────┐    ┌──────────────┐       │
│  │ typescript-  │    │ Starry Night │    │    Remark    │       │
│  │api-extractor │    │ Highlighter  │    │   Plugins    │       │
│  └──────┬───────┘    └──────┬───────┘    └──────┬───────┘       │
│         │                   │                   │               │
│         └───────────────────┼───────────────────┘               │
│                             ▼                                   │
│                    ┌──────────────┐                             │
│                    │     HAST     │  (JSON-serializable AST)    │
│                    └──────┬───────┘                             │
│                           │                                     │
│                           ▼                                     │
│                    ┌──────────────┐                             │
│                    │  precompute  │  (injected into source)     │
│                    └──────┬───────┘                             │
└───────────────────────────┼─────────────────────────────────────┘
                            │
┌───────────────────────────┼─────────────────────────────────────┐
│                           ▼              RUNTIME                │
│                    ┌──────────────┐                             │
│                    │  hastToJsx   │  (HAST → React.ReactNode)   │
│                    └──────┬───────┘                             │
│                           │                                     │
│                           ▼                                     │
│                    ┌──────────────┐                             │
│                    │   Rendered   │                             │
│                    │  Components  │                             │
│                    └──────────────┘                             │
└─────────────────────────────────────────────────────────────────┘

Why this architecture?

  • Syntax highlighting libraries are large (~500KB+) - keep them out of client bundles
  • TypeScript parsing is slow - do it once, cache the results
  • HAST is JSON - webpack can cache it, and it serializes across server/client boundaries

Data Flows

Code Demos Pipeline

When you write a demo with createDemo():

demos/basic/index.ts (createDemo call)
         │
         ▼ [loadPrecomputedCodeHighlighter loader]
         │
    ┌────┴────┐
    │ Parse   │  Find createDemo() call, extract variants
    │ Factory │
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │  Load   │  Read each variant file + dependencies
    │  Files  │  Track all files for webpack watching
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │ Starry  │  Syntax highlight with grammar scopes
    │  Night  │  Add line gutters
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │Transform│  TypeScript → JavaScript (optional)
    │  TS→JS  │  via Sucrase
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │ Inject  │  Insert { precompute: { ... } } into source
    │  HAST   │
    └────┬────┘
         │
         ▼
    createDemo(import.meta.url, Component, { precompute: { ... } })

Key files:

Type Documentation Pipeline

Type documentation has two paths that converge on the same highlighting steps:

Build-time path (precomputed)

When you write types with createTypes():

types.ts (createTypes call)
         │
         ▼ [loadPrecomputedTypes loader]
         │
    ┌────┴────┐
    │ Parse   │  Find createTypes() call
    │ Factory │
    └────┬────┘
         │
         ▼ [loadServerTypes with sync: true]
         │
         ▼ [Worker Thread - preserves TS language service cache]
         │
    ┌──────────┐
    │typescript│  Parse TypeScript AST
    │   -api-  │  Extract exports, components, props
    │extractor │
    └────┬─────┘
         │
         ▼
    ┌─────────┐
    │ Format  │  Format props, find DataAttributes/CssVars enums
    │  Types  │  Parse JSDoc descriptions as markdown
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │  Write  │  Generate and write types.md
    │ types.md│
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │Highlight│  Syntax highlight type strings → HAST
    │  Types  │
    └────┬────┘
         │
         ▼
    createTypes(import.meta.url, Component, { precompute: { ... } })

Runtime path (from existing types.md)

When types are already generated, loadServerTypes can read an existing types.md instead of re-extracting from TypeScript source. This is useful in RSC pages, custom build tools, or when types are committed to the repository:

types.md (already on disk)
         │
         ▼ [loadServerTypes with sync: false (default)]
         │
    ┌──────────┐
    │  Parse   │  Read types.md
    │ Markdown │  Parse back into structured TypesSourceData
    └────┬─────┘
         │
         ▼
    ┌─────────┐
    │Highlight│  Same highlighting as the build-time path
    │  Types  │  Type strings → HAST
    └────┬────┘
         │
         ▼
    HighlightedTypesMeta with HAST fields

Both paths produce identical HighlightedTypesMeta with syntax-highlighted HAST — the only difference is where the type data comes from.

Key files:

Search Pipeline

 
                    [Out of band - transformMarkdownMetadata]
                                     │
    page.mdx files ──────────────────┼──▶ index page.mdx files
    (title, description, sections)   │    (aggregated metadata)
                                     │
─────────────────────────────────────┼─────────────────────────
                                     │
app/sitemap/index.ts (createSitemap call)
         │
         ▼ [loadPrecomputedSitemap loader]
         │
    ┌────┴────┐
    │ Parse   │  Find createSitemap() call
    │ Factory │  Extract page imports
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │  Load   │  Import each index page.mdx
    │ Indexes │  Read pre-extracted metadata
    └────┬────┘
         │
         ▼
    ┌─────────┐
    │ Build   │  Create Orama-compatible schema
    │  Index  │  Index all searchable content
    └────┬────┘
         │
         ▼
    { schema: {...}, data: {...} }  // Injected as precompute
         │
         ▼ [Runtime - Client]
         │
    ┌──────────┐
    │ useSearch│  Create Orama instance from precomputed data
    │   hook   │  Provide search API to components
    └──────────┘

Key files:

Coordinated Loading & Streaming

The pipelines above decide what crosses the server→client boundary and how small it is. A separate runtime architecture decides when deferred content swaps in — and coordinates that swap across every block on the page so it settles as one update instead of a staggered cascade. It is layered, and each layer is usable on its own:

useStream · abstractCreateStream   load a list of pieces   (client stream · server/build)
      ▲
CoordinatedLazy     one deferred piece: fallback ⇄ content swap + coordinated settle
                    (CoordinatedLazy, createCoordinatedLazy, useChunk, LazyContent)
      ▲
useCoordinated      the settle gate + page-wide coordination

Each layer composes on the one below and is usable on its own.

The settle gate

createSettleGate() is the shared "all sources settled" primitive. A source register()s on mount and calls the returned settle() once it reaches its stable layout; the gate opens when every source has settled. It opens once — a source mounting after the page settled adopts the open state rather than re-closing it. Two opt-in completion signals extend it for content that arrives over time: expect(n) (known count) and markLast() (last chunk).

One module-global instance, pageSettleGate, is what the page-wide layoutShiftGate and every CoordinatedLazy swap register with — so useCoordinated can hold a page-wide transform/variant change until the initial swaps have all landed.

The swap — CoordinatedLazy

CoordinatedLazy generalizes the fallback↔content swap that was previously welded into CodeHighlighterClient:

Server: pre-render BOTH                Client: state-driven swap
  fallback = <ContentLoading/>           1. force-mount fallback once
  content  = <Content/>          ──▶     2. fallback hoists data up
                                         3. ready → swap to content
                                            (hoisted data flows down)
  • Force-mount-once — even when the content is precomputed, the loading component mounts for one commit so its hoist hook runs (the same trick CodeHighlighter uses to recover its decompression dictionary).
  • Hoist up / content-context down — the fallback hoists data via useCoordinatedFallback; the swap collects it and hands it to the content via useCoordinatedContent, so the full content reuses what the fallback already fetched (a DEFLATE dictionary, or a low-resolution chart line).
  • Nested suppression — an inner instance nested inside an outer's still-showing fallback stays in its own fallback, collapsing a "fallback → content → fallback → content" flicker into one transition.
  • Settle registration — each swap registers with the page gate (and, inside a controller, that controller's gate too) so the page can coordinate.

A codeblock frame is the simplest instance: its plain-text data.fallback → highlighted HAST is exactly a loading→full swap, and a multi-frame file is serial chunking of one source.

Self-loading pieces & streaming

createCoordinatedLazy() builds a self-loading component that loads one piece's data and swaps via CoordinatedLazy. Its source is a discriminated union (no overloads, no runtime return-sniffing):

source.modeLoads
dataone value directly (optionally a quick initial first)
urlsa list of chunk URLs, then each URL's data
streampushes chunks into an array and yields — progressive reveal

useChunk() loads a single chunk. Driving a list is the layer above, split by who loads: useStream() streams the list in on the client (accumulating into state) and useStreamController() scopes a group so the page can tell when they have all loaded — via known count or a last-chunk flag — while abstractCreateStream() is the server/build counterpart that injects precomputed data. A list controller hands its gate down as the ambient gate, so each chunk's CoordinatedLazy registers with it without a threaded gate prop. On the server, each chunk is its own async component under a per-chunk Suspense boundary, so React flushes them as they resolve.

Speculative preload

When the fallback hoists its data, CoordinatedLazy fires a preload(hoisted) callback. A consumer uses it to start dynamic import()s of the heavy helpers the data implies — a transform function, say — in parallel with loading the full content, instead of the content mounting and then requesting them in a serial roundtrip. PreloadProvider / usePreload dedup those imports across every instance on the page, so a shared helper is fetched once.

Why this architecture?

  • The swap, hoist, and coordination logic is identical whether content is precomputed, server-loaded, or client-loaded — so it lives in one tested primitive instead of being copied per feature.
  • Coordinating swaps avoids layout-shift cascades: blocks settle together, and a later page-wide change can wait for the initial settle.
  • Streaming plus per-chunk Suspense lets large content (a detailed chart, a long file) reveal progressively without blocking first paint.

Key files: useCoordinated/ (settle gate + page coordination), CoordinatedLazy/ (the swap plus createCoordinatedLazy / useChunk / LazyContent / the *Server render functions — one deferred piece), useStream/ (useStream / useStreamController / streamChunks — client list loading), ChunkProvider/ (lazy client-loader context + usePreload dedup), abstractCreateStream/ (the server/build createStream factory).

The Factory Pattern

All precomputable features use the same pattern:

// 1. Define a factory in your app
export const createDemo = createDemoFactory({ DemoContent });

// 2. Use the factory in index.ts files
export const DemoCheckbox = createDemo(import.meta.url, Checkbox);

// 3. Webpack loader detects the factory call
// 4. Loader processes and injects precompute
// 5. Factory receives precomputed data at runtime

Why import.meta.url?

  • Provides the file's absolute URL at both build and runtime
  • Loaders can derive names, paths, and relationships from the URL
  • Enables filesystem-based conventions (like demos/basic/index.ts)

See Built Factories for details.

Worker Thread Architecture

Type processing uses a singleton worker thread to preserve the TypeScript language service cache:

┌─────────────────────────────────────────────────────────────┐
│                    Main Thread (Webpack)                    │
│                                                             │
│  Loader 1 ──┐                                               │
│  Loader 2 ──┼──▶ WorkerManager ──▶ Worker Thread            │
│  Loader 3 ──┘         │                   │                 │
│                       │                   │                 │
│                       │          ┌────────────────┐         │
│                       │          │ TS Language    │         │
│                       │          │ Service Cache  │         │
│                       │          └────────────────┘         │
│                       │                                     │
│              Request/Response via postMessage               │
└─────────────────────────────────────────────────────────────┘

Why a worker?

  • TypeScript's language service maintains a cache of parsed files
  • Creating a new program for each loader call is expensive
  • A persistent worker preserves the cache across all loader invocations
  • Uses Symbol.for() on process to ensure singleton across Turbopack contexts

See workerManager.ts for implementation.

Module Organization
ModuleEnvironmentPurpose
CodeHighlighter/IsomorphicMain code display component
CodeProvider/ClientClient-side code loading context
CoordinatedLazy/IsomorphicSwap + self-loading piece + Suspense
useStream/ClientClient list loading + completion
ChunkProvider/ClientLazy client loaders + preload dedup
abstractCreate*/ServerFactory utilities ('server-only')
use*/ClientReact hooks (state/effects)
pipeline/loadPrecomputed*BuildWebpack loaders
pipeline/loadIsomorphicCodeVariantIsomorphicRuntime code loading/transforms
pipeline/loadServer*ServerRSC data loading
pipeline/syncTypesBuildTypeScript parsing, worker threads
withFairGardenDocs/BuildNext.js configuration plugin

Key Design Decisions

Why HAST?
  1. JSON-serializable - Can be cached by webpack, stored in bundles, passed across server/client boundary
  2. Framework-agnostic - Convert to React, Vue, or any framework
  3. Transformable - Rich ecosystem of rehype plugins for post-processing
  4. Safer - Avoids dangerouslySetInnerHTML and its XSS risks

Why webpack loaders over runtime processing?
  1. Caching - Webpack caches loader results, rebuilds only changed files
  2. Dependency tracking - Loaders can declare file dependencies for watch mode
  3. Zero runtime cost - Client never parses or highlights code
  4. Works with RSC - Precomputed data is just JSON, serializes cleanly

Why the factory pattern?
  1. Single source of truth - Factory controls all behavior, index files are copy-paste
  2. Future-proof - Change factory implementation without touching hundreds of files
  3. Type-safe - TypeScript infers types from factory configuration
  4. Loader-friendly - Predictable pattern for loaders to detect and process

Performance Characteristics
OperationWhenTypical Time
TypeScript parsing (cold)Build500-2000ms
TypeScript parsing (warm)Build50-200ms
Syntax highlightingBuild10-50ms per file
HAST → JSX conversionRuntimeunder 10ms
Search index creationRuntime (once)50-100ms
Search queryRuntimeunder 10ms

The worker thread architecture makes TypeScript parsing ~10x faster after the first file by preserving the language service cache.