> ## Documentation Index
> Fetch the complete documentation index at: https://docs.buildwithtrace.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Pipeline

> The offline Python pipeline that builds the Symbols and Footprints catalogs — idempotent sync, KiCad parsers, CERN import, index shapes, and CI.

The browseable catalog is **pre-built offline** by Python scripts (locally or in CI) and
committed as JSON. Nothing in this pipeline runs at request time — the web app only reads
the finished index.

<Info>
  Run these scripts from the repo root with Python 3.12+. They clone upstream libraries into
  temp dirs, parse them, and write the index files under `data/`.
</Info>

## Sources

| Platform   | Source | Repo                                                 | License               |
| ---------- | ------ | ---------------------------------------------------- | --------------------- |
| Symbols    | Trace  | `buildwithtrace/trace-kicad-symbols-lib` (GitHub)    | KiCad Library License |
| Symbols    | CERN   | `gitlab.com/ohwr/cern-kicad-libs` (symbol side)      | CERN-OHL-P-2.0        |
| Footprints | Trace  | `buildwithtrace/trace-kicad-footprints-lib` (GitHub) | KiCad Library License |
| Footprints | CERN   | `gitlab.com/ohwr/cern-kicad-libs` (footprint side)   | CERN-OHL-P-2.0        |

Each indexed record carries a `source` field (`trace` | `cern`); on id collisions, Trace
wins de-dupe.

## `scripts/sync.py` — idempotent controller

Tracks the last-synced commit hash per source in `data/.sync_state.json` and only
re-clones/re-indexes a source whose upstream `HEAD` changed.

<CodeGroup>
  ```bash All sources theme={null}
  python3 scripts/sync.py            # check all, sync what changed
  python3 scripts/sync.py --force    # re-sync everything
  python3 scripts/sync.py --status   # print state, no sync
  ```

  ```bash One source (symbols) theme={null}
  python3 scripts/sync.py --source trace   # Trace symbols
  python3 scripts/sync.py --source cern    # CERN symbols
  ```

  ```bash One source (footprints) theme={null}
  python3 scripts/sync.py --source trace   # Trace footprints
  python3 scripts/sync.py --source cern    # CERN footprints
  ```
</CodeGroup>

On the **footprints** repo, sync persists each source's raw `.pretty` libraries into
`data/trace-footprints/` and `data/cern-footprints/` (both gitignored) and rebuilds the
**combined** `footprints_index.json` from both on every run, so neither source overwrites
the other.

## Indexers

<CardGroup cols={2}>
  <Card title="build_index.py (symbols)" icon="diagram-project">
    Parses every `.kicad_sym` via balanced-paren extraction, resolves `extends`
    inheritance, classifies categories, and writes `data/symbols_index.json` (compact) plus
    per-library `data/libraries/{Name}.json` (full pin data).
  </Card>

  <Card title="build_footprints.py (footprints)" icon="microchip">
    Accepts multiple `--data-dir` args (infers `source` from the dir name), recursively
    scans `.pretty` dirs, parses `.kicad_mod` (+ legacy `.module`) for pad count/types/tags/
    3D model, de-dupes by id, and writes `data/footprints_index.json`.
  </Card>
</CardGroup>

`scripts/import_cern.py` shallow-clones the CERN repo and copies the relevant libraries
(`.kicad_sym` → `data/cern-symbols/` on the symbols repo; `.pretty` dirs →
`data/cern-footprints/` on the footprints repo), writing a `manifest.json`.

## Index shapes

<CodeGroup>
  ```jsonc symbols_index.json theme={null}
  {
    "version": 1, "total_symbols": 30943, "total_libraries": 251,
    "libraries": [ { "id": "MCU_ST_STM32F4", "name": "MCU_ST_STM32F4", "symbolCount": 312 } ],
    "symbols": [ {
      "id": "Library:Name", "name": "...", "library": "...", "category": "mcu",
      "description": "...", "keywords": ["..."], "pinCount": 64, "source": "trace"
    } ]
  }
  ```

  ```jsonc footprints_index.json theme={null}
  {
    "version": 1, "total_footprints": 24690, "total_libraries": 214,
    "libraries": [ { "id": "battery", "name": "Battery", "footprintCount": 123 } ],
    "footprints": [ {
      "id": "...", "name": "...", "library": "...", "description": "...", "tags": "...",
      "padCount": 8, "pads": [ { "number": "1", "type": "smd" } ],
      "model3d": null, "source": "trace"
    } ]
  }
  ```
</CodeGroup>

<Note>
  The compact `symbols_index.json` omits pin arrays; full pin data lives in per-library files
  loaded on demand. Footprints carry their pad array inline in the index.
</Note>

## CI

`.github/workflows/sync-libraries.yml` runs the sync on a daily cron (and manual dispatch),
and if the index changed, opens a PR (`peter-evans/create-pull-request`). The symbols
workflow is labeled `symbols`; the footprints workflow is labeled `footprints`.