Documentation

Appearance

Magic the Gathering ingest ​

How Magic was wired into the scan pipeline on 2026-05-07.

TL;DR ​

  • Data source: Scryfall default_cards bulk (every printing, ~85 k physical cards after filtering MTGO/Arena digital-only rows).
  • No migration needed — the multi-TCG schema (cards.tcg, cards.base_id, cards.variant) added for One Piece accommodates MTG unchanged.
  • MTG does not go through tcg-vision (Tormknd's FAISS index doesn't cover Magic) — MTG scans resolve via /scan-lab (our pgvector + DINOv2 pipeline).
  • Observed throughput: ~213 cards/sec on ingest (Scryfall CDN at concurrency 12), ~9 cards/sec on embedding (DINOv2 small CPU).

How to run the ingest ​

bash
# Full catalog (Scryfall default-cards, ~85 k cards, ~13 GB images)
docker compose --env-file .env -f docker/compose.yml run --rm \
  -e INGEST_TCG=mtg ingest

# Single set (smoke test — Limited Edition Alpha = 295 cards / ~80 MB)
docker compose --env-file .env -f docker/compose.yml run --rm \
  -e INGEST_TCG=mtg -e INGEST_SETS=lea ingest

# Smaller alternative — `unique_artwork` (one row per illustration, ~57 k)
docker compose --env-file .env -f docker/compose.yml run --rm \
  -e INGEST_TCG=mtg -e INGEST_MTG_BULK=unique_artwork ingest

Then populate the embeddings:

bash
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/embedCatalogLab.js" \
  -e INGEST_LAB_TCG=mtg ingest

MTG variants (vs One Piece) ​

Unlike OPTCG (which returns multiple entries sharing the same card_set_id for parallel arts), Scryfall already de-duplicates printings via distinct collector_number values within a set. So MTG doesn't need assignVariantIds — every Scryfall row is already a distinct printing with its own PK id.

The only case where one physical card spawns multiple rows: double-faced cards (transform / mdfc). The front gets the base id; the back gets ${baseId}_back with variant='back'. Both faces are indexed independently in card_embeddings_lab because they're independently scannable.

Why not tcg-vision? ​

tcg-vision (Tormknd's sidecar) ships a pre-computed FAISS index covering Pokémon (~26 k) + One Piece (~4 k) + a sliver of Lorcana. Magic isn't in it. Two options:

  1. Ask Tormknd to add Magic to his tgc-app and rebuild his FAISS — no code on our side but we depend on his cadence.
  2. Use our pgvector pipeline — ingest from Scryfall, embed with our embeddings: sidecar, query via /scan-lab. Zero external dependency but it's the first real prod use of the pgvector path.

We took option 2 because (a) we wanted to validate our pipeline anyway to eventually retire the Tormknd dependency, and (b) Magic is the ideal opportunity — a full TCG that doesn't exist anywhere else on our backend.

See also ​