Documentation

Appearance

Catalog updates — new sets, new cards ​

Operational runbook for keeping the backend in sync with upstream when new cards or new expansions ship. Read this when a new Pokémon set drops, when Bandai releases an OP starter deck, when Wizards prints a Standard set — anything that changes the universe of scannable cards.

TL;DR ​

  • All ingest + embed steps are idempotent. Re-running the same command after a partial run picks up exactly where it left off; a full re-run after a new set adds the delta and skips everything already on disk + in DB.
  • A new set takes ~1-5 min of operator time depending on the TCG. The slowest part is embedding new cards (~10 cards/s on the DINOv2 sidecar).
  • One asymmetry: Pokémon and One Piece scans rely on Tormknd's FAISS index (frozen at our container build). New cards land in our DB but won't resolve through /scan-compat until Tormknd rebuilds. Resolved 2026-05-07 — Phase F flipped the RN app to /scan-lab for all three TCGs, so the asymmetry is gone. New cards from any TCG are scannable immediately after embed.

End-user impact when a new set drops ​

The mobile-app user interacts with two things: the catalog (browsing cards, viewing metadata + image) and the scanner (take a photo, get an identification). Those two functions touch different storage axes that don't refresh at the same rate.

What the user doesData layerAvailable once…Operator step
Browse the catalog by set / raritycards + sets rows in PostgresIngest finishes (~30 s - 1 min)INGEST_TCG=… ingest
See a card's image in its detail screendata/images/<set>/<num>/large.jpg filesIngest finishes (download included)INGEST_TCG=… ingest (no extra flag)
Search by nameTrigram index on cards.nameIngest finishes (the trigram index auto-maintains)INGEST_TCG=… ingest
Scan a card from the new set (MTG)card_embeddings_lab vectors (pgvector HNSW)Embed finishes (~30 s - 5 min depending on set size)INGEST_LAB_TCG=mtg embedCatalogLab
Scan a card from the new set (Pokémon / OP)Tormknd's FAISS index (frozen at build)Tormknd rebuilds his repo OR we flip to /scan-lab (Phase F)Nothing on our side suffices today

In short, for a new set:

  • MTG → Zero friction. Ingest + embed (~1-2 min total) = scannable and browsable.
  • Pokémon / OP → Catalog + images + name search: instant after ingest. Scan via the app: depends on Tormknd, or wait for Phase F.

This asymmetry is the main friction we're trying to remove with Phase F. Until that's done, scanning a freshly-released Pokémon booster card will return "no-match" on the mobile app, even though we have the data in DB.

Lifecycle: from upstream release to scannable ​

   Day 0                Day 0 - 7              Day 0 (us)            Day 0+ (us)
[Set released] → [Upstream publishes] → [Operator runs ingest] → [Operator runs embed] → [Scannable]
   (Wizards / Bandai /        (Scryfall /             (~10 s - 1 min            (~30 s - 5 min)
    The Pokémon                TCGdex /                per TCG)
    Company)                   OPTCG API)

The dashed gap between "set released" and "upstream publishes" varies a lot:

  • Scryfall: same-day or next-day refresh of default_cards bulk. Best in class.
  • TCGdex: typically 2-7 days after the official Pokémon release. Community-driven.
  • OPTCG API: usually same-week as Bandai's release.

Per-TCG cadence and commands ​

TCGSourceWhen upstream sees the new setOur commandTime
MagicScryfall default_cards bulkDay 0 (often within hours of the set going live)INGEST_TCG=mtg ingest then INGEST_LAB_TCG=mtg embedCatalogLab~1 min ingest + ~30 s embed per ~300-card set
PokémonTCGdex public API2-7 days post-releaseINGEST_TCG=pokemon INGEST_SETS=<id> ingest then embed~30 s ingest + ~30 s embed per ~200 cards
One PieceOPTCG API (optcgapi.com)Same week as BandaiINGEST_TCG=onepiece INGEST_SETS=<id> ingest then embed~10 s ingest + ~30 s embed per ~200 cards

Step-by-step recipes ​

All commands run from the tcg-collector-backend repo root, with .env populated.

Magic — a new set just dropped on Scryfall ​

bash
# 1. Re-ingest. The bulk file is auto-refreshed if Scryfall has a newer
#    `updated_at`, so this picks up new cards transparently. Existing
#    images on disk are skipped.
docker compose --env-file .env -f docker/compose.yml run --rm \
  -e INGEST_TCG=mtg ingest

# 2. Embed only the new cards (the NOT EXISTS filter skips already-embedded).
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/embedCatalogLab.js" \
  -e INGEST_LAB_TCG=mtg ingest

# 3. Smoke — pick a card from the new set, scan it via /scan-lab.
curl -s -F "photo=@/path/to/test.jpg" http://<vps>:8080/scan-lab | jq .match

Pokémon — a new set just landed on TCGdex ​

bash
# 1. Re-ingest only the new set (use INGEST_SETS=all if you want a full sweep).
docker compose --env-file .env -f docker/compose.yml run --rm \
  -e INGEST_TCG=pokemon -e INGEST_SETS=<set-id> ingest

# 2. Embed for /scan-lab path. Until Phase F, /scan-compat won't see the
#    new cards because Tormknd's FAISS is frozen — but when we flip the
#    RN app to /scan-lab, the new cards become scannable.
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/embedCatalogLab.js" \
  -e INGEST_LAB_TCG=pokemon -e INGEST_LAB_SETS=<set-id> ingest

# 3. Smoke — same as Magic.

One Piece — a new booster set or starter deck ​

bash
# 1. Find the set id on optcgapi.com (e.g. OP-16 for the next booster).
docker compose --env-file .env -f docker/compose.yml run --rm \
  -e INGEST_TCG=onepiece -e INGEST_SETS=<set-id> ingest

# 2. Embed.
docker compose --env-file .env -f docker/compose.yml run --rm \
  --entrypoint "node dist/embedCatalogLab.js" \
  -e INGEST_LAB_TCG=onepiece -e INGEST_LAB_SETS=<set-id> ingest

# 3. Smoke. Note: ST starter decks are still excluded post-#101 revert
#    until variant logic is exercised on them — see scan-lab-plan.md.

Post-update verifications ​

Did the rows land? ​

sql
-- Per-TCG card + set counts (run inside the postgres container).
SELECT tcg, count(*) AS cards, count(DISTINCT set_id) AS sets
FROM cards GROUP BY tcg;

-- New cards from the last 24h.
SELECT id, name, set_id FROM cards
WHERE fetched_at > now() - interval '24 hours'
ORDER BY fetched_at DESC LIMIT 20;

Did embeddings land? ​

sql
-- Should match (cards count - cards without an image).
SELECT count(*) FROM card_embeddings_lab ce
JOIN cards c ON c.id = ce.card_id
WHERE c.tcg = '<tcg>';

Self-recall sanity check ​

Pick any indexed card, query its own embedding, expect sim ≈ 1.0:

sql
WITH q AS (SELECT embedding FROM card_embeddings_lab WHERE card_id = '<some-id>' LIMIT 1)
SELECT ce.card_id, c.name, (1 - (ce.embedding <=> q.embedding) / 2)::numeric(5,4) AS sim
FROM card_embeddings_lab ce
JOIN cards c ON c.id = ce.card_id, q
WHERE c.tcg = '<tcg>'
ORDER BY ce.embedding <=> q.embedding
LIMIT 3;

The first row should have sim = 1.0000. If it doesn't, the embed step missed it — re-run with EMBEDDINGS_FORCE=true for that subset to overwrite stale vectors.

Edge cases ​

Errata / reprint with a new image ​

If a card gets re-illustrated (rare but happens for Pokémon promos and MTG Mystery Booster), the upstream image_uris URL changes but the id stays the same. Our ingest will:

  • Upsert the cards row (image_base updated)
  • Re-download the image (the disk path embeds the id — collision detected, file overwritten)
  • NOT re-embed by default — the NOT EXISTS skip means the old vector stays. Override with EMBEDDINGS_FORCE=true to refresh embeddings for a known TCG.

A new variant for an existing card (OP parallel art, MTG showcase) ​

For OP, assignVariantIds already handles parallel-art collisions: a new entry sharing card_set_id with an existing one becomes id=<base>_v2 (or _v3, etc.) on next ingest. Disk path key is the variant-suffixed id, so no overwrite.

For MTG, Scryfall already emits a distinct collector_number per variant, so the variant lands as a fresh row with its own id (e.g. neo-100 vs neo-327 for showcase). No collision.

A new language for a set already ingested ​

Pokémon-only concern (TCGdex multi-lang). Re-running with INGEST_LANG=ja (or whatever new lang) lands the language-specific entries as separate rows, gated by the first-seen-wins de-dup in index.ts.

Tormknd FAISS lag for Pokémon / OP ​

Until Phase F flips the RN app to /scan-lab, scans for Pokémon and OP go through tcg-vision's baked FAISS. New cards we ingest will not be scannable through /scan-compat until either:

  1. Tormknd rebuilds his FAISS image folders + ships a new release; we bump TCG_APP_COMMIT and rebuild our tcg-vision image. Inertia depends on him.
  2. We finish Phase F and route everything through /scan-lab. Then a new Pokémon set is identical to a new MTG set: ingest + embed, done.

In the interim, the new cards are still useful for catalog browsing (they're in cards), just not for scanning via the live RN app.

Why everything is idempotent ​

Each layer of the pipeline has a deterministic skip rule, so re-running anywhere along the chain is safe:

LayerSkip rule
Set / card upsertsINSERT … ON CONFLICT (id) DO UPDATE — refreshes the row, never duplicates
Image downloadif (await exists(abs)) return skipped — same path, second run skips disk write
Scryfall bulk downloadCached filename embeds updated_at; same manifest = same filename = re-use
EmbeddingNOT EXISTS (… ce.card_id = c.id AND ce.model = $1) — skips already-embedded cards for that model

The implication: you cannot break the catalog by re-running. The worst case is you waste a few seconds doing nothing. This is the property that makes a cron-driven nightly refresh practical.

Future automation (not yet implemented) ​

A nightly cron job could run all three TCG ingests + embeds:

bash
# /etc/cron.d/tcg-catalog-refresh (illustrative — not yet wired)
0 4 * * * cd /opt/tcg-collector-backend && \
  docker compose --env-file .env -f docker/compose.yml run --rm \
    -e INGEST_TCG=mtg ingest >> /var/log/tcg-ingest.log 2>&1
0 5 * * * cd /opt/tcg-collector-backend && \
  docker compose --env-file .env -f docker/compose.yml run --rm \
    --entrypoint "node dist/embedCatalogLab.js" -e INGEST_LAB_TCG=mtg ingest \
    >> /var/log/tcg-embed.log 2>&1

Cost: ~2-5 min of compute per night per TCG when nothing's changed (Scryfall manifest check + skip), much more when a set drops. The bulk file cache + idempotent skip rules make this safe even if Claude / a human forgets to trigger it manually.

Troubleshooting ​

"I ingested but /scan-lab still says no-match" ​

You forgot the embed step. Ingest writes to cards + data/images/ only; without the embed step, card_embeddings_lab has no row for the new card and pgvector returns nothing. Run the embed command for that TCG.

"Embed reports done: 0" ​

The NOT EXISTS skip thinks everything's already embedded. If you really want to re-embed (e.g. after a Scryfall image refresh), set EMBEDDINGS_FORCE=true to bypass the skip.

"Upstream is returning 4xx / 5xx" ​

  • Scryfall: rare; their bulk endpoint is robust. Check https://api.scryfall.com/bulk-data manually.
  • TCGdex: occasional; the ingest's retry-with-backoff handles transient failures. If it consistently fails for one set, that set's data may not yet be published.
  • OPTCG API: their host (optcgapi.com) has had downtime; deferred Bandai L3 fallback (see docs/INGEST.md §One Piece) would close that gap but isn't wired.

"I see fewer rows than expected" ​

Likely a collision after sanitization. For MTG specifically, ~1.7 % of default_cards rows collide with another printing after Unicode collector_number sanitization (100 and 100★ both become set-100). The image is identical between collided rows, so the scan still resolves correctly — but the row count is lower than the ingest log's "rowsKept" number.

See also ​