Recognition optimization strategy

Status: working strategy captured on 2026-05-14; sealed accuracy diagnosis updated on 2026-05-20.

This page records the scanner recognition discussion for cards, slabs, and sealed products. It is not a final architecture freeze. It is the current direction for making scan results faster, more accurate, and more explainable while the Android app remains a test surface.

Goal

The target scanner must feel instant, but the real product promise is accuracy: a user should be able to scan a raw card, a graded card, or a sealed product and understand why SKN picked that result.

The long-term recognition target is:

• very high top-1 accuracy for clean captures;
• explicit ambiguity when products are visually too close;
• top-K diagnostics in debug mode;
• support for raw cards, slabs, graded labels, booster packs, displays, ETBs, tins, blisters, collections, and decks;
• an explainable result that can combine visual similarity, OCR evidence, product metadata, language, and pricing context.

Current mental model

The current visual retrieval stack should be understood as two separate responsibilities.

DINOv2 is the visual encoder. It turns an image into an embedding: a vector of numbers that represents the visual content. It does not directly answer "this is Base Set Aerodactyl sealed product". It produces a representation where visually similar images should be close together.

FAISS and pgvector are nearest-neighbour search engines. They search through embeddings that were already computed for reference assets and return the closest candidates.

The current sealed lab flow is therefore:

reference sealed images
-> DINOv2 embeddings
-> vector index

phone capture
-> DINOv2 embedding
-> nearest-neighbour search
-> top-K sealed candidates
-> ambiguity guard / result

This is visual similarity, not OCR. It can be fast and useful, but it does not reliably read product names, set codes, language, grading labels, or barcodes.

Re-indexing versus retraining

The current system uses DINOv2 frozen. Most improvements should come from better indexing and evaluation before any model fine-tuning.

Re-indexing means:

new reference image
-> DINOv2 embedding
-> insert into FAISS / pgvector index

This is relatively fast, operationally simple, and should happen whenever the reference corpus changes.

Retraining or fine-tuning means modifying model weights. That is a later-stage investment that requires a labelled dataset, hard negatives, GPU budget, and stable evaluation metrics. For sealed products, it should not be the first fix.

Recommended order:

1. Build a labelled phone-photo benchmark.
2. Improve reference coverage and metadata.
3. Add product-type and language filters.
4. Improve crop and perspective handling.
5. Add OCR and hybrid reranking.
6. Add a trained reranker or metric-learning fine-tune only after the benchmark shows where retrieval fails.

Why sealed is harder than cards

Sealed products are not card-shaped objects. A booster pack, ETB, display, blister, tin, and collection box have different aspect ratios, reflective surfaces, logos, product names, regional text, and repeated artwork.

The current ambiguous-neighbor behavior is intentional. If the top candidate is too close to the runner-up, the backend should refuse to pretend it knows. This is a false-positive control, not the final accuracy solution.

The next accuracy gains for sealed should come from:

• sealed-specific object detection and cropping;
• product-type filters from the UI;
• OCR over top-K candidates;
• a hybrid reranker that weighs visual similarity, text evidence, language, set metadata, and product type;
• benchmark images captured from real phones, including glare, angle, screenshots, screens, plastic wrap, and partial crops.

Current sealed accuracy diagnosis — 2026-05-16

The first pass over ten SKN-owned French sealed photos showed the real blocker clearly:

• before French references existed, strict benchmark result was 0/10 exact or top-K recoveries;
• unfiltered visual retrieval still returned useful neighbours: 8 of 10 images returned ambiguous-neighbor, and 2 of 10 images returned a non-French top-1 match;
• filtered retrieval with lang=fr returned no-match-after-filter on all 10 images;
• the indexed sealed catalog contained en, de, ja, and ko products, but no fr sealed products.

The 2026-05-16 bootstrap fix added a separate skn-owned source with the ten French owned photos as approved references, then re-ingested the sealed index. The benchmark now reports:

• 10/10 exact matches without filters;
• 10/10 exact matches with --use-label-filters;
• skn-owned rows are approved, stable-ID bootstrap references.

This result validates the French sealed ingest/query/debug path. It is not yet a production market-accuracy score, because the benchmark photos are also the bootstrap reference images.

The 2026-05-20 follow-up diagnosis addressed a false mental model from field testing: the sealed scanner was not comparing only against the ten skn-owned photos. The active index contains the broad ptcg-assets corpus plus those owned references. The practical issue was that the app did not show candidate source clearly, and the backend reranker only had the narrow first candidate set available.

The backend now:

• retrieves a wider internal sealed candidate pool before OCR/text reranking;
• still returns a compact debug top-K to the mobile app;
• exposes candidate source and productType so failures can be grouped by ptcg-assets versus skn-owned and by sealed family;
• maps Wizards OCR terms to real ptcg-assets set IDs: base1, base2, base3, and base5.

The 2026-05-23 follow-up paused skn-owned for normal sealed scans. Those rows are useful for owned-reference wiring and benchmark experiments, but they can dominate real scans because they are a small, unbalanced bootstrap set instead of a balanced market catalog. Runtime /scan-sealed-lab now defaults to source=ptcg-assets; use source=skn-owned or source=all only when the test deliberately needs those rows.

This does not make generic sealed matching equivalent to a booster-only demo. It makes the debug and reranking honest. Tormknd's visible booster path is constrained to boosters; SKN's mobile Sealed mode is currently broader and must compare booster packs, ETBs, displays, blisters, tins, collection boxes, and decks unless a subtype filter or detector-backed routing is added.

Two observed failure modes matter most:

• Language / SKU confusion. A French Base Set Blastoise booster photo can retrieve the German Base Set Blastoise booster as top-1 because the visual artwork is close and no French reference exists.
• Product-type confusion. A Wizards blister photo can retrieve booster packs or unrelated blisters because the current flow embeds the whole image and does not yet detect or enforce the product type.

This confirms that the next sealed improvements should not be a simple threshold tweak. The priority is now:

1. collect unseen French sealed phone photos and keep them out of the reference set;
2. keep SKN-owned references opt-in until field evaluation proves they improve unseen photos;
3. add more owned or licensed French sealed references;
4. add product-type detection/filtering;
5. add OCR and hybrid reranking over the visual top-K;
6. keep using the sealed benchmark before changing model weights.

DogBreed_Classifier takeaways

The local DogBreed_Classifier repository is not directly reusable for sealed recognition. It is a closed-set TensorFlow/Keras classifier trained for dog breeds, and the local .h5 files are Git LFS pointers rather than downloaded model weights.

It is still useful as a reference pattern:

• image folders grouped by class;
• train/validation split;
• transfer learning;
• augmentation;
• top-1 accuracy tracking;
• small inference app that returns a label and confidence.

For SKN, the useful adaptation is not "use the dog model". The useful adaptation is to create a sealed-product benchmark and possibly a sealed classifier/reranker once enough labelled examples exist.

YOLO role

YOLO is useful for detection, not final product identity.

The practical pipeline should be:

raw phone image
-> YOLO detects card / slab / sealed / graded label / barcode zone
-> crop or segment the relevant region
-> DINOv2 + vector search retrieves visual candidates
-> OCR reads targeted text zones
-> hybrid reranker chooses or refuses a result

YOLOv10 is attractive for low-latency detection because its design removes NMS during inference and focuses on real-time efficiency. YOLO11 is the more pragmatic first candidate for this project because the Ultralytics ecosystem supports detection, segmentation, classification, oriented boxes, training, validation, and export from one workflow.

Initial YOLO classes should stay broad:

• card
• slab
• sealed
• graded_label
• barcode

Only after broad detection works should the model learn finer sealed classes such as booster_pack, booster_box, elite_trainer_box, tin, blister, and collection_box.

Legal note: Ultralytics YOLO11 is distributed under AGPL-3.0 or commercial Enterprise terms. Any production use in a closed commercial app must go through a licensing review before shipping.

Real-photo dataset for sealed detection

ptcg-assets can bootstrap sealed typology because its folders already encode useful product families: packshots, Elite Trainer Boxes, displays, collections, blisters, tins, mini-tins, booster bundles, and decks. That makes it a good source for generating a first internal YOLO dataset with broad classes and simple boxes.

It is not enough by itself. ptcg-assets images are mostly clean catalog images. The Android scanner will receive real phone captures with table backgrounds, hands, glare, plastic reflections, perspective, partial crops, screen photos, low light, and products that are not centered. A detector trained only on clean catalog assets can learn "catalog image" instead of "real sealed object".

YOLO should classify typology, not exact product identity:

sealed_booster_pack
sealed_blister
sealed_booster_display
sealed_elite_trainer_box
sealed_collection_box
sealed_tin
sealed_booster_bundle
card
slab
slab_label

Do not train YOLO classes such as Base Set Blastoise Booster FR. Exact product identity belongs to DINOv2 retrieval, metadata filters, OCR, and reranking after the crop.

Recommended dataset targets:

• POC useful: 30 to 50 real phone photos per priority class, about 200 to 350 photos total for booster_pack, blister, elite_trainer_box, booster_display, collection_box, slab, and card.
• Stable beta: 100 to 200 real phone photos per class, about 700 to 1,400 photos total for the same classes.
• Product-grade: 300+ real photos per class, 2,000+ photos over time, with hard negatives and repeated failure cases.

The practical first target is:

50 ETB
50 booster packs
50 blisters
50 displays
50 collection boxes
30 slabs
30 cards

That gives roughly 310 real photos. Keep the split honest:

• training set: used to train/fine-tune YOLO;
• validation set: used while tuning;
• test set: never used for training or tuning, used only for reported accuracy.

For a first 310-photo set, a pragmatic split is about 250 train / 60 test, or 220 train / 30 validation / 60 test if a validation split is needed. The test photos must remain unseen and should include ugly captures, not only clean photos.

Sealed field-evaluation protocol

Before training YOLO or changing similarity thresholds, SKN needs an honest field-evaluation loop for sealed retrieval.

The backend now separates two fixture roles:

• tests/fixtures/sealed-samples: SKN-owned bootstrap references that can be ingested as skn-owned sealed products.
• tests/fixtures/sealed-eval: SKN-owned or explicitly permitted phone photos used only for evaluation. These rows must use evaluationOnly: true and are blocked from SKN-owned reference ingest.

This distinction matters. If the same image is both reference and benchmark sample, a 100% score only proves wiring. If the sample is unseen by the index, failures are useful product data.

Run the evaluation with:

cd tcg-collector-backend/src/api
npm run benchmark:sealed:eval -- --json /tmp/sealed-field-eval.json

Use the report to choose the next engineering action:

• expected product in top-K but not top-1: add OCR and hybrid reranking;
• correct product type missing from top-K: add references or improve crop;
• wrong product type dominates: add typology detection or force a UI filter;
• all candidates visually noisy: collect more field captures and prioritize detector crop;
• recurring language/SKU confusion: add language evidence and region-specific references.

The report exposes these machine-readable fields so failure review can be grouped across many photos:

• diagnosis=expected-in-topk, nextAction=add-ocr-reranking;
• diagnosis=wrong-product-type, nextAction=add-type-filter-or-detector-crop;
• diagnosis=wrong-language, nextAction=add-language-evidence-or-reference;
• diagnosis=expected-absent, nextAction=add-reference-or-improve-crop;
• diagnosis=unlabeled-review, nextAction=label-sample.

Use the aggregate summary.diagnoses and summary.nextActions counts to decide whether the next sprint should focus on OCR/reranking, typology/crop, language/reference coverage, or more reference/crop work.

OCR and reranking

OCR should not be the first blocking step. Whole-image OCR is noisy on shiny packaging, plastic wrap, angled photos, and small foil typography.

OCR should be targeted after candidate retrieval:

visual top-K
-> OCR on product title / set code / language / graded label / barcode zones
-> compare OCR evidence to candidate metadata
-> rerank or mark ambiguous

For slabs, OCR should focus on:

• grading company;
• grade value;
• label card name;
• certification number;
• set or year when visible.

For sealed, OCR should focus on:

• product name;
• set name;
• language;
• product type;
• barcode or product code when visible.

For raw cards, OCR remains focused on:

• collector number;
• set code or set symbol;
• rarity;
• language when needed.

Target architecture

capture / imported image
-> object detection: card / slab / sealed / label / barcode
-> crop, deskew, and quality scoring
-> DINOv2 visual embedding
-> vector retrieval with metadata filters
-> targeted OCR on relevant zones
-> hybrid reranking
-> confidence and ambiguity policy
-> final result or explicit ambiguous result

Debug mode must preserve every candidate and every score. The normal scanner should show a fast, clean result, while Scan Debug should expose the full reasoning path.

Scan Debug cases can also be uploaded to the backend lab archive. This turns individual phone scans into reviewable machine-learning intake: input image, optional crop, backend JSON, top-K candidates, scores, timings, and failure reason. These cases are not training data until a reviewer adds the expected product, correctness verdict, rights status, and failure label.

Mobile framing status

As of 2026-05-20, the React Native scanner exposes manual frame presets before capture:

• card: standard card frame and tighter zoomed frame;
• slab: full slab frame and label-focused frame;
• sealed: vertical booster pack, rectangular box, square product, and wide display frames.

Each preset owns both the guide geometry and the requested camera zoom. This improves field testing immediately because users no longer have to force every sealed product into the same vertical overlay.

This is not automatic object detection yet. True auto framing still belongs to the detector-backed path:

live camera / imported image
-> detect card / slab / sealed object
-> pick the matching crop geometry
-> crop or warp
-> send the normalized image to retrieval

Until that path ships, the manual preset is the user-controlled hint. The Pack preset is intentionally wired as a booster-specific lab path: the app sends preprocess=sealed&typeHint=booster_pack, so the backend can mirror the tcg-app2 booster flow by cropping the pack first while keeping the hint softer than the backend's hard type=booster_pack A/B filter. Box, Square, and Wide remain broader sealed-lab presets for comparison until their detector/type routing is proven.

Benchmark before model changes

The project needs a small but real benchmark before adding heavier ML.

Minimum useful sealed benchmark:

• 5 to 10 real phone photos per target product for early smoke tests;
• 30 to 50 photos per confusing product family for meaningful accuracy work;
• negative examples from visually similar sealed products;
• captures with glare, angle, low light, crop errors, screen photos, and packaging reflections;
• expected product id, type, language, and notes.

Metrics:

• top-1 accuracy;
• top-3 and top-5 recall;
• ambiguity rate;
• false-positive rate;
• latency by stage;
• OCR hit rate on targeted zones;
• correction success when user picks from top-K.

Recommended roadmap

Now

• Keep the five card endpoints for app testing.
• Keep the sealed endpoint as the lab path.
• Add imported-image scanning to speed up repeated tests.
• Preserve top-K diagnostics in Scan Debug.
• Archive Scan Debug cases for later human labelling and benchmark promotion.
• Use the Pack preset as the booster-specific path, and compare it against the broader sealed presets.
• Collect real phone samples for sealed failures.

Next

• Add broad YOLO detection for card / slab / sealed.
• Apply sealed-specific crop before DINOv2 retrieval.
• Add safe product-type hints for box, square, and wide sealed presets after benchmark evidence.
• Add OCR over sealed top-K candidates.
• Add a hybrid reranker and keep ambiguous-neighbor as a safe refusal path.

Later

• Train a sealed reranker or classifier using the benchmark dataset.
• Consider metric-learning fine-tuning only after the benchmark proves the failure mode.
• Add slab-specific label OCR and grading-company parsing.
• Add production-safe asset sources and licensing controls before public serving.

External references

• DINOv2 paper
• FAISS documentation
• pgvector repository
• YOLOv10 paper
• Ultralytics YOLOv10 documentation
• Ultralytics YOLO11 documentation

Related pages

• Sealed scanner lab
• Modern TCG scanner architecture
• Scan evaluation protocol
• Scan UI/UX competitive review
• Asset licensing and source risk register