Documentation

Appearance

Session 2026-05-05 — VPS Option B (Tailscale) deployment end-to-end ​

Full session log: what we did, what broke, what we learned. Read top-down for the whole story, or jump to a specific decision.

Starting context ​

Hetzner Ubuntu Noble VPS (ubuntu-8gb-nbg1-1, public IP 188.245.104.87). Already bootstrapped (Docker, Go, gh installed; snag-tcg/* repos cloned). fail2ban applied in a previous session. SSH key-only hardening deferred.

Session goal: execute vps-tailscale-deployment.md Steps 1-6 (VPS side), then extend with full debugging when the phone failed to load scanned card images.

Phase 1 — Tailscale deployment (Steps 1-6 of the guide) ​

Step 1 — Install Tailscale + auth ​

  • curl -fsSL https://tailscale.com/install.sh -o /tmp/tailscale-install.sh && bash /tmp/tailscale-install.sh (preferred over direct pipe to avoid RTK + sudo gotchas)
  • sudo tailscale up launched in background, auth URL read from stdout, validated in laptop browser
  • VPS tailnet IP: 100.108.38.24

Step 2 — Pull the backend ​

Already done at the start of the session: checkout main + pull on all 4 repos (routines, tcg-collector-backend, poc-tcg-collector, documentation). The documentation repo didn't have the Tailscale guides locally — they were on remote branches (origin/docs/vps-tailscale-deployment, origin/docs/vps-public-deployment). Pulling main brought them in (already merged).

Step 3 — Configure .env ​

Generated 2 secrets via openssl rand -hex 32, displayed in plaintext ONCE for password manager save, then written to .env after user GO:

VarValue
PGPASSWORDhex 64 secret
API_WRITE_TOKENhex 64 secret
LOCAL_IMAGE_BASE_URLhttp://100.108.38.24:8081/images

chmod 600 .env, gitignored by .gitignore:15.

Step 4 — ufw ​

Chosen (vs port binding) since aligned with future Option A. Rules:

ALLOW IN on tailscale0      # everything via tailnet
DENY 8000, 8080, 8081, 5432  # defense in depth
ALLOW 22/tcp                # SSH allowed before enable, otherwise session cut
ufw --force enable

--force essential to avoid the "are you sure?" prompt that hangs without TTY.

Real talk: Docker bypasses ufw via DOCKER-USER/PREROUTING chains. So DENY 8000/8080/8081 doesn't really block Docker-published ports — the Hetzner Cloud Firewall upstream does the actual public-surface filtering. ufw still useful for 5432 (Postgres direct) and readability.

Step 5 — Docker stack — first partial attempt ​

bash
sg docker -c "docker compose -f docker/compose.yml up -d --build postgres api nginx tcg-vision"

sg docker -c because the SSH/tmux/Claude session inherited Unix groups frozen at fork() from before usermod -aG docker kael. tmux attach does not respawn any process → no point reconnecting for this session. Detailed in vps-troubleshooting §1.

Result: 3 services up out of 4. nginx failed to start:

mkdirat .../usr/share/nginx/html/images: read-only file system

Bug #1 — missing nginx mountpoint ​

docker/compose.yml stacks 2 bind mounts:

yaml
volumes:
  - ../app:/usr/share/nginx/html:ro                     # mount #1
  - ../data/images:/usr/share/nginx/html/images:ro      # mount #2 — INSIDE mount #1

Mount #2 needs an images/ mountpoint inside mount #1. The source app/images/ doesn't exist (git ignores empty directories). On strict Linux/runc, creating the mountpoint inside a :ro overlay is refused. Docker Desktop on Windows was lenient → never showed up before.

Fix applied: mkdir app/images on VPS. Upstream PR opened and merged: tcg-collector-backend#93 — adds app/images/.gitkeep so the mountpoint is checked out on clone.

After mkdir: recreate nginx → 4/4 services up.

Step 6 — First ingest ​

bash
sg docker -c "docker compose -f docker/compose.yml run --rm ingest"

With INGEST_SETS=base1,sv01: 8.3 seconds (vs 30 min announced by the guide). 360 cards, 720 images (low + high), 27.7 MB on disk, 0 fails.

Phase 2 — SSH key-only hardening ​

Deferred from previous session, picked up on user request. Procedure:

bash
echo "PasswordAuthentication no" | sudo tee /etc/ssh/sshd_config.d/99-disable-password.conf
sudo sshd -t   # validation
sudo systemctl reload ssh

Ubuntu Noble note: ssh.service is disabled but ssh.socket is enabled → socket activation. systemctl reload ssh works regardless.

Verification:

permitrootlogin without-password
pubkeyauthentication yes
passwordauthentication no       ← applied
kbdinteractiveauthentication no

Confirmed laptop-side: ssh kael@vps works (key), ssh -o PreferredAuthentications=password ... rejected.

Phase 3 — Documentation (vps-troubleshooting.md) ​

After several traps hit during the session, bilingual doc created to capture recurring patterns:

  1. Docker permission denied post-usermod -aG docker (Unix groups frozen at fork, tmux attach doesn't help)
  2. apt upgrade hanging on whiptail (DEBIAN_FRONTEND=noninteractive + NEEDRESTART_MODE=a)
  3. gpg --dearmor + curl pipe → "invalid radix64" (save first)
  4. gpg --dearmor + no-TTY → "cannot open /dev/tty" (--batch --yes)
  5. RTK + sudo → "rtk: command not found" (/usr/bin/sudo /usr/bin/...)
  6. gh auth login device code (Claude must read stdout, not refuse)

PR merged: documentation#7.

Phase 4 — Step 7 (laptop) — repoint the mobile app ​

User-side, on laptop: poc-tcg-collector/src/scan/backendConfig.ts:45 modified to point at 100.108.38.24:8000 instead of 100.76.15.57:8000 (laptop). PR #107 merged. APK rebuilt, installed on the OnePlus Nord 4.

First test from phone: Network request failed.

Phase 5 — Debug "Network request failed" ​

Bug #2 — network_security_config.xml hardcoded whitelist ​

User inspected poc-tcg-collector/android/app/src/main/res/xml/network_security_config.xml and found a cleartext IP whitelist hardcoded on the old laptop IP:

xml
<domain-config cleartextTrafficPermitted="true">
    <domain includeSubdomains="true">localhost</domain>
    <domain includeSubdomains="true">127.0.0.1</domain>
    <domain includeSubdomains="true">10.0.2.2</domain>
    <domain includeSubdomains="true">100.76.15.57</domain>   ← laptop only
</domain-config>

Android API 28+ blocks HTTP cleartext by default, except for hosts whitelisted here. The new VPS IP 100.108.38.24 wasn't there → <Image> (Fresco) refuses the fetch.

Fix: add 100.108.38.24 to the whitelist. PR #126 merged. APK rebuilt + reinstalled.

Phase 6 — Debug "Card details OK but image won't load" ​

The scan POST goes through (visible in tcg-vision logs), card details display in the app, but the image stays blank and no GET /images/... shows up in nginx logs.

Bug #3 — Docker Compose doesn't load .env ​

Inspected effective env in the tcg-vision container:

LOCAL_IMAGE_BASE_URL=

Empty. But the repo-root .env does have LOCAL_IMAGE_BASE_URL=http://100.108.38.24:8081/images.

Cause: Docker Compose v2 looks for .env in the project directory, which by default is the directory of the compose.yml (so docker/). Not in cwd. docker/.env doesn't exist → no var loaded → ${LOCAL_IMAGE_BASE_URL:-} resolves to empty string. Same for ${PGPASSWORD:-tcg} falling back to default tcg.

Consequences:

  • Postgres was running with default password tcg (while exposed on the tailnet — security issue)
  • tcg-vision was NOT rewriting image URLs (returning the default TCGdex CDN URL, not our nginx)

Fix:

bash
# 1. Align Postgres live with the .env secret
PGPASSWORD_REAL=$(grep ^PGPASSWORD= .env | cut -d= -f2)
sg docker -c "docker exec tcg-collector-postgres-1 psql -U tcg -c \"ALTER USER tcg WITH PASSWORD '$PGPASSWORD_REAL';\""

# 2. Always pass --env-file explicitly
sg docker -c "docker compose --env-file .env -f docker/compose.yml up -d --force-recreate"

Post-recreate verification:

LOCAL_IMAGE_BASE_URL=http://100.108.38.24:8081/images   ✅
POSTGRES_PASSWORD=492c…b5fe                             ✅

Takeaway: always --env-file .env when running docker compose from the repo root. To document upstream in tcg-collector-backend/CLAUDE.md, or fix by removing dangerous defaults from compose (${PGPASSWORD:?missing} instead of ${PGPASSWORD:-tcg}), or by moving .env to docker/.env.

Bug #4 — FAISS vs local disk coverage ​

After the env fix, the app does receive URLs rewritten to nginx. But still no image loading.

Laptop test with curl -I http://100.108.38.24:8081/images/base1/2/high.webp → 200 OK (image available). Test in phone Chrome at the same URL → image displayed. So network OK + server OK + cleartext config OK.

So the app remains. Inspecting poc-tcg-collector/src/scan/ScanResultCard.tsx:21:

tsx
{details.image ? (
  <Image source={{ uri: details.image }} ... />
) : null}

If details.image is null/empty → <Image> never renders → 0 GETs, consistent with empty nginx logs.

Exhaustive test on VPS:

bash
curl -X POST -F "photo=@data/images/base1/25/high.webp" http://100.108.38.24:8000/scan-compat
# → match: xy12-29 Lamantine
# → image: http://100.108.38.24:8081/images/xy12/29/high.webp

I uploaded a base1/25 card (Dewgong) and FAISS returned a different Lamantine, from set xy12. xy12 is not on disk → the returned image URL 404s.

SourceCards
FAISS index in tcg-vision30,370
Locally ingested sets (base1 + sv01)360
Coverage1.1%

→ 98.9% of real-photo scans (noise, perspective, glare) will land on a card we don't have locally. URL → 404. And Android <Image> silently drops it (either before the request, or after receiving the 404).

Chosen fix (Option A) — re-ingest all sets ✅ ​

bash
sed -i 's/^INGEST_SETS=.*/INGEST_SETS=all/' .env
sg docker -c "docker compose --env-file .env -f docker/compose.yml run --rm ingest"

Actual result: 7 min 36 s (not the announced hours), 208 sets, 23,159 cards, 43,078 images, 1.84 GB on disk. FAISS coverage went from 1.1 % to 76 %.

Long-term fix (Option C) — TCGdex fallback on tcg-vision side ✅ ​

Patched and merged: tcg-collector-backend#94. _build_card_image_url now checks os.path.exists() before rewriting the URL; missing file → TCGdex CDN fallback instead of a guaranteed 404. compose.yml now mounts data/images:ro for tcg-vision, and the new LOCAL_IMAGE_DATA_DIR env var (defaults to /app/data/images) enables the check. Covers the residual ~24 % of cards not on disk (Pokémon JP, One Piece, Lorcana, etc.).

Final state ​

What works today ​

  • VPS up, secured (fail2ban + ufw + Hetzner FW + SSH key-only)
  • Tailscale node 100.108.38.24 reachable from the phone
  • Docker stack 4/4 services healthy (postgres, api, nginx, tcg-vision)
  • .env properly loaded via --env-file
  • App identifies cards (POST scan-compat → 200, details displayed)

What's left to finish ​

Everything identified during the session was closed before it ended:

  • Ingest all sets ✅ done (7 min, 23k cards, 76% FAISS coverage)
  • Upstream PR fix/tcg-vision-image-fallback ✅ merged (backend#94)
  • Upstream --env-file note ✅ enforced via backend#95 (compose hard-fails when PGPASSWORD is missing + CLAUDE.md updated + header in .env.example)
  • Step 9 (optional, laptop side): docker compose down to free up laptop Docker resources — still pending user choice

Phase 7 — Late-session security hardening ​

After the initial doc, security fix to prevent trap #3 (.env silently not loaded → Postgres on default password) from recurring:

  • ${PGPASSWORD:-tcg} × 4 services → ${PGPASSWORD:?missing PGPASSWORD — pass --env-file .env or export it}. Compose hard-errors out if PGPASSWORD isn't loaded, instead of booting with a known default.
  • CLAUDE.md documents --env-file .env in every docker compose command.
  • Header in .env.example reminds about the requirement.

PR merged: tcg-collector-backend#95. Verified both sides:

  • With --env-file .env → strong password resolves, everything works
  • Without → explicit fail at compose config-time, message points to the fix

Traps hit (summary) ​

  1. tmux attach doesn't reload Unix groups (groups frozen at child process fork)
  2. app/images/ mountpoint missing on strict Linux — fixed by backend#93
  3. Android network_security_config.xml cleartext whitelist by IP — fixed by poc#126
  4. Docker Compose v2 looks for .env in project dir = compose.yml directory, not cwd — fixed by backend#95 (fail-fast instead of silent fallback)
  5. tcg-vision blanket-rewrites to nginx without checking if file exists → silent 404 on Android Fresco — fixed by backend#94

Security measures applied ​

  • Postgres PGPASSWORD random hex 64
  • API API_WRITE_TOKEN random hex 64
  • SSH key-only, password disabled
  • ufw active, default deny incoming
  • .env chmod 600, gitignored

Internal references ​