Skip to content

Back up SHL game files with production archiver and standalone local runner - #113

Open
nisamson wants to merge 6 commits into
mainfrom
feature/shl-games-archive
Open

nisamson wants to merge 6 commits into
mainfrom
feature/shl-games-archive

Conversation

@nisamson

@nisamson nisamson commented Oct 8, 2026 •

Copy link
Copy Markdown
Owner

Summary

  • Add a six-hour production Quartz job that mirrors Apache-listed SHL CSV files into the separate shuttle-shl/shl-games-archive repository, using Key Vault credentials and in-process LibGit2Sharp.
  • Share synchronization logic through Shuttle.GameArchive and add a one-shot local runner with an isolated Aspire ArchiveOnly profile, persistent Gitea, and a Key Vault emulator; no API, SQL, or Azure login is needed.
  • Use depth-2 shallow temporary clones through LibGit2Sharp, with normal pushes preserving the remote's complete history. Document HTTP(S) transport requirements and staging/checkout disk overhead.
  • Record and skip successful non-Apache HTML directories, retaining removed content in Git history. HTTP failures, including listed-file 404s, and incomplete downloads abort publication.
  • Archive only .csv source files with case-insensitive matching, excluding every other extension and extensionless files. Previously managed non-CSV files are removed only after eligible downloads succeed; unmanaged files, the root README, generated manifest, and historical commits are preserved.
  • Document counts, sampled contents, and season distribution for observed source formats, including intentionally excluded formats. The 2026-10-08 listing inventory estimates 5,649 eligible CSV files / 0.481 GiB (493 MiB), outside the 40 custom-index boundaries. ZIPs are excluded rather than opened to extract CSVs.
  • Combine managed-file ownership and skipped-directory metadata in a strict, versioned manifest.json with a formal JSON Schema, ordinal path ordering, fixed property order, and stable LF formatting.
  • Document local testing, archive repository layout, exclusions, 404 semantics, and production activation.
  • Add user-approved, temporary NuGetAuditSuppress entries for the five known ImageSharp advisories only in the non-production Vision tool and its tests. NuGet Audit and warnings-as-errors remain enabled, with no exceptions in the deployed API/WebClient. Document the accepted local risk and removal on upgrade.

Validation

  • 152 targeted archive tests passed, covering crawling, runner guards, manifest validation, Git publishing/no-change behavior, failure preservation, and exclusions.
  • Extension coverage verifies case-insensitive CSV inclusion and rejection of every other inventoried format, unknown/extensionless files, and .csv.zip. Excluded entries still undergo path/collision checks, and directory names never determine eligibility. A Git regression verifies non-CSV files are never downloaded, failed eligible downloads preserve excluded managed files, successful runs remove only owned files, history remains intact, and ignored-content changes cause no commit.
  • Smart-HTTP Git integration coverage verifies exactly two reachable commits in temporary shallow clones, successful snapshot publication, preservation of all remote commits, and a shallow depth-2 clone after publication.
  • The complete Shuttle.Tests project passes in a clean worktree, excluding unrelated staged FHM changes.
  • Affected Aspire AppHost builds without warnings or errors.
  • Formal Draft 2020-12 schema and documented manifest example validated independently.
  • Full solution restore and all 843 tests passed in a clean worktree immediately before pushing: dotnet test --verbosity minimal -maxcpucount:1 -p:BuildInParallel=false (561 main, 202 bUnit, 6 E2E, 74 Vision). Serialized builds avoid an observed concurrent Blazor static-assets cache lock.
  • MSBuild evaluation verifies exactly five advisory exceptions in each Vision project, none in API/WebClient, and auditing/warnings-as-errors enabled in all four projects.
  • Earlier local fixture smoke runs verified Gitea publishing, no-change restarts, failure preservation, and persistence across AppHost restart; production publish artifacts were generated without deployment.

Production activation

This PR does not deploy or populate credentials. Bootstrap infrastructure/RBAC with FIRST_RUN=true, put the repository-scoped bot PAT directly in Key Vault, assess backfill capacity, and verify the first production snapshot. The production AppHost enables the archive by default; it can be paused with Archive:Enabled=false.

Custom HTML indexes remain intentionally excluded. The discussed Polly resilience policy is not implemented here; existing retry behavior is retained. No legacy manifest migration is included because the archive has not yet been deployed.

@nisamson
nisamson marked this pull request as ready for review October 8, 2026 18:21

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant