Back up SHL game files with production archiver and standalone local runner

.github/copilot-instructions.md

@@ -15,6 +15,8 @@ consistent when you change one. `Shuttle.Analysis/README.md` documents the CLI t
 
 - `dotnet restore` — restore. NuGetAudit runs here and **fails restore** on any known
   vulnerable package (audit warnings are errors).
+  Five ImageSharp advisories are temporarily suppressed only in the non-production
+  `Shuttle.Fhm.Vision` and its tests (see its README); other advisories still fail restore.
 - `dotnet build` / `dotnet build --no-restore -c Debug` — build all / after a restore.
 - `dotnet build <Project>/<Project>.csproj` — build one project.
 - `dotnet test` — full suite (`Shuttle.Tests`, xunit.v3 via Microsoft Testing Platform).
@@ -38,6 +40,21 @@ Two shipped apps over shared libraries, orchestrated by an Aspire AppHost.
   through `ISelfRegisteringJob`, and pull from the upstream SHL Index/Portal APIs
   (`Shuttle.Shl.Api.Client`) to keep the DB current. Ships the CrystalQuartz dashboard at
   `/quartz`. Authenticates API requests with JWT bearer (Entra ID, default scheme).
+  A six-hour `GameArchiveJob` (enabled in production, opt-in locally) mirrors public `/games/` files into the
+  separate `shuttle-shl/shl-games-archive` repository using a Key Vault secret;
+  local development uses an emulated vault and persistent Gitea
+  (see `docs/game-archive.md`).
+  Shared archive services live in `Shuttle.GameArchive`. The local
+  `Shuttle.GameArchive.Runner` uses the AppHost's `ArchiveOnly` profile for one
+  real-source sync to persistent Gitea via the vault emulator, without API,
+  SQL, Azure login, or Quartz; Git/vault destinations must be loopback.
+  Non-Apache HTML directories are skipped with warning logs and a committed
+  versioned `manifest.json` combining owned file paths and skipped directories
+  (schema: `docs/game-archive-manifest.schema.json`); their managed files are
+  removed from the current snapshot but remain in history. HTTP/download
+  failures still abort publishing.
+  Only source files ending in `.csv` are archived (case-insensitive matching);
+  previously managed non-CSV files are removed only after a successful sync.
 - **`Shuttle.WebClient`** — standalone Blazor WebAssembly front end using **Fluent UI Blazor**
   and MSAL. No server host; it calls `Shuttle.Api` over HTTP.
 - **`Shuttle.EFCore`** — data layer: `ShlDbContext`, entities, migrations. **Azure SQL** with

AGENTS.md

@@ -21,6 +21,28 @@ websites, stores it in Azure SQL, and serves it to the front end.
 `Shuttle.Api` periodically pulls from them (via its background Quartz jobs) to keep the
 database current.
 
+`GameArchiveJob` is a separate six-hour Quartz job, enabled by default in the
+production AppHost (but not local development). When enabled it mirrors the public Apache directory
+`https://simulationhockey.com/games/` into the separate
+`shuttle-shl/shl-games-archive` Git repository. It uses an Azure Key Vault
+secret for the `shuttle-bot` Git credential; local development uses an
+emulated vault and persistent Gitea. The job must complete all downloads
+before changing the checkout and must not delete files after failed scans.
+Successful non-Apache HTML directories are intentionally skipped and documented
+in the versioned `manifest.json` with warning logs. It combines sorted owned
+file paths and skipped-directory metadata; its JSON Schema is in
+`docs/game-archive-manifest.schema.json`. Their managed
+files are removed from the current snapshot, but retained in Git history;
+HTTP/download failures still abort without publishing.
+Only source files ending in `.csv` are archived (case-insensitive matching);
+previously managed non-CSV files are removed only after a successful sync.
+See `docs/game-archive.md`.
+`Shuttle.GameArchive` owns the shared archive services; the API owns only the
+Quartz job. `Shuttle.GameArchive.Runner` performs a local one-shot sync through
+the AppHost's `ArchiveOnly` launch profile (real SHL source by default, loopback
+Git/vault destinations only), without API, SQL, Azure login, or Quartz. Gitea
+persists and remains running after the runner exits; restart the runner manually.
+
 ### Competitions
 
 The broader game includes several competitions that this solution reports on:
@@ -259,6 +281,9 @@ SDK-style .NET 10 solution (`SHLAnalytics.sln`) using Central Package Management
 
 - `dotnet restore` — restore packages. NuGetAudit runs here and **fails the restore** on
   any known-vulnerable package (warnings are treated as errors).
+  The only temporary exceptions are five project-local ImageSharp advisory
+  suppressions in `Shuttle.Fhm.Vision` and its tests, which are not deployed;
+  see `Shuttle.Fhm.Vision/README.md`. Other advisories still fail restore.
 - `dotnet build` — build the whole solution.
 - `dotnet build --no-restore -c Debug` — build without re-restoring (after a restore).
 - `dotnet build <Project>/<Project>.csproj` — build a single project.

Directory.Packages.props

@@ -14,11 +14,14 @@
     <PackageVersion Include="Aspire.Hosting.Azure.AppContainers" Version="13.1.0" />
     <PackageVersion Include="Aspire.Hosting.Azure.AppService" Version="13.4.6" />
     <PackageVersion Include="Aspire.Hosting.Azure.Functions" Version="13.4.6" />
+    <PackageVersion Include="Aspire.Hosting.Azure.KeyVault" Version="13.4.6" />
     <PackageVersion Include="Aspire.Hosting.Azure.Sql" Version="13.4.6" />
     <PackageVersion Include="Aspire.Hosting.Azure.Storage" Version="13.1.0" />
     <PackageVersion Include="Aspire.Microsoft.Data.SqlClient" Version="13.4.6" />
     <PackageVersion Include="AngleSharp" Version="1.5.0" />
-    <PackageVersion Include="Azure.Identity" Version="1.18.0" />
+    <PackageVersion Include="Azure.Identity" Version="1.21.0" />
+    <PackageVersion Include="Azure.Security.KeyVault.Secrets" Version="4.11.1" />
+    <PackageVersion Include="AzureKeyVaultEmulator.Aspire.Hosting" Version="3.1.3" />
     <PackageVersion Include="Azure.Monitor.OpenTelemetry.AspNetCore" Version="1.4.0" />
     <PackageVersion Include="BinarySerializer" Version="10.1.0" />
     <PackageVersion Include="Blazor.LocalStorage.WebAssembly" Version="9.0.1" />
@@ -32,6 +35,7 @@
     <PackageVersion Include="FlexLabs.EntityFrameworkCore.Upsert" Version="10.0.0-rc1" />
     <PackageVersion Include="JetBrains.Annotations" Version="2025.2.4" />
     <PackageVersion Include="linq2db.EntityFrameworkCore" Version="10.1.0" />
+    <PackageVersion Include="LibGit2Sharp" Version="0.32.0" />
     <PackageVersion Include="Markdig" Version="0.42.0" />
     <PackageVersion Include="Mapster" Version="10.0.12" />
     <PackageVersion Include="Microsoft.ApplicationInsights.WorkerService" Version="2.23.0" />
@@ -69,6 +73,8 @@
     <PackageVersion Include="Microsoft.FluentUI.AspNetCore.Components.Icons" Version="$(FluentUIVersion)" />
     <PackageVersion Include="Microsoft.Extensions.Hosting" Version="10.0.3" />
     <PackageVersion Include="Microsoft.Extensions.Hosting.Abstractions" Version="10.0.3" />
+    <PackageVersion Include="Microsoft.Extensions.Http" Version="10.0.3" />
+    <PackageVersion Include="Microsoft.Extensions.Options.ConfigurationExtensions" Version="10.0.3" />
     <PackageVersion Include="Microsoft.Extensions.Http.Polly" Version="10.0.3" />
     <PackageVersion Include="Microsoft.Extensions.Http.Resilience" Version="10.1.0" />
     <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.9" />

README.md

@@ -32,7 +32,9 @@ or a recreate — so scouts can evaluate draftees at a glance.
 **Shipped apps**
 
 - **`Shuttle.Api`** — ASP.NET Core Web API that serves the backend and hosts scheduled
-  background ingestion jobs (Quartz.NET, persistent SQL job store).
+  background ingestion jobs (Quartz.NET, persistent SQL job store). A scheduled job
+  mirrors public game files from `https://simulationhockey.com/games/` into
+  [`shuttle-shl/shl-games-archive`](https://github.com/shuttle-shl/shl-games-archive).
 - **`Shuttle.WebClient`** — Blazor WebAssembly (standalone) front end built with Fluent UI Blazor.
 
 **Shared libraries**
@@ -46,16 +48,37 @@ or a recreate — so scouts can evaluate draftees at a glance.
   domain models and shared calculation utilities.
 - **`Shuttle.ServiceDefaults`** — shared Aspire service defaults (OpenTelemetry, health
   endpoints).
+- **`Shuttle.GameArchive`** — shared crawling, Key Vault credentials, and Git snapshot
+  synchronization for the production job and local runner.
 
 **Orchestration & tests**
 
 - **`Shuttle.Backend.Aspire`** — .NET Aspire AppHost that orchestrates the backend and
   wires it to Azure SQL and Application Insights.
 - **`Shuttle.Tests`** — the test suite (xUnit).
+- **`Shuttle.GameArchive.Runner`** — one-shot local archive executable, launched by the
+  AppHost's `ArchiveOnly` profile without the API or SQL stack.
 
 Authentication throughout uses Entra ID via `Microsoft.Identity.Web`, and observability is
 provided by OpenTelemetry.
 
+### Game-file archive
+
+The game-file archive runs by default in the production AppHost; the ordinary local
+API requires explicit enablement. For an isolated, one-shot sync of the real SHL
+source to local Gitea, with Docker running:
+
+```powershell
+dotnet run --project Shuttle.Backend.Aspire\Shuttle.Backend.Aspire.csproj --launch-profile ArchiveOnly
+```
+
+This profile needs no Azure login or SQL. See
+[`docs/game-archive.md`](docs/game-archive.md) for local Gitea + Key Vault Emulator
+development, production Key Vault setup, and safe activation.
+Non-Apache HTML directories are documented in the archive's
+`manifest.json`'s `skippedDirectories` and excluded from the current snapshot;
+previously archived content remains in Git history.
+
 ## But for where?
 
 Here!

SHLAnalytics.sln

@@ -51,6 +51,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Shuttle.BinarySerde.Common"
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Shuttle.Fhm.Serde.Sqlite", "Shuttle.Fhm.Serde.Sqlite\Shuttle.Fhm.Serde.Sqlite.csproj", "{6F3675E7-86BB-4FE6-8736-F4E4880824A1}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Shuttle.GameArchive", "Shuttle.GameArchive\Shuttle.GameArchive.csproj", "{EFBF8896-46D1-499B-87E8-72B1B0A9243D}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Shuttle.GameArchive.Runner", "Shuttle.GameArchive.Runner\Shuttle.GameArchive.Runner.csproj", "{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -349,6 +353,30 @@ Global
 		{6F3675E7-86BB-4FE6-8736-F4E4880824A1}.Release|x64.Build.0 = Release|Any CPU
 		{6F3675E7-86BB-4FE6-8736-F4E4880824A1}.Release|x86.ActiveCfg = Release|Any CPU
 		{6F3675E7-86BB-4FE6-8736-F4E4880824A1}.Release|x86.Build.0 = Release|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Debug|x64.Build.0 = Debug|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Debug|x86.Build.0 = Debug|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Release|Any CPU.Build.0 = Release|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Release|x64.ActiveCfg = Release|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Release|x64.Build.0 = Release|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Release|x86.ActiveCfg = Release|Any CPU
+		{EFBF8896-46D1-499B-87E8-72B1B0A9243D}.Release|x86.Build.0 = Release|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Debug|x64.ActiveCfg = Debug|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Debug|x64.Build.0 = Debug|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Debug|x86.ActiveCfg = Debug|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Debug|x86.Build.0 = Debug|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Release|Any CPU.Build.0 = Release|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Release|x64.ActiveCfg = Release|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Release|x64.Build.0 = Release|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Release|x86.ActiveCfg = Release|Any CPU
+		{0338BD6C-A1F0-4899-AB6F-5245EA0533D6}.Release|x86.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE

Shuttle.Api/Jobs/GameArchiveJob.cs

@@ -0,0 +1,40 @@
+using Quartz;
+using Shuttle.GameArchive;
+using Microsoft.Extensions.Options;
+
+namespace Shuttle.Api.Jobs;
+
+public sealed class GameArchiveJob(
+    IServiceProvider services,
+    IOptions<GameArchiveOptions> options,
+    ILogger<GameArchiveJob> logger) : ISelfRegisteringJob {
+    public static readonly JobKey JobKey = new(nameof(GameArchiveJob), "archive");
+    public static readonly TriggerKey TriggerKey = new(nameof(GameArchiveJob) + "Trigger", "archive");
+
+    public async Task Execute(IJobExecutionContext context) {
+        if (!options.Value.Enabled) {
+            logger.LogInformation("Skipping disabled SHL game archive sync");
+            return;
+        }
+        logger.LogInformation("Starting SHL game archive sync");
+        try {
+            using var scope = services.CreateScope();
+            var synchronizer = scope.ServiceProvider.GetRequiredService<GameArchiveSynchronizer>();
+            await synchronizer.SynchronizeAsync(context.CancellationToken);
+            logger.LogInformation("Finished SHL game archive sync");
+        } catch (Exception ex) when (ex is not OperationCanceledException) {
+            logger.LogError(ex, "SHL game archive sync failed");
+            throw new JobExecutionException(ex, refireImmediately: false);
+        }
+    }
+
+    public static IServiceCollectionQuartzConfigurator RegisterJob(IServiceCollectionQuartzConfigurator qc) {
+        qc.ScheduleJob<GameArchiveJob>(
+            trigger => trigger.WithIdentity(TriggerKey)
+                .ForJob(JobKey)
+                .WithSimpleSchedule(schedule => schedule.WithIntervalInHours(6).RepeatForever())
+                .WithDescription("Mirror public SHL game files to the archive repository"),
+            job => job.WithIdentity(JobKey).StoreDurably().DisallowConcurrentExecution());
+        return qc;
+    }
+}

Shuttle.Api/Program.cs

@@ -10,6 +10,7 @@
 using Shuttle.Api.Services.Scouting;
 using Shuttle.Api.Services.Users;
 using Shuttle.EFCore;
+using Shuttle.GameArchive;
 using Shuttle.ServiceDefaults;
 using Shuttle.Shl.Api.Client;
 
@@ -60,6 +61,7 @@
 builder.Services.AddShlApiClients();
 
 builder.AddShuttleDatabase();
+builder.Services.AddGameArchive(builder.Configuration, builder.Environment);
 builder.AddQuartz();
 
 builder.Services.AddUserService();

Shuttle.Api/Shuttle.Api.csproj

@@ -27,7 +27,6 @@
         <PackageReference Include="OpenTelemetry.Instrumentation.Quartz" />
         <PackageReference Include="OpenTelemetry.Extensions.Hosting" />
 
-        <PackageReference Include="Azure.Identity" />
         <PackageReference Include="Microsoft.EntityFrameworkCore" />
         <PackageReference Include="Microsoft.Extensions.Caching.Hybrid" />
         <PackageReference Include="Microsoft.EntityFrameworkCore.Design" />
@@ -46,6 +45,7 @@
 
     <ItemGroup>
         <ProjectReference Include="..\Shuttle.EFCore\Shuttle.EFCore.csproj" />
+        <ProjectReference Include="..\Shuttle.GameArchive\Shuttle.GameArchive.csproj" />
         <ProjectReference Include="..\Shuttle.Models\Shuttle.Models.csproj" />
         <ProjectReference Include="..\Shuttle.Shl.Api.Client\Shuttle.Shl.Api.Client.csproj" />
         <ProjectReference Include="..\Shuttle.ServiceDefaults\Shuttle.ServiceDefaults.csproj" />

Shuttle.Api/Startup.cs

@@ -60,6 +60,7 @@ public static void AddQuartz(this WebApplicationBuilder builder) {
             });
             HelloJob.RegisterJob(q);
             DbUpdateJob.RegisterJob(q);
+            GameArchiveJob.RegisterJob(q);
         });
         builder.Services.AddQuartzServer(o => {
                 o.AwaitApplicationStarted = true;