Add version-scoped dependency resolution for multi-version IGs

[GinoCanessa]

Summary

SUSHI treats every loaded external dependency as one flat canonical namespace. That
breaks R5-authored IGs that use the IG Publisher's multi-version package model
to generate R4 and R4B packages from a single source, because such an IG may
declare different dependency packages for different generated FHIR versions, and
those packages may legitimately publish the same canonical URL with
version-specific content. With a flat namespace one package wins globally by load
order, so the other version's artifacts are validated against the wrong definition.

This branch teaches SUSHI to read the multi-version parameters the IG already
carries — fhirVersion, generate-version, per-version dependency extensions, and
the r4-inclusion / r4b-inclusion / r5-inclusion membership lists — and to use
them to rank dependency candidates during fishing.

There is no author-facing FSH change. No new spelling on any canonical, alias,
or InstanceOf:, and no overload of | (which already means artifact-version
pinning). An IG that declares no version-scoped dependency takes an unchanged code
path.

Addresses Issue #1510

Using this feature

This applies to IGs that are authored in one FHIR version and also generate packages for other FHIR versions (for example, an R5 IG that also publishes R4 and R4B packages), when those versions need different dependency packages. There is no new FSH syntax and no new SUSHI option. SUSHI reads the multi-version settings the IG Publisher already uses, in sushi-config.yaml:

• Target versions: fhirVersion, plus a generate-version parameter for each additional version (e.g., r4, r4b).
• Per-version dependencies: the http://hl7.org/fhir/tools/StructureDefinition/ig-dependency-for-version extension on a dependency names the package id and version to use for a target version, or removes the dependency from that version (use: remove). A dependency with these extensions only applies to the versions it names.
• Per-version artifacts: the r4-inclusion, r4b-inclusion, and r5-inclusion parameters list the artifacts (Type/id or canonical URL) that belong to each version. Unlisted artifacts belong to every version.

SUSHI only changes how it resolves definitions when at least one dependency uses ig-dependency-for-version. When it does, SUSHI resolves each artifact's references using the packages selected for the FHIR versions that artifact belongs to. Packages selected only for other versions are used only as a last resort. Otherwise, SUSHI behaves exactly as before.

• IG Publisher documentation: Multi-version IGs
• Example: the FHIR Subscriptions R5 Backport IG's sushi-config.yaml (R5 source generating R4 and R4B packages, on the IG's tickets branch)

Trimmed excerpt from that configuration:

fhirVersion:
  - 5.0.0
parameters:
  generate-version:
    - r4
    - r4b
  r4-inclusion:
    - StructureDefinition/backport-subscription-notification-r4
  r4b-inclusion:
    - StructureDefinition/backport-subscription-notification
dependencies:
  hl7.fhir.uv.xver-r5.r4:
    id: hl7.fhir.uv.xver-r5.r4
    version: 0.1.0
    extension:
      - url: http://hl7.org/fhir/tools/StructureDefinition/ig-dependency-for-version
        extension:
          - url: packageId
            valueString: hl7.fhir.uv.xver-r5.r4
          - url: version
            valueString: 0.1.0
          - url: fhirVersion
            valueString: 4.0.1

Motivation

The concrete case is the FHIR Subscriptions R5 Backport IG
(hl7.fhir.uv.subscriptions-backport), which is authored at fhirVersion: 5.0.0
and generates R4 and R4B packages.

hl7.fhir.uv.xver-r5.r4 and hl7.fhir.uv.xver-r5.r4b both publish

http://hl7.org/fhir/5.0/StructureDefinition/profile-SubscriptionStatus

and the two definitions are structurally different: in the R4 package it is
type: Basic (R4 has no SubscriptionStatus resource), and in the R4B package it
is type: SubscriptionStatus (R4B does). The two packages share 1,880
same-named artifacts, 11 of which differ in type in exactly this
Basic-vs-native way.

Today SUSHI binds that URL once. Because DEFAULT_SORT's byLoadOrder(false)
means last-loaded wins, and the R4B xver package is declared last, the R4 Basic
shape the IG actually authors is validated against the R4B SubscriptionStatus
definition — and every R4 status instance fails.

This is a SUSHI capability gap rather than an IG authoring problem: the
disambiguating inputs are already in sushi-config.yaml, and SUSHI reads none of
them. The rules implemented here are ported from the Publisher's own
multi-version-IGs.md §3–§4 and its isDepApplicableForVersion /
isIncludedInVersion logic, so SUSHI and the Publisher agree on membership.

Design

Three ideas carry the whole change:

1. A pure model. src/ig/VersionScopes.ts is a self-contained, logger-free
   statement of the Publisher's rules. It is constructed during dependency loading
   and is fully unit-testable without fixtures.
2. An ambient frame stack. FHIRDefinitions.inVersionScopeOf(key, fn) pushes
   the target versions for one artifact, runs fn, and pops in a finally.
   Exporters wrap their per-artifact entry points; no exporter learns anything
   about packages, and public Fishable signatures are unchanged.
3. Demote, never delete. Scoped fishing ranks candidates by
   (typeRank, bandRank) instead of filtering them. An out-of-version candidate is
   still reachable, so a reference that resolves today cannot become newly
   unresolvable just because scope is active. typeRank is compared before
   bandRank, so existing type precedence is untouched.

lodash.sortBy is stable, so every tie preserves today's FPL order.

Changes

Model

• src/ig/VersionScopes.ts (new) — target-version derivation, per-version
  dependency applicability and overrides, artifact membership from the inclusion
  lists, package banding (in-scope / broad / out-of-version), and structured
  configuration diagnostics. Accepts every token in the Publisher's family table
  (r4/4.0/4.0.1, r4b/4.3/4.3.0, r5/5.0/5.0.0) and nothing else,
  so 4.1, 4.2, 3.0, and 6.0 are reported rather than silently
  reinterpreted. Reads parameter.code as either a string or an R5 Coding, so
  an ImplementationGuide resource supplied as input keeps its parameters.

Resolution

• src/fhirdefs/FHIRDefinitions.ts — the frame stack, scoped candidate
  ranking, and one shared scoped-lookup helper used by fishForFHIR,
  fishForMetadata, and fishForMetadatas. The no-scope path early-returns before
  any of it. inVersionScopeOf carries a compile-time SyncOnly<T> constraint:
  the stack is LIFO and pops when fn returns, so an async callback would let a
  synchronous sibling pop the wrong frame. That is now a type error, not a latent
  ranking bug.
• src/utils/MasterFisher.ts — delegates inVersionScopeOf with the same
  constraint.

Loading and reporting

• src/utils/Processing.ts — builds and installs the model from the same
  dependency coordinates that were attempted for loading, and loads the packages
  named by per-version overrides, which are otherwise ranked in-scope but
  absent from the package DB. Override loads are guarded against reordering a
  coordinate that a later top-level dependency (or FHIR core) also declares.
  Emits the model's configuration diagnostics.

Export

• src/export/artifactScopeKeys.ts (new) — builds a scope key from FSH
  metadata alone, never by fishing, since entering the scope is what makes
  fishing correct in the first place. URLs come from the existing
  getUrlFromFshDefinition, so a ^url caret rule wins.
• src/export/{StructureDefinition,Instance,ValueSet,CodeSystem,Mapping}Exporter.ts
  and FHIRExporter.ts — per-artifact scope frames, plus a startup report for
  inclusion entries that match no exported artifact and for ids listed under
  inconsistent type prefixes.

Diagnostics

Unparseable version-scope configuration is now reported instead of silently
widening a dependency to every version. A partly-bad dependency warns; a dependency
whose version extensions all fail to parse is an error and lands
out-of-version everywhere, because that case changes which package wins a
collision and must not ship under a zero exit code.

Testing

npm run check is clean: 106 suites, 3614 tests, tsc && eslint clean,
Prettier clean.

New/extended coverage (~1,340 added test lines):

• test/ig/VersionScopes.test.ts — the Publisher token table and its
  rejections, Coding parameter codes, membership by Type/id / bare id /
  canonical URL, package banding, per-version overrides and remove, the
  non-widening all-failed dependency, patch-wildcard (1.2.x) versions, and the
  authored-coordinate demotion.
• test/fhirdefs/FHIRDefinitions.test.ts — in-version preference, demotion
  below untagged packages, type rank ahead of band, an unlisted artifact belonging
  to every version, an override winning a collision, no re-query after a scoped
  miss, and a @ts-expect-error case proving an async callback is rejected at
  compile time.
• test/utils/Processing.test.ts — override coordinates reaching
  loadPackage, load-order preservation, and the emitted warn/error diagnostics.
• test/export/artifactScopeKeys.test.ts (new) — the canonical default, the
  ^url override, ValueSet/CodeSystem types, an assigned instance URL, and the
  deliberate absence of an invented one.
• test/export/*Exporter.test.ts — frame entry and restoration, including
  restoration after a nested export and after a throw.

Regression

npm run regression (gh:master vs local): 294 repositories processed, 281
compared SAME, 0 DIFF. The 13 without a comparison all aborted on
Request failed with status code 504 (package-registry gateway timeouts), not
output differences.

Caveat: this is a 294/1100 sample, stopped deliberately rather than run to
completion (~9h at the observed rate). No output difference was observed in any
comparison. Happy to run the full pass or npm run regression:last-year if you
would like completion before merge.

How to verify against a real IG

npm ci && npm run build
node dist/app.js <path-to-fhir-subscription-backport-ig>

The backport IG's xver collision produced a large error cluster (~36 direct plus
cascade failures) before this change. With this branch the collision errors are
gone; the errors that remain are IG-side and unrelated to SUSHI
(SearchParameter.xpathUsage assignments that R5 removed, and resources defined
both in input/fsh/ and input/resources/).

Compatibility & risk

• No change for single-version IGs. VersionScopes reports itself
  unconfigured when no dependency declares a version extension, and
  inVersionScopeOf returns fn() before touching the stack. The regression
  sample is the evidence.
• Ranking cannot make a reference unresolvable — candidates are demoted, never
  filtered, so the failure mode is "a wrong answer that was already the answer",
  not a new "not found".
• Cost: two FPL queries per fish instead of one while a frame is active. The
  unscoped path is unchanged and one previously redundant query was removed.
• No new dependencies, no Fishable signature changes, no fork of
  fhir-package-loader, and no change to FISHING_ORDER, DEFAULT_SORT, or the
  unscoped path.
• One behavior change reaches valid, working configurations: a dependency that a
  per-version override replaces is now absent from that version instead of
  remaining broadly available. It is isolated in its own commit (dd8db92) and can
  be reverted alone.

Related

• issue Support for multiple versions of FHIR in a single IG #1510

Builds on the multi-version rules in the IG Publisher's multi-version-IGs.md
(isDepApplicableForVersion, isIncludedInVersion).

[cmoesel]

Wow. What a pull request! Thanks for the sizable contribution, @GinoCanessa. I apologize for the delay, but it took me a while to wrap my head around it (well, actually, it took me a while to find the time to wrap my head around it!).

This seems to work well, and once I followed it through the debugger, it made sense to me. I might have been able to understand it quicker if the comments were a bit more human-friendly; I've asked you to reword some of the more egregious AI-speak.

I think it would also be helpful if you could include a brief description of the overall approach somewhere (maybe at the top of src/ig/VersionScopes.ts?). That description should include a link to the best documentation available (which I assume to be github.com/HL7/fhir-ig-publisher/multi-version-IGs.md). I'm asking for this so that others who land in this codebase can get an overall understanding more quickly (including future me when I forget all of this stuff). Again, brief is OK, especially if you point to some external documentation -- but I think something would be helpful. If you'd rather add an .md than code comments, that's OK too.

See my individual comments for additional specific feedback.

Last, please also update the PR description so that somewhere near the top it includes some guidance on how to use this feature (since release notes will link there). Even if it's just a link to the docs, that would be helpful. A link to the subscription backport IG that uses this might also be helpful.

Assuming you make more changes based on my review, I'll want to run another regression on this. If you get the chance to run one, then I'll trust yours. But if you don't get a chance, I'll just run it over here.

Thanks again for the excellent PR. This is a signficant new feature and I'm glad that SUSHI will support it (and that I didn't have to implement it!).

[cmoesel]

That's a good point -- and a bug. getUrlFromFshDefinition definitely shouldn't be assuming that an instance without a url rule is a StructureDefinition. My (unverified) guess is that when the definition was first defined, it did not accept an Instance as an argument -- and whenever that was added the fallback to StructureDefinition was not adjusted. I've filed #1647 for this bug.

[GinoCanessa]

Perfect - I left getUrlFromFshDefinition alone in this PR and we can come back to it afterwards. This PR is not relying on the fallback, so I assume we are good in the meantime.

[cmoesel]

Consider renaming this to canonicalBase (or something like it) to make it clear that it is the canonical base for the IG, not the canonical of the resource this is scoped to.

[GinoCanessa]

Renamed to igCanonicalBase to be clear it is the IG's base and not the artifact's URL.

[cmoesel]

Maybe I am missing something, but shouldn't you be able to map these all at once using the resourceType in the JSON? E.g.,

Suggested change

	const exportedKeys: ArtifactScopeKey[] = [
	...this.pkg.profiles,
	...this.pkg.extensions,
	...this.pkg.logicals,
	...this.pkg.resources
	].map(sd => ({ resourceType: 'StructureDefinition', id: sd.id, url: sd.url }));
	exportedKeys.push(
	...this.pkg.valueSets.map(vs => ({ resourceType: 'ValueSet', id: vs.id, url: vs.url })),
	...this.pkg.codeSystems.map(cs => ({ resourceType: 'CodeSystem', id: cs.id, url: cs.url })),
	...this.pkg.instances.map(instance => ({
	resourceType: instance.resourceType,
	id: instance.id,
	url: instance.url
	}))
	);
	const exportedKeys: ArtifactScopeKey[] = [
	...this.pkg.profiles,
	...this.pkg.extensions,
	...this.pkg.logicals,
	...this.pkg.resources,
	...this.pkg.valueSets,
	...this.pkg.codeSystems,
	...this.pkg.instances
	].map(r => ({ resourceType: r.resourceType, id: r.id, url: r.url }));

[GinoCanessa]

Well, one of us missed something (hint: it was not you =). Much simpler - added a test that checks the reported type, id, and URL for all seven kinds of artifact.

[cmoesel]

This is totally subjective, but consider avoiding the multiline ternary by just adding the scope key if necessary. E.g.,

Suggested change

	const def = info?.resourcePath
	? this.findResourceJSON(item, {
	type: normalizeTypes(types),
	scope: scopeForInfo(info),
	sort: DEFAULT_SORT
	})
	: this.findResourceJSON(item, {
	type: normalizeTypes(types),
	sort: DEFAULT_SORT
	});
	const options: FindResourceInfoOptions = {
	type: normalizeTypes(types),
	sort: DEFAULT_SORT
	};
	// if there is an active version scope and a match in scope, use that scope
	const info = this.rankedScopedResourceInfos(item, types)?.[0];
	if (info?.resourcePath) {
	options.scope = scopeForInfo(info);
	}
	const def = this.findResourceJSON(item, options);

[GinoCanessa]

Yep, easier to read.

[cmoesel]

Is the phrase "dependency coordinates" commonly used? Or is it AI speak? It feels awkward to me. Is there a better way to phrase it?

[GinoCanessa]

It is AI-speak, but I am 60% bot now so I didn't notice. Since it just meant "package id and version", removed "coordinate" and reworded the comment.

/**
 * Builds the version scopes from the same dependency list used for loading (each entry's package
 * id and requested version), installs them on the FHIR definitions, and logs configuration
 * problems. If no dependency uses the ig-dependency-for-version extension, version scoping stays
 * off and definitions resolve exactly as they do for a single-version IG; otherwise it also logs
 * a startup summary.
 */

[cmoesel]

Again, not the most human-friendly description.

[GinoCanessa]

Updated to describe what is actually checked. Also fixed the duplicate line.

[cmoesel]

How is this different than the prefers a per-version override over the package it replaces test on lines 1650-1664?

[GinoCanessa]

It wasn't - removed and updated the tests to check positive and negative paths.

[cmoesel]

I'm not sure what narrows the frame means, so I don't know what this test is really checking. Especially since it seems to be the same as demotes explicitly out-of-version candidates below untagged packages (1716-1723) and the 2nd half of resolves a collision to the package named by a per-version override (1699-1704) and the second half of prefers a per-version override over the package it replaces (1658-1663).

So, yeah... it seems like we're testing the same thing a lot of times.

[GinoCanessa]

Yep, fair. "Frame" was internal language used during development and the only thing different on this test as the input. Renamed the test for clarity and added comments. To close the loop, this is the merged-winner of the two.

[cmoesel]

This test feels weird to me. I guess it's just saying that if you don't call the inVersionScopeOf function that you can still fish for broad packages?

[GinoCanessa]

Close - it checks that when version scoping is configured but no artifact is being exported, lookups use the normal package order.

Testing to make sure the last-loaded package (example.r4b) wins, not the one that's available to every version (example.broad). It guards against scoping leaking into lookups made outside an export. The old "fast path" name also claimed something the test doesn't measure. Renamed and added comments.

[cmoesel]

I don't think I understand what is meant by No .r4b sibling is invented.

[GinoCanessa]

It meant SUSHI can't assume an .r4b version of a package exists just because the IG targets R4B. Reworded, and renamed the test.

[GinoCanessa]

Thanks for the detailed review (again =). Pushed changes and replied in each thread (I think).

• Overview: There's now a short, plain-language overview in the doc comment on the VersionScopes class in src/ig/VersionScopes.ts. It explains how the configuration, artifact membership, and package ranking fit together, notes that IGs without version-specific dependencies are unaffected, and links the Publisher's multi-version-IGs.md. It also points out that the ranking is SUSHI's own, not something the Publisher defines.
• Comments and tests: I reworded the comments you flagged, plus similar ones you didn't, and removed the ones that described earlier versions of the code. I also merged the duplicate override tests and renamed the ones that were hard to follow.
• Two behavior changes came out of your questions:
  • Automatic dependencies now keep their low/high priority in every target version, including when a dependency is replaced for one version (your question about the extensions package in Processing.ts).
  • Instances whose id is listed under more than one resource type are no longer matched on the id alone. SUSHI works out which entry applies and reports an error when it can't tell (your question about id clashes in VersionScopes.ts).
• PR description: I added a "Using this feature" section at the top, with links to the Publisher docs and to the Subscriptions Backport IG's multi-version configuration.
• Regression: I ran the full regression, but it looks like the source was out of date (e.g., there were changes to ImplementationGuide.name that are unrelated to this PR). I think that was most of the changes, but am not completely sure. Also was hitting TAR_BAD_ARCHIVE on several due to the packages2/FPL issue in our other thread.

npm run check passes at 333b243: 108 suites, 3804 tests passed (8 skipped, 7 todo), and lint and Prettier are clean.