Skip to content

Improve performance with large CodeSystems - #1637

Merged
cmoesel merged 10 commits into
FHIR:masterfrom
vadi2:large-codesystem-performance
Sep 30, 2026
Merged

cmoesel merged 10 commits into
FHIR:masterfrom
vadi2:large-codesystem-performance

Conversation

@vadi2

@vadi2 vadi2 commented Sep 2, 2026 •

Copy link
Copy Markdown
Contributor

Description:

SUSHI slows down sharply as a CodeSystem grows. Profiling a 9,347-concept CodeSystem (about 56,000 code caret rules for designations and properties) showed two costs that grow with the number of rules, so this PR fixes both:

  1. Rebuilding the StructureDefinition for every caret rule. setPropertyOnDefinitionInstance called instance.getOwnStructureDefinition(fisher) (a package lookup plus StructureDefinition.fromJSON) on each call. For a CodeSystem that is once per code caret rule, and it was 83% of the build. The function now takes the StructureDefinition as an optional final parameter (default unchanged), and CodeSystemExporter and ValueSetExporter pass in the one they already built. Sharing one StructureDefinition across an entity's rules has precedent: the validation pass in these exporters already did it, and InstanceExporter reuses cached StructureDefinitions across all of an instance's rules. In the same code path, findConceptPath and setConcepts index concepts by code instead of scanning the concept list for every rule, and setPropertyOnInstance fills a large array from its current end instead of from index 0 (the old loop re-read the array length each iteration, and each iteration pushed one element, so starting at the current length adds exactly the same elements).

  2. Scanning every rule to find a definition's id, url, name, or version. findAssignmentByPath did a findLast over all rules, and FSHTank.fish does that for every entity it checks on every fish. In an IG with several large CodeSystems, every lookup of a Reference or code system walked tens of thousands of rules, which was 63% of what remained after fix 1. The found rule is now cached per rules array. Rules arrays are only appended to or replaced wholesale (as applyInsertRules does), so the cache is invalidated when the array is replaced, when its length changes, or when its last rule changes. Because the rule rather than its value is cached, changes to a rule's value are still seen, and a cached rule is re-checked against the lookup on each hit because CodeSystemExporter rewrites a code caret rule's path in place once it resolves the concept.

Measurements (Node 22, sushi on the same machine):

Build master fix 1 fix 1 + 2
One CodeSystem with 9,347 concepts, FSHOnly 147 s, 750 MB 15 s, 594 MB 8 s, 598 MB
IG with 32 large CodeSystems written in FSH not measured (many minutes) 142 s 52 s

For comparison, the same IG with those 32 CodeSystems pre-rendered to JSON in input/ (the workaround the project uses today) builds in 31 s. Most of the remaining gap is ANTLR parsing of the large FSH files.

Output is unchanged: the generated JSON for the 9,347-concept CodeSystem is byte-identical to master, and every other resource in the IG is identical to the pre-rendered build.

Developed with AI assistance (Claude Code). The diff, tests, and measurements were reviewed before submitting.

Testing Instructions:

Run npm run check. New tests in test/fshtypes/FshCodeSystem.test.ts, test/fshtypes/Instance.test.ts and test/import/FSHTank.test.ts cover the cache being invalidated when rules are added, removed, replaced, their values change, or a code caret rule's path is resolved. New tests in test/export/CodeSystemExporter.test.ts and test/export/ValueSetExporter.test.ts pin the output when several caret rules target the same extension slice and a numerically indexed extension, since the shared StructureDefinition now carries the slices added by earlier rules.

To see the performance difference, run sushi on a project containing a CodeSystem with a few thousand concepts that each have a designation or property.

Related Issue:

None.

https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97

setPropertyOnDefinitionInstance rebuilt the instance's StructureDefinition
for every rule. A CodeSystem with thousands of concepts has tens of
thousands of code caret rules, so that dominated the build. The exporters
now pass in the StructureDefinition they already built, concepts are
indexed by code instead of scanned, and filling a large array no longer
restarts from index 0.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
findAssignmentByPath scanned every rule on a definition, and the tank does
that for every entity on every fish. Cache the result per rules array,
invalidating when rules are added or removed or the array is replaced.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
CodeSystemExporter rewrites a code caret rule's path in place, so a cached
rule may stop matching its lookup. Re-check it on a hit and rescan if it no
longer matches. Also clarify comments and add tests for the Instance branch
and for reusing the StructureDefinition across extension caret rules.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97
Rename the cache for what it holds, return early on a cache hit, keep the
original conceptIndices local in findConceptPath, and flatten the array
fill branches. No behavior change.

Claude-Session: https://claude.ai/code/session_01AKNbxASp6vABedSLJMGW97

@cmoesel cmoesel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the contribution, @vadi2. Caching adds more complication, but in this case it seems there is a very good payoff. We'll take it!

I've left one comment for you to consider. Let me know what you think.

This also needs to be rebased or merged from master.

Thanks again!

Comment thread src/export/CodeSystemExporter.ts Outdated
@vadi2

vadi2 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @cmoesel! Merged master, and setConcepts now builds the index once and findConceptPath reuses it - simpler, same speed (~4.3 s vs 82 s on master). One behaviour difference from master to flag: mixing a numeric index with two aliases of the same extension now gives two well-formed extensions instead of master's malformed three, because the shared StructureDefinition keeps the slice from validation.

Extension: MyExt
Id: my-ext
* value[x] only string

ValueSet: MixedVS
* ^extension[0].url = "http://example.org/numeric"
* ^extension[0].valueString = "n"
* ^extension[MyExt][0].valueString = "one"
* ^extension[my-ext][1].valueString = "two"

master:

[{"url": "http://example.org/numeric", "valueString": "one"},
 {"url": "http://example.org/StructureDefinition/my-ext"},
 {"valueString": "two"}]

this branch:

[{"url": "http://example.org/numeric", "valueString": "one"},
 {"url": "http://example.org/StructureDefinition/my-ext", "valueString": "two"}]

@vadi2

vadi2 commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

@cmoesel friendly ping

@cmoesel

cmoesel commented Sep 29, 2026

Copy link
Copy Markdown
Member

Thanks for the friendly ping, @vadi2. I started to re-review and then was hit with a bunch of distractions. I'll try to finish up the review in the next few days.

@cmoesel cmoesel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again, @vadi2. The updates look good and I've confirmed that this does not introduce any regressions. I also saw improved performance for some IGs in regression as well. Nicely done!

@cmoesel
cmoesel merged commit 2a6ef3d into FHIR:master Sep 30, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants