Skip to content

fix(store): keep one copy of each node per branch - #1512

Open
BeArchiTek wants to merge 18 commits into
stablefrom
ai-bug-pipeline-node-store-duplicate-copies
Open

BeArchiTek wants to merge 18 commits into
stablefrom
ai-bug-pipeline-node-store-duplicate-copies

Conversation

@BeArchiTek

@BeArchiTek BeArchiTek commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Why

A process that reuses one client, such as a worker or a scheduled sync, used more memory on every query. Each time a query returned a node that was already in the client store, the store added another copy and kept the previous one. Lookups returned only the newest copy, and no method freed the earlier ones. client.store.count() counted the copies, not the nodes.

Goal: the store keeps one copy of each node per branch, so its memory use stays bounded by the number of distinct nodes.

Non-goals:

Closes #1506

What changed

Behavioural changes:

  • When the store receives a node whose id it already holds on that branch, it replaces the previous copy and removes that copy's ID, HFID and key entries. Keys set with store.set(node, key=...) move to the new copy.
  • When the same node object is stored again after its HFID changed, for example after a rename and save(), the previous HFID no longer finds it.
  • client.all() and client.filters() store a related node, for example from prefetch_relationships, include or only, through a private store method. A related node replaces a copy that was also stored as a related node, so peers stay current. It does not replace a node returned by a query for its own kind or stored by save(), because a related node built from relationship data can hold fewer fields and would otherwise replace that node for lookups by HFID and key. The check and the write happen under the branch lock.
  • A node stored before it had an id, then stored again after it got one, keeps one entry, and its previous HFID entries are removed.
  • client.store.count() returns the number of distinct nodes.
  • When the new copy has no HFID, for example from a query with only that did not request it, a lookup by the earlier HFID returns the new copy. Before, it returned the earlier copy. The store guide described the earlier behaviour and is updated.
  • A lookup on a branch the store does not hold returns "not found" without adding an empty entry for that branch.

Implementation notes:

  • The store indexed nodes by _internal_id, which the SDK generates again for every node object, so two objects for the same Infrahub node were stored under two keys.
  • The decision about which copy to keep is in one method, NodeStoreBranch._resolve_existing(). It returns the new copy today. Merging, as feature: Update existing objects within the SDK store #347 asks, would mean changing only that method.
  • _index() and _unindex() add and remove a node's entries. The store records the keys and HFID entries it wrote for each node and removes exactly those, because a node's HFID can change after it is stored, for example after a local rename. _unindex() only removes entries that still point to the copy being removed.
  • A lookup by HFID whose entry points to a node no longer in the store raises NodeNotFoundError, or returns None with raise_when_missing=False.
  • Each branch store has a re-entrant lock around set(), get() and count(), because the sync client's batch saves nodes from several threads and set() now reads and changes several dictionaries in turn. It is re-entrant because computing an HFID inside set() can read the same store again. There is no test for the lock: a threaded test passed with and without it.
  • No new public methods.

What stayed the same:

  • Lookups by id and HFID return the copy from the latest query, except in the HFID case above. A lookup by key now does too; before, a key kept pointing to the copy it was set with.
  • Nodes without an id, such as nodes not yet saved, are stored and found as before.
  • The same id on two branches is still two entries.

How to review

  1. infrahub_sdk/client.py (the two places that store related nodes) and infrahub_sdk/store.py: NodeStoreBranch.set(), _set_related(), _store(), _resolve_existing(), _index(), _unindex(), _get_by_hfid(), and NodeStoreBase._get() / _set() / _set_related().
  2. Tests:
    • tests/unit/sdk/test_client.py::test_method_all_twice_keeps_one_store_entry_per_node fetches the same node twice through client.all(). It checks the store holds one entry with the values from the second response, and does not check which object the store keeps, so it remains valid if the copies are merged later. test_method_all_with_prefetch_keeps_stored_node_over_peer_copy checks that a related node from prefetch_relationships does not replace a node already stored. test_store.py::test_node_store_set_related_replaces_only_a_related_copy checks that a related copy replaces an earlier related copy but not a node stored with set().
    • tests/unit/sdk/test_store.py covers replacement (keys, old and new HFID), one and two new copies without HFID, a local rename before a new query, a rename of the same object stored again, and the unknown-branch lookup.
  3. docs/docs/python-sdk/guides/store.mdx and changelog/1506.fixed.md.

How to test

uv run pytest tests/unit/sdk/test_store.py tests/unit/sdk/test_store_branch.py tests/unit/sdk/test_client.py
uv run invoke lint-code

Results on this branch:

  • Each new test fails on v1.24.0 and passes with the fix.
  • invoke lint-code passes.
  • Unit suite: 3147 passed, 8 failed. The 8 are in tests/unit/pytest_plugin/test_plugin.py and fail the same way on v1.24.0: on the machine used, the infrahub-testcontainers pytest plugin raises RuntimeError: invalid CPU frequency data at start-up.

Impact & rollout

  • Backward compatibility: no API change. Code that relied on client.store.count() counting copies, or on finding an earlier copy by its _internal_id, sees the new behaviour.
  • Infrahub: Infrahub calls the client store. Its CI has not yet been run against this branch.

Checklist

  • Tests added/updated
  • Changelog entry added (changelog/1506.fixed.md)
  • External docs updated (docs/docs/python-sdk/guides/store.mdx)
  • Internal .md docs updated: not needed, no file under dev/ or AGENTS.md describes the store
  • Infrahub CI run against this branch

🤖 Generated with Claude Code

Benoit Kohler and others added 11 commits November 10, 2026 13:56
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NodeStoreBranch indexed nodes by their per-object _internal_id, so each
re-fetch of the same node added a new copy and left the previous one
unreachable but still referenced. The store now replaces the previous
copy when it receives a node whose id it already holds, through a single
_resolve_existing() decision point, and removes the old copy's id, HFID
and key entries.

node.delete() now removes the node from the client store, and a lookup
on a branch the store has never seen no longer creates an empty branch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node fetched with `only` and without its HFID fields replaced the
stored copy and removed its HFID entry, so a lookup by HFID that used to
return the earlier copy found nothing. The replacing copy now takes over
the previous HFID entries when it has no HFID of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing a copy recomputed its HFID, which can change after the node is
indexed (local rename, renamed peer), and an HFID carried to a copy
without one was never removed. Both left HFID entries pointing to a node
no longer in the store, and a lookup by that HFID raised KeyError.

The store now records the keys and HFID entries written for each node
and removes exactly those, a lookup by HFID raises NodeNotFoundError if
its entry points to a removed node, and a removal no longer scans every
key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing a deleted node from the store changes what RelatedNode.peer and
peer-based HFIDs return after a delete, and does not cover nodes deleted
by a cascade. It is not needed to stop the duplicate copies, so it moves
to its own change. The replace decision point now says it is to be
reworked to merge copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sync batch saves nodes from several threads, and set() now reads
and changes several dictionaries in turn. A re-entrant lock per branch
store keeps them consistent; computing an HFID inside set() can read the
same store again. Creating a branch store uses setdefault so two threads
cannot each create one.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BeArchiTek BeArchiTek added the type/bug Something isn't working as expected label Nov 10, 2026
@github-actions github-actions Bot added the type/documentation Improvements or additions to documentation label Nov 10, 2026
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Deploying infrahub-sdk-python with  Cloudflare Pages  Cloudflare Pages

Latest commit: b31f5f4
Status: ✅  Deploy successful!
Preview URL: https://e9faf1db.infrahub-sdk-python.pages.dev
Branch Preview URL: https://ai-bug-pipeline-node-store-d.infrahub-sdk-python.pages.dev

View logs

@codecov

codecov Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 96.72131% with 2 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
infrahub_sdk/store.py 96.72% 1 Missing and 1 partial ⚠️
@@            Coverage Diff             @@
##           stable    #1512      +/-   ##
==========================================
+ Coverage   87.85%   87.90%   +0.05%     
==========================================
  Files         157      157              
  Lines       15656    15703      +47     
  Branches     2114     2117       +3     
==========================================
+ Hits        13754    13804      +50     
+ Misses       1341     1337       -4     
- Partials      561      562       +1     
Flag Coverage Δ
integration-tests 43.04% <73.77%> (+0.08%) ⬆️
python-3.10 63.10% <86.88%> (+0.18%) ⬆️
python-3.11 63.08% <86.88%> (+0.17%) ⬆️
python-3.12 63.10% <86.88%> (+0.18%) ⬆️
python-3.13 63.10% <86.88%> (+0.18%) ⬆️
python-3.14 63.09% <86.88%> (+0.17%) ⬆️
python-3.15 63.09% <86.88%> (+0.17%) ⬆️
python-filler-3.12 23.01% <9.83%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
infrahub_sdk/store.py 83.40% <96.72%> (+2.85%) ⬆️

... and 2 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@BeArchiTek
BeArchiTek marked this pull request as ready for review November 10, 2026 15:04
@BeArchiTek
BeArchiTek requested a review from a team as a code owner November 10, 2026 15:04

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread infrahub_sdk/store.py Resolved
Comment thread infrahub_sdk/store.py Resolved
Comment thread changelog/1506.fixed.md Resolved
Benoit Kohler and others added 5 commits November 10, 2026 22:44
A node renamed and stored again, for example by save(), stayed reachable
by its previous HFID because set() only unindexed a different copy.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
set() removes the previous copy before it adds the new one, so a count()
from another thread between the two steps returned one node less.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he HFID first

A node stored without an id and stored again after it got one kept its
previous HFID entries. Computing the HFID before removing any entry keeps
the store unchanged when that computation raises.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A related node built from relationship data can hold fewer fields than a
node already stored. With one copy per id, it replaced that node for
lookups by HFID and key. client.all() and client.filters() now store a
related node only when the store holds no node with that id.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 4 files (changes from recent commits).

Reply with feedback, questions, or to request a fix.

View guided diff | Turn on auto-fix | Re-trigger cubic

Comment thread docs/docs/python-sdk/guides/store.mdx Resolved
Comment thread infrahub_sdk/client.py Resolved
Comment thread infrahub_sdk/client.py Resolved
A related copy was stored only when no node with its id was stored, so a
later query could never refresh a peer, and the check and the write were
two separate steps. The store now records which nodes came from
relationship data and decides under the branch lock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@polmichel polmichel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If I read this correctly, the rule for which copy the store keeps is:

  • v1.24.0: a later copy never replaces an earlier one. Both stay in _objs, and the later copy takes over only the entries it writes.
  • This PR: a later copy replaces the earlier one, except when the newcomer is a related node. Then the newcomer is dropped.

So which copy you read depends on the order they arrive in:

First stored Then arrives v1.24.0 keeps This PR keeps
full node from a query thin peer, id only thin peer ✗ full node ✓
thin node from save() full peer from prefetch_relationships full peer ✓ thin node ✗

The second row is what I am unsure about for a minor release: after save(), RelatedNode.peer stops returning the copy the server just sent and returns the object built locally, whose other attributes are not loaded. I ran it against a local Infrahub and it reproduces: the store returns None for an attribute the server had just been given a value for.

One more thing, maybe out of scope here: InfrahubOperation.process_nodes, which transforms.py and generator.py call, stores peers with set rather than _set_related. On the same tag, client.all(prefetch_relationships=True) returns the stale local copy and the transform path returns the server's one. Same data, two rules depending on the entry point.

Lookups also change, independently of that rule:

Lookup v1.24.0 This PR
by id latest copy latest copy
by key the copy the key was set with latest copy
by human-friendly ID latest copy, or the earlier one when the newer has no HFID latest copy
earlier copy in _objs kept, unreachable once every entry is rewritten removed

Could we keep the released rule for which copy wins, and remove only the leak? Or have I missed something, and this is a deliberate trade?

save() stores the object built locally, so the guard against related
copies kept it over the copy the server returned with
prefetch_relationships, and process_nodes stored peers with set() anyway.
The latest copy now wins again, as in earlier releases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@BeArchiTek

Copy link
Copy Markdown
Contributor Author

@polmichel You read it correctly, and it was not a deliberate trade.

The guard from dd830da and dc7ff34 used where a copy came from to guess which copy holds more fields: a related node could replace only an earlier related copy. That guess is wrong for save(), which stores the object built locally, so the copy the server returned with prefetch_relationships was dropped, as you reproduced.

Fixed in b31f5f4:

  • client.all() and client.filters() store related nodes with set() again, so the latest copy wins as in v1.24.0. process_nodes already used set(), so both entry points now follow the same rule.
  • The store still keeps one copy per id, which is the memory fix this PR is for.
  • test_method_all_with_prefetch_replaces_saved_node_with_peer_copy covers your second row: a node stored without type, then client.all(prefetch_relationships=True) returning the peer with type. It fails on dc7ff34 with None == 'SITE' and passes now.
  • The guide and the changelog no longer describe the guard.

Your first row (a full node replaced by an id-only peer) goes back to the v1.24.0 behaviour. Keeping fields from both copies is the merge that _resolve_existing() describes as later work. It is not in this PR.

On lookups, these differences remain, because the earlier copy is no longer kept:

  • By key: the key now returns the latest copy, not the copy it was set with.
  • By human-friendly ID: when the new copy has no HFID, the earlier HFID still finds the node, and it returns the new copy.
  • The earlier copy is removed from _objs instead of staying there unreachable.

@polmichel polmichel left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First feedback

This PR adds another behavior which would deserve its own changelog

"A lookup by key or HFID now returns the latest copy of a node, even when that copy was stored without the key or has no HFID. Before, a key returned the object stored with that key, and an HFID returned the last copy that had one. A copy with fewer fields, such as a related node returned with prefetch_relationships or include, or a group member that a tracking session with delete_unused_nodes=True reads back when it updates its group, therefore replaces the object you stored under a key, and attribute values that only the earlier object held are no longer returned."

Second feedback

This code exhibits a failing test which passes on stable branch and can be problematic in my opinion as nodes from generator could get back to the user without being complete.

class PersonFromStoreTransform(InfrahubTransform):
    query = "person_with_cars"

    async def transform(self, data: dict) -> dict:
        person = self.store.get(key="John", kind="TestingPerson")
        return {"name": person._get_attribute("name").value, "age": person._get_attribute("age").value}

@pytest.fixture
def person_with_cars_schema() -> dict[str, Any]:
    person = NodeSchema(
        name="Person",
        namespace="Testing",
        human_friendly_id=["name__value"],
        attributes=[
            {"name": "name", "kind": "Text", "unique": True},
            {"name": "age", "kind": "Number", "optional": True},
        ],
        relationships=[{"name": "cars", "peer": "TestingCar", "cardinality": "many", "optional": True}],
    ).convert_api()
    car = NodeSchema(
        name="Car",
        namespace="Testing",
        human_friendly_id=["name__value"],
        attributes=[{"name": "name", "kind": "Text", "unique": True}],
        relationships=[{"name": "owner", "peer": "TestingPerson", "cardinality": "one", "optional": True}],
    ).convert_api()
    return {"version": "1.0", "nodes": [person.model_dump(), car.model_dump()]}

async def test_transform_store_returns_main_node_by_hfid_when_its_peer_references_it_back(
    clients: BothClients, person_with_cars_schema: dict[str, Any]
) -> None:
    """In one response, the car's owner is an id-only copy of the person: it must not hide the person's values."""
    transform = PersonFromStoreTransform(
        client=clients.standard, infrahub_node=InfrahubNode, convert_query_response=True, branch="main"
    )
    transform.client.schema.set_cache(person_with_cars_schema, branch="main")
    data = {
        "TestingPerson": {
            "edges": [
                {
                    "node": {
                        "id": PERSON_ID,
                        "__typename": "TestingPerson",
                        "name": {"value": "John"},
                        "age": {"value": 42},
                        "cars": {
                            "edges": [
                                {
                                    "node": {
                                        "id": CAR_ID,
                                        "__typename": "TestingCar",
                                        "name": {"value": "Volt"},
                                        "owner": {"node": {"id": PERSON_ID, "__typename": "TestingPerson"}},
                                    }
                                }
                            ]
                        },
                    }
                }
            ]
        }
    }

    assert await transform.run(data=data) == {"name": "John", "age": 42} ## Age and name are None

A fix could be to add this code inside infrahub_sdk/operation.py, line 67->72

 # A related copy of a node that the response also returns as a main node holds fewer fields: keep the main node.
  main_ids = {node.id for node in self._nodes if node.id}
  related_nodes = [node for node in self._related_nodes if node.id not in main_ids]
  for node in self._nodes + related_nodes:
      if node.id:
          self._init_client.store.set(node=node)

Comment thread changelog/1506.fixed.md
@@ -0,0 +1 @@
The client store now keeps one entry per node and branch. Before, each query that returned a node already in the store, such as a repeated `client.get()`, `client.all()` or `client.filters()`, added another copy and kept the previous one in memory, so a long-running process that reused one client used more memory on every query. A lookup by ID, HFID or key returns the copy from the latest query, and `client.store.count()` now returns the number of distinct nodes. When the new copy has no HFID, for example from a query with `only` that did not request it, a lookup by the earlier HFID now returns the new copy instead of the earlier one. This includes a related node returned with a query, for example with `prefetch_relationships`, `include` or `only`: it replaces the stored copy, also one stored by `save()`, as in earlier releases. Looking up a node on a branch the store does not hold no longer adds an empty entry for that branch.

@polmichel polmichel Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find this changelog quite long, and it includes wording from intermediate steps, such as “as in earlier releases”. Would you agree that we should shorten it?

Just a suggestion made using Claude
The client store now keeps one copy of each node per branch. Before, each query that returned a node already in the store, such as a repeated client.get(), client.all() or client.filters(), added another copy and kept the previous one in memory, so a long-running process that reused one client used more memory on every query. client.store.count() now returns the number of distinct nodes rather than the number of copies.

The lookup changes for keys and HFIDs would go in a separate changelog/1506.changed.md, as suggested in my review.

</TabItem>
</Tabs>

- The resulting related objects for objects retrieved using the SDK client's query methods, when we use the `prefetch_relationships` argument.

@polmichel polmichel Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a double space after the "when we use".
Then, we can modify the end of the sentence to add the other cases
"prefetch_relationships or include argument."

That way, we can remove the additional paragraph line 140 #1512 (comment)

A node returned by a [query with `only`](../../python-sdk/guides/query_data#fetching-only-the-named-attributes-and-relationships), or a peer returned with it, replaces the copy stored under the same `id`, even one that contains more fields, so check `is_loaded` on a field of a node from the store before you read it. A lookup by human-friendly ID returns the earlier copy when the new node has no human-friendly ID, for example when the query named neither `hfid` nor the fields that make up the human-friendly ID.
The store keeps one copy of each node per branch. A node returned by a later query replaces the copy stored under the same `id`, and a lookup by `id`, human-friendly ID or key returns the copy from the latest query. This includes a node returned by a [query with `only`](../../python-sdk/guides/query_data#fetching-only-the-named-attributes-and-relationships), even when the earlier copy contains more fields, so check `is_loaded` on a field of a node from the store before you read it. When the new copy has no human-friendly ID, for example when the query named neither `hfid` nor the fields that make up the human-friendly ID, a lookup by the earlier human-friendly ID returns the new copy.

A related node returned with a query, for example with `prefetch_relationships`, `include` or `only`, follows the same rule: it replaces the copy stored under the same `id`, also one stored by `save()` or returned by a query for its own kind.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one is repetitive and probably due to the back-and-forth, we can remove it and add my last modification to capture these changes

Comment thread infrahub_sdk/store.py
self._keys_by_internal_id: dict[str, set[str]] = {}
self._hfids_by_internal_id: dict[str, set[tuple[str, tuple]]] = {}
# The sync batch saves nodes from several threads. Re-entrant because computing an HFID can read this store.
self._lock = threading.RLock()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having this new lock makes me think we should also add unit tests covering this behavior. I can see two options here:

  1. either keep lock as an internal attribute of this class and mock it with monkeypatch, to test the behavior and check that the store acquires the lock when necessary
  2. or inject lock into this class as a dependency, so we can replace it with a fake object in unit tests

The second option is cleaner, but I guess it’s harder to set up. So the first option is fine, we'll set up properly this dependency injection later.

Internal thoughts for future improvements:

  • Write access versus read access?
  • Do a similar InfrahubLock object in the SDK?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type/bug Something isn't working as expected type/documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: NodeStore keeps every copy of a node fetched more than once, so its memory use grows without limit

2 participants