Repository navigation
fix(store): keep one copy of each node per branch - #1512
BeArchiTek wants to merge 18 commits into
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
NodeStoreBranch indexed nodes by their per-object _internal_id, so each re-fetch of the same node added a new copy and left the previous one unreachable but still referenced. The store now replaces the previous copy when it receives a node whose id it already holds, through a single _resolve_existing() decision point, and removes the old copy's id, HFID and key entries. node.delete() now removes the node from the client store, and a lookup on a branch the store has never seen no longer creates an empty branch. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node fetched with `only` and without its HFID fields replaced the stored copy and removed its HFID entry, so a lookup by HFID that used to return the earlier copy found nothing. The replacing copy now takes over the previous HFID entries when it has no HFID of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing a copy recomputed its HFID, which can change after the node is indexed (local rename, renamed peer), and an HFID carried to a copy without one was never removed. Both left HFID entries pointing to a node no longer in the store, and a lookup by that HFID raised KeyError. The store now records the keys and HFID entries written for each node and removes exactly those, a lookup by HFID raises NodeNotFoundError if its entry points to a removed node, and a removal no longer scans every key. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Removing a deleted node from the store changes what RelatedNode.peer and peer-based HFIDs return after a delete, and does not cover nodes deleted by a cascade. It is not needed to stop the duplicate copies, so it moves to its own change. The replace decision point now says it is to be reworked to merge copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sync batch saves nodes from several threads, and set() now reads and changes several dictionaries in turn. A re-entrant lock per branch store keeps them consistent; computing an HFID inside set() can read the same store again. Creating a branch store uses setdefault so two threads cannot each create one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deploying infrahub-sdk-python with
|
| Latest commit: |
b31f5f4
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://e9faf1db.infrahub-sdk-python.pages.dev |
| Branch Preview URL: | https://ai-bug-pipeline-node-store-d.infrahub-sdk-python.pages.dev |
Codecov Report❌ Patch coverage is
@@ Coverage Diff @@
## stable #1512 +/- ##
==========================================
+ Coverage 87.85% 87.90% +0.05%
==========================================
Files 157 157
Lines 15656 15703 +47
Branches 2114 2117 +3
==========================================
+ Hits 13754 13804 +50
+ Misses 1341 1337 -4
- Partials 561 562 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
All reported issues were addressed across 6 files
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
A node renamed and stored again, for example by save(), stayed reachable by its previous HFID because set() only unindexed a different copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
set() removes the previous copy before it adds the new one, so a count() from another thread between the two steps returned one node less. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he HFID first A node stored without an id and stored again after it got one kept its previous HFID entries. Computing the HFID before removing any entry keeps the store unchanged when that computation raises. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A related node built from relationship data can hold fewer fields than a node already stored. With one copy per id, it replaced that node for lookups by HFID and key. client.all() and client.filters() now store a related node only when the store holds no node with that id. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
All reported issues were addressed across 4 files (changes from recent commits).
Reply with feedback, questions, or to request a fix.
View guided diff | Turn on auto-fix | Re-trigger cubic
A related copy was stored only when no node with its id was stored, so a later query could never refresh a peer, and the check and the write were two separate steps. The store now records which nodes came from relationship data and decides under the branch lock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
polmichel
left a comment
There was a problem hiding this comment.
If I read this correctly, the rule for which copy the store keeps is:
v1.24.0: a later copy never replaces an earlier one. Both stay in_objs, and the later copy takes over only the entries it writes.- This PR: a later copy replaces the earlier one, except when the newcomer is a related node. Then the newcomer is dropped.
So which copy you read depends on the order they arrive in:
| First stored | Then arrives | v1.24.0 keeps |
This PR keeps |
|---|---|---|---|
| full node from a query | thin peer, id only |
thin peer ✗ | full node ✓ |
thin node from save() |
full peer from prefetch_relationships |
full peer ✓ | thin node ✗ |
The second row is what I am unsure about for a minor release: after save(), RelatedNode.peer stops returning the copy the server just sent and returns the object built locally, whose other attributes are not loaded. I ran it against a local Infrahub and it reproduces: the store returns None for an attribute the server had just been given a value for.
One more thing, maybe out of scope here: InfrahubOperation.process_nodes, which transforms.py and generator.py call, stores peers with set rather than _set_related. On the same tag, client.all(prefetch_relationships=True) returns the stale local copy and the transform path returns the server's one. Same data, two rules depending on the entry point.
Lookups also change, independently of that rule:
| Lookup | v1.24.0 |
This PR |
|---|---|---|
by id |
latest copy | latest copy |
by key |
the copy the key was set with | latest copy |
| by human-friendly ID | latest copy, or the earlier one when the newer has no HFID | latest copy |
earlier copy in _objs |
kept, unreachable once every entry is rewritten | removed |
Could we keep the released rule for which copy wins, and remove only the leak? Or have I missed something, and this is a deliberate trade?
save() stores the object built locally, so the guard against related copies kept it over the copy the server returned with prefetch_relationships, and process_nodes stored peers with set() anyway. The latest copy now wins again, as in earlier releases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@polmichel You read it correctly, and it was not a deliberate trade. The guard from dd830da and dc7ff34 used where a copy came from to guess which copy holds more fields: a related node could replace only an earlier related copy. That guess is wrong for Fixed in b31f5f4:
Your first row (a full node replaced by an id-only peer) goes back to the v1.24.0 behaviour. Keeping fields from both copies is the merge that On lookups, these differences remain, because the earlier copy is no longer kept:
|
There was a problem hiding this comment.
First feedback
This PR adds another behavior which would deserve its own changelog
"A lookup by key or HFID now returns the latest copy of a node, even when that copy was stored without the key or has no HFID. Before, a key returned the object stored with that key, and an HFID returned the last copy that had one. A copy with fewer fields, such as a related node returned with prefetch_relationships or include, or a group member that a tracking session with delete_unused_nodes=True reads back when it updates its group, therefore replaces the object you stored under a key, and attribute values that only the earlier object held are no longer returned."
Second feedback
This code exhibits a failing test which passes on stable branch and can be problematic in my opinion as nodes from generator could get back to the user without being complete.
class PersonFromStoreTransform(InfrahubTransform):
query = "person_with_cars"
async def transform(self, data: dict) -> dict:
person = self.store.get(key="John", kind="TestingPerson")
return {"name": person._get_attribute("name").value, "age": person._get_attribute("age").value}
@pytest.fixture
def person_with_cars_schema() -> dict[str, Any]:
person = NodeSchema(
name="Person",
namespace="Testing",
human_friendly_id=["name__value"],
attributes=[
{"name": "name", "kind": "Text", "unique": True},
{"name": "age", "kind": "Number", "optional": True},
],
relationships=[{"name": "cars", "peer": "TestingCar", "cardinality": "many", "optional": True}],
).convert_api()
car = NodeSchema(
name="Car",
namespace="Testing",
human_friendly_id=["name__value"],
attributes=[{"name": "name", "kind": "Text", "unique": True}],
relationships=[{"name": "owner", "peer": "TestingPerson", "cardinality": "one", "optional": True}],
).convert_api()
return {"version": "1.0", "nodes": [person.model_dump(), car.model_dump()]}
async def test_transform_store_returns_main_node_by_hfid_when_its_peer_references_it_back(
clients: BothClients, person_with_cars_schema: dict[str, Any]
) -> None:
"""In one response, the car's owner is an id-only copy of the person: it must not hide the person's values."""
transform = PersonFromStoreTransform(
client=clients.standard, infrahub_node=InfrahubNode, convert_query_response=True, branch="main"
)
transform.client.schema.set_cache(person_with_cars_schema, branch="main")
data = {
"TestingPerson": {
"edges": [
{
"node": {
"id": PERSON_ID,
"__typename": "TestingPerson",
"name": {"value": "John"},
"age": {"value": 42},
"cars": {
"edges": [
{
"node": {
"id": CAR_ID,
"__typename": "TestingCar",
"name": {"value": "Volt"},
"owner": {"node": {"id": PERSON_ID, "__typename": "TestingPerson"}},
}
}
]
},
}
}
]
}
}
assert await transform.run(data=data) == {"name": "John", "age": 42} ## Age and name are None
A fix could be to add this code inside infrahub_sdk/operation.py, line 67->72
# A related copy of a node that the response also returns as a main node holds fewer fields: keep the main node.
main_ids = {node.id for node in self._nodes if node.id}
related_nodes = [node for node in self._related_nodes if node.id not in main_ids]
for node in self._nodes + related_nodes:
if node.id:
self._init_client.store.set(node=node)
| @@ -0,0 +1 @@ | |||
| The client store now keeps one entry per node and branch. Before, each query that returned a node already in the store, such as a repeated `client.get()`, `client.all()` or `client.filters()`, added another copy and kept the previous one in memory, so a long-running process that reused one client used more memory on every query. A lookup by ID, HFID or key returns the copy from the latest query, and `client.store.count()` now returns the number of distinct nodes. When the new copy has no HFID, for example from a query with `only` that did not request it, a lookup by the earlier HFID now returns the new copy instead of the earlier one. This includes a related node returned with a query, for example with `prefetch_relationships`, `include` or `only`: it replaces the stored copy, also one stored by `save()`, as in earlier releases. Looking up a node on a branch the store does not hold no longer adds an empty entry for that branch. | |||
There was a problem hiding this comment.
I find this changelog quite long, and it includes wording from intermediate steps, such as “as in earlier releases”. Would you agree that we should shorten it?
Just a suggestion made using Claude
The client store now keeps one copy of each node per branch. Before, each query that returned a node already in the store, such as a repeated client.get(), client.all() or client.filters(), added another copy and kept the previous one in memory, so a long-running process that reused one client used more memory on every query. client.store.count() now returns the number of distinct nodes rather than the number of copies.
The lookup changes for keys and HFIDs would go in a separate changelog/1506.changed.md, as suggested in my review.
| </TabItem> | ||
| </Tabs> | ||
|
|
||
| - The resulting related objects for objects retrieved using the SDK client's query methods, when we use the `prefetch_relationships` argument. |
There was a problem hiding this comment.
There is a double space after the "when we use".
Then, we can modify the end of the sentence to add the other cases
"prefetch_relationships or include argument."
That way, we can remove the additional paragraph line 140 #1512 (comment)
| A node returned by a [query with `only`](../../python-sdk/guides/query_data#fetching-only-the-named-attributes-and-relationships), or a peer returned with it, replaces the copy stored under the same `id`, even one that contains more fields, so check `is_loaded` on a field of a node from the store before you read it. A lookup by human-friendly ID returns the earlier copy when the new node has no human-friendly ID, for example when the query named neither `hfid` nor the fields that make up the human-friendly ID. | ||
| The store keeps one copy of each node per branch. A node returned by a later query replaces the copy stored under the same `id`, and a lookup by `id`, human-friendly ID or key returns the copy from the latest query. This includes a node returned by a [query with `only`](../../python-sdk/guides/query_data#fetching-only-the-named-attributes-and-relationships), even when the earlier copy contains more fields, so check `is_loaded` on a field of a node from the store before you read it. When the new copy has no human-friendly ID, for example when the query named neither `hfid` nor the fields that make up the human-friendly ID, a lookup by the earlier human-friendly ID returns the new copy. | ||
|
|
||
| A related node returned with a query, for example with `prefetch_relationships`, `include` or `only`, follows the same rule: it replaces the copy stored under the same `id`, also one stored by `save()` or returned by a query for its own kind. |
There was a problem hiding this comment.
This one is repetitive and probably due to the back-and-forth, we can remove it and add my last modification to capture these changes
| self._keys_by_internal_id: dict[str, set[str]] = {} | ||
| self._hfids_by_internal_id: dict[str, set[tuple[str, tuple]]] = {} | ||
| # The sync batch saves nodes from several threads. Re-entrant because computing an HFID can read this store. | ||
| self._lock = threading.RLock() |
There was a problem hiding this comment.
Having this new lock makes me think we should also add unit tests covering this behavior. I can see two options here:
- either keep lock as an internal attribute of this class and mock it with
monkeypatch, to test the behavior and check that the store acquires the lock when necessary - or inject lock into this class as a dependency, so we can replace it with a fake object in unit tests
The second option is cleaner, but I guess it’s harder to set up. So the first option is fine, we'll set up properly this dependency injection later.
Internal thoughts for future improvements:
- Write access versus read access?
- Do a similar
InfrahubLockobject in the SDK?
Why
A process that reuses one client, such as a worker or a scheduled sync, used more memory on every query. Each time a query returned a node that was already in the client store, the store added another copy and kept the previous one. Lookups returned only the newest copy, and no method freed the earlier ones.
client.store.count()counted the copies, not the nodes.Goal: the store keeps one copy of each node per branch, so its memory use stays bounded by the number of distinct nodes.
Non-goals:
remove()orclear()methods on the store. These would be new public API and belong in a separate change todevelop.node.delete(). It changes whatRelatedNode.peerand peer-based HFIDs return after a delete, and needs to handle nodes deleted by a cascade, so it is a separate change.Closes #1506
What changed
Behavioural changes:
idit already holds on that branch, it replaces the previous copy and removes that copy's ID, HFID and key entries. Keys set withstore.set(node, key=...)move to the new copy.save(), the previous HFID no longer finds it.client.all()andclient.filters()store a related node, for example fromprefetch_relationships,includeoronly, through a private store method. A related node replaces a copy that was also stored as a related node, so peers stay current. It does not replace a node returned by a query for its own kind or stored bysave(), because a related node built from relationship data can hold fewer fields and would otherwise replace that node for lookups by HFID and key. The check and the write happen under the branch lock.id, then stored again after it got one, keeps one entry, and its previous HFID entries are removed.client.store.count()returns the number of distinct nodes.onlythat did not request it, a lookup by the earlier HFID returns the new copy. Before, it returned the earlier copy. The store guide described the earlier behaviour and is updated.Implementation notes:
_internal_id, which the SDK generates again for every node object, so two objects for the same Infrahub node were stored under two keys.NodeStoreBranch._resolve_existing(). It returns the new copy today. Merging, as feature: Update existing objects within the SDK store #347 asks, would mean changing only that method._index()and_unindex()add and remove a node's entries. The store records the keys and HFID entries it wrote for each node and removes exactly those, because a node's HFID can change after it is stored, for example after a local rename._unindex()only removes entries that still point to the copy being removed.NodeNotFoundError, or returnsNonewithraise_when_missing=False.set(),get()andcount(), because the sync client's batch saves nodes from several threads andset()now reads and changes several dictionaries in turn. It is re-entrant because computing an HFID insideset()can read the same store again. There is no test for the lock: a threaded test passed with and without it.What stayed the same:
idand HFID return the copy from the latest query, except in the HFID case above. A lookup by key now does too; before, a key kept pointing to the copy it was set with.id, such as nodes not yet saved, are stored and found as before.idon two branches is still two entries.How to review
infrahub_sdk/client.py(the two places that store related nodes) andinfrahub_sdk/store.py:NodeStoreBranch.set(),_set_related(),_store(),_resolve_existing(),_index(),_unindex(),_get_by_hfid(), andNodeStoreBase._get()/_set()/_set_related().tests/unit/sdk/test_client.py::test_method_all_twice_keeps_one_store_entry_per_nodefetches the same node twice throughclient.all(). It checks the store holds one entry with the values from the second response, and does not check which object the store keeps, so it remains valid if the copies are merged later.test_method_all_with_prefetch_keeps_stored_node_over_peer_copychecks that a related node fromprefetch_relationshipsdoes not replace a node already stored.test_store.py::test_node_store_set_related_replaces_only_a_related_copychecks that a related copy replaces an earlier related copy but not a node stored withset().tests/unit/sdk/test_store.pycovers replacement (keys, old and new HFID), one and two new copies without HFID, a local rename before a new query, a rename of the same object stored again, and the unknown-branch lookup.docs/docs/python-sdk/guides/store.mdxandchangelog/1506.fixed.md.How to test
Results on this branch:
v1.24.0and passes with the fix.invoke lint-codepasses.tests/unit/pytest_plugin/test_plugin.pyand fail the same way onv1.24.0: on the machine used, theinfrahub-testcontainerspytest plugin raisesRuntimeError: invalid CPU frequency dataat start-up.Impact & rollout
client.store.count()counting copies, or on finding an earlier copy by its_internal_id, sees the new behaviour.Checklist
changelog/1506.fixed.md)docs/docs/python-sdk/guides/store.mdx)dev/orAGENTS.mddescribes the store🤖 Generated with Claude Code