fix(batch): release references to a rejected object in server-side batching

[joaquinhuigomez]

In server-side batching, a reference whose source or target object was rejected by the server is never sent, and batch.stream() doesn't exit cleanly either.

When the server reports an object error, the results handler pops the object from __objs_cache but never removes its uuid from __uuid_lookup — only the success branch does that. ReferencesBatchRequest.__pop_items holds back any reference whose uuids are still in the lookup, so a reference touching the failed object is held forever: the background loop never reaches its stop sentinel, _BgThreads.join(timeout) returns without raising, and the context exit takes 2 × (insert + 5) seconds — about three minutes at the default insert timeout — before the reference is silently dropped. failed_references stays empty, a BgBatchLoop thread keeps spinning after exit, and an explicit flush() never returns. The async client raises WeaviateBatchStreamError: Background batch tasks did not terminate after forced shutdown instead, which points at the wrong cause. The client-side batching path already removes every non-re-added uuid from its lookup, failed ones included.

Fix is four lines: discard the uuid in both error branches, mirroring the success branch (sequential locks in sync.py, nested in async_.py, matching each file's existing style). The reference is then released and sent, and whatever the server does with it is reported honestly — it shows up in failed_references if rejected — instead of being lost. Exit drops to the normal ~2 s. A uuid repeated across several error messages is safe: the second pop hits KeyError and continues before the discard, and set.discard is idempotent.

Tests: mock_tests/test_batch_stream_references.py (new file, to stay clear of #2172's edits to test_batch.py) — a mock BatchStream rejects object BAD and any reference touching it; sync and async cases assert the reference reaches the server, failed_objects == [BAD], the reference is in failed_references, exit under 5 s, and (sync) no BgBatchLoop thread left alive. Both fail on main without hanging (sync: assert 12.01 < 5; async: the forced-shutdown error after 6 s). pytest test 526 passed; mock_tests 67 passed; ruff (0.14.7 and 0.9.9), flake8 and pyright clean. Merge-tested against open #2141, #2172, #2180 and #2166: no textual overlap, new tests pass on each merged tree; #2141's unsent-data check only fires when the threads are dead, so it wouldn't have caught this — the two complement each other.

One thing I'd flag for review rather than assert: I haven't checked whether a live server rejects a reference to a missing object or stores it dangling; the fix guarantees the reference is sent and its outcome reported, not which outcome. An alternative would be to fail such references client-side as ErrorReference without sending them — happy to switch if that's the preferred semantics.

[orca-security-eu]

Orca Security Scan Summary

Status	Check	Issues by priority
Passed	Infrastructure as Code	0   0   0   0	View in Orca
Passed	SAST	0   0   0   0	View in Orca
Passed	Secrets	0   0   0   0	View in Orca
Passed	Vulnerabilities	0   0   0   0	View in Orca

[weaviate-git-bot]

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?