You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Client-side batching: wholesale chunk failures are reported under chunk-local indices, so errors collide and describe the wrong objects #2165
With client-side batching, when a whole chunk fails to be sent, BatchObjectReturn.errors
reports fewer failures than actually happened, and the surviving keys describe the wrong
objects.
BatchObjectReturn documents the opposite (weaviate/collections/classes/batch.py, class
docstring):
The keys of the errors and uuids dictionaries will always be equivalent to the original_index of the objects as you added them to the batching loop
Root cause
The wholesale-failure handler in _BatchBase.__send_batch keys its errors by the position inside the chunk that was sent:
Every other place that fills this dictionary uses the global index — obj.index in grpc_batch.py:157, sync.py:367/:410, async_.py:409/:452 — because BatchObjectReturn.__add__ merges chunk results with a plain self.errors.update(other.errors)
(classes/batch.py:231) and does no re-keying. So with a chunk size smaller than the batch,
chunk N re-uses keys 0..len(chunk)-1 and overwrites chunk N-1's entries.
Two observable symptoms:
len(result.errors) under-reports the number of failed objects.
If one chunk succeeds and a later chunk fails wholesale, the successful chunk's uuids
keys and the failed chunk's errors keys collide and describe different objects — result.uuids[2] and result.errors[2] refer to two different inputs, so a caller that
retries "the failed indices" re-inserts rows that already went in and skips rows that did not.
Reproduction
Offline, mock gRPC only, no server and no API keys. Four objects with fixed_size(batch_size=2), and the mock rejects every object of both chunks (which makes _BatchGRPC.objects raise WeaviateInsertManyAllFailedError, landing in that except Exception):
Activity