Repository navigation
fix: accept any Sequence (not only list) as filter values - #2169
joaquinhuigomez wants to merge 1 commit into
Conversation
`FilterValuesList` is typed as a `Sequence`, so `contains_any(("a", "b"))`
and any other non-list sequence is valid input. Both serialisers gated on
`isinstance(value, list)` instead, so a tuple was dropped from the gRPC
`Filters` message without any error, and the REST path raised a bare
`ValueError: Unknown filter value type: <class 'tuple'>`.
Normalise the value once with `_to_value_list()` and feed that to the
array helpers and to the REST parser. `str`/`bytes` are sequences too but
are single filter values, so they are excluded and keep their current
handling, including the empty-string case.
Fixing the serialisers rather than the builders covers every entry point
at once: `_FilterByProperty.equal()` and friends also accept
`FilterValuesList`, and `_FilterByTime.contains_any()` had the same gap.
`_FilterById.contains_any()` already normalised its `Sequence[UUID]` input.
There was a problem hiding this comment.
Orca Security Scan Summary
| Status | Check | Issues by priority | |
|---|---|---|---|
| Infrastructure as Code | View in Orca | ||
| SAST | View in Orca | ||
| Secrets | View in Orca | ||
| Vulnerabilities | View in Orca |
|
To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge. |
|
I have read and agree to the Weaviate Contributor License Agreement and approve the next steps. |
chrikrah
left a comment
There was a problem hiding this comment.
@joaquinhuigomez with the client at the merge base 142d798, a Weaviate 1.39.7 server rejects the tuple filter with unknown value type <nil>, in fetch_objects, aggregate.over_all and delete_many alike. At 200b57e all three match news and sports. Approving.
$ PYTHONPATH=<tree at sha> python probe_server.py <sha> # Weaviate 1.39.7 container; objects news, sports, weather
# filter: Filter.by_property("category").contains_any(("news", "sports")) on a TEXT property
142d798 fetch contains_any(list) ['news', 'sports']
142d798 fetch contains_any(tuple) raised WeaviateQueryError: ... unknown value type <nil>.
142d798 aggregate over_all contains_any(tuple) raised WeaviateQueryError: ... details = "parse params: extract filters: unknown value type <nil>" ...
142d798 delete_many contains_any(tuple), then remaining raised WeaviateDeleteManyError: ... batch delete params: unknown value type <nil>.
200b57e fetch contains_any(tuple) ['news', 'sports']
200b57e aggregate over_all contains_any(tuple) 2
200b57e delete_many contains_any(tuple), then remaining deleted=2 remaining=1
$ curl -s https://raw.githubusercontent.com/weaviate/weaviate/v1.27.0/adapters/handlers/grpc/v1/filters.go | grep -n 'unknown value type' | tr -s '\t' ' '
137: return filters.Clause{}, fmt.Errorf("unknown value type %v", filterIn.TestValue)
# v1.27.0 is the oldest server this client accepts: weaviate/connect/v4.py:964
# the description's 533 passed and 38 failing tests, reproduced on Python 3.12.3
$ python -m pytest test -q
490 passed, 1 skipped # merge base 142d798
533 passed, 1 skipped # head 200b57e
$ python -m pytest test/collection/test_filter.py -q
76 passed # head 200b57e
38 failed, 38 passed # head tests, filters.py from 142d798: all 38 failures are new tests
# the 5 new tests that pass there: test_empty_tuple_input x3, test_string_filter_values_are_not_sequences x2
non-blocking: unknown value type <nil> is the string a user with this bug would search for. Could you put it in the description in place of "no error, no warning"?
Passing a tuple where a filter expects a list of values produces a filter with no value at all.
Filter.by_property("category").contains_any(("news", "sports"))serializes to a gRPCFiltersmessage carrying the operator and target but novalue_text_array, and the query runs with it — no error, no warning, even withvalidate_arguments=True. The REST path raises a bareValueError: Unknown filter value type: <class 'tuple'>instead of aWeaviateInvalidInputError.FilterValuesListis declared asSequence[...], so a tuple type-checks, and_FilterById.contains_anyalready accepts anySequence— the property and time filters just never normalized. Rather than patch each of the nine_FilterByPropertymethods that takeFilterValues(which would still leave_FilterByTimeexposed), the serializer now converts any non-str/bytesSequenceto a list once, in a small_to_value_listhelper used by both the gRPC and REST paths. The str/bytes exclusion is load-bearing:""is a zero-lengthSequenceandequal("")has to keep working — there is a regression test for it.Tests parametrize the existing filter-to-gRPC cases over list and tuple and assert identical protos, plus the REST path; 38 of them fail on main.
pytest test533 passed; ruff, flake8 and pyright clean.