Skip to content

fix: accept any Sequence (not only list) as filter values - #2169

Open
joaquinhuigomez wants to merge 1 commit into
weaviate:mainfrom
joaquinhuigomez:fix/filter-accepts-sequence-values
Open

joaquinhuigomez wants to merge 1 commit into
weaviate:mainfrom
joaquinhuigomez:fix/filter-accepts-sequence-values

Conversation

@joaquinhuigomez

Copy link
Copy Markdown

Passing a tuple where a filter expects a list of values produces a filter with no value at all. Filter.by_property("category").contains_any(("news", "sports")) serializes to a gRPC Filters message carrying the operator and target but no value_text_array, and the query runs with it — no error, no warning, even with validate_arguments=True. The REST path raises a bare ValueError: Unknown filter value type: <class 'tuple'> instead of a WeaviateInvalidInputError.

FilterValuesList is declared as Sequence[...], so a tuple type-checks, and _FilterById.contains_any already accepts any Sequence — the property and time filters just never normalized. Rather than patch each of the nine _FilterByProperty methods that take FilterValues (which would still leave _FilterByTime exposed), the serializer now converts any non-str/bytes Sequence to a list once, in a small _to_value_list helper used by both the gRPC and REST paths. The str/bytes exclusion is load-bearing: "" is a zero-length Sequence and equal("") has to keep working — there is a regression test for it.

Tests parametrize the existing filter-to-gRPC cases over list and tuple and assert identical protos, plus the REST path; 38 of them fail on main. pytest test 533 passed; ruff, flake8 and pyright clean.

`FilterValuesList` is typed as a `Sequence`, so `contains_any(("a", "b"))`
and any other non-list sequence is valid input. Both serialisers gated on
`isinstance(value, list)` instead, so a tuple was dropped from the gRPC
`Filters` message without any error, and the REST path raised a bare
`ValueError: Unknown filter value type: <class 'tuple'>`.

Normalise the value once with `_to_value_list()` and feed that to the
array helpers and to the REST parser. `str`/`bytes` are sequences too but
are single filter values, so they are excluded and keep their current
handling, including the empty-string case.

Fixing the serialisers rather than the builders covers every entry point
at once: `_FilterByProperty.equal()` and friends also accept
`FilterValuesList`, and `_FilterByTime.contains_any()` had the same gap.
`_FilterById.contains_any()` already normalised its `Sequence[UUID]` input.

@orca-security-eu orca-security-eu Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Orca Security Scan Summary

Status Check Issues by priority
Passed Passed Infrastructure as Code high 0   medium 0   low 0   info 0 View in Orca
Passed Passed SAST high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Secrets high 0   medium 0   low 0   info 0 View in Orca
Passed Passed Vulnerabilities high 0   medium 0   low 0   info 0 View in Orca

@weaviate-git-bot

Copy link
Copy Markdown

To avoid any confusion in the future about your contribution to Weaviate, we work with a Contributor License Agreement. If you agree, you can simply add a comment to this PR that you agree with the CLA so that we can merge.

beep boop - the Weaviate bot 👋🤖

PS:
Are you already a member of the Weaviate Forum?

@joaquinhuigomez

Copy link
Copy Markdown
Author

I have read and agree to the Weaviate Contributor License Agreement and approve the next steps.

@chrikrah chrikrah left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@joaquinhuigomez with the client at the merge base 142d798, a Weaviate 1.39.7 server rejects the tuple filter with unknown value type <nil>, in fetch_objects, aggregate.over_all and delete_many alike. At 200b57e all three match news and sports. Approving.

$ PYTHONPATH=<tree at sha> python probe_server.py <sha>   # Weaviate 1.39.7 container; objects news, sports, weather
# filter: Filter.by_property("category").contains_any(("news", "sports")) on a TEXT property
142d798 fetch contains_any(list)                       ['news', 'sports']
142d798 fetch contains_any(tuple)                      raised WeaviateQueryError: ... unknown value type <nil>.
142d798 aggregate over_all contains_any(tuple)         raised WeaviateQueryError: ... details = "parse params: extract filters: unknown value type <nil>" ...
142d798 delete_many contains_any(tuple), then remaining raised WeaviateDeleteManyError: ... batch delete params: unknown value type <nil>.
200b57e fetch contains_any(tuple)                      ['news', 'sports']
200b57e aggregate over_all contains_any(tuple)         2
200b57e delete_many contains_any(tuple), then remaining deleted=2 remaining=1
$ curl -s https://raw.githubusercontent.com/weaviate/weaviate/v1.27.0/adapters/handlers/grpc/v1/filters.go | grep -n 'unknown value type' | tr -s '\t' ' '
137: return filters.Clause{}, fmt.Errorf("unknown value type %v", filterIn.TestValue)
# v1.27.0 is the oldest server this client accepts: weaviate/connect/v4.py:964
# the description's 533 passed and 38 failing tests, reproduced on Python 3.12.3
$ python -m pytest test -q
490 passed, 1 skipped     # merge base 142d798
533 passed, 1 skipped     # head 200b57e
$ python -m pytest test/collection/test_filter.py -q
76 passed                 # head 200b57e
38 failed, 38 passed      # head tests, filters.py from 142d798: all 38 failures are new tests
# the 5 new tests that pass there: test_empty_tuple_input x3, test_string_filter_values_are_not_sequences x2

non-blocking: unknown value type <nil> is the string a user with this bug would search for. Could you put it in the description in place of "no error, no warning"?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants