Commit Graph

13 Commits

Author SHA1 Message Date
Andrey Vasnetsov
3ce4035df3 allow to read related issues with collection-level access (#5609)
* allow to read related issues with collection-level access

* refactor to use requirements

* patch test

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
2024-12-09 15:29:11 +01:00
Andrey Vasnetsov
633d996e61 HasVector filtering condition (#5303)
* include vector storage into struct vector index

* implement has_vector

* generate schemas

* refactor query filter optimizer so avoid too many function arguments

* test + fix for sparse vectors

* Update lib/segment/src/index/struct_payload_index.rs

Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>

* Update lib/segment/src/index/query_optimization/optimizer.rs

Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>

* fmt

---------

Co-authored-by: Jojii <15957865+JojiiOfficial@users.noreply.github.com>
2024-10-25 18:47:03 +02:00
Luis Cossío
025f4e3066 Expose on_disk text index (#5074)
* map index: fix reachable code marked as unreachable

* plumber work to get mmap text index to interfaces

* test: add fixture for mmap text index, always create mmap segment

* various fixes

- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index

* update grpc docs and openapi

* implement return of files

* review nit

* fix after rebase

---------

Co-authored-by: generall <andrey@vasnetsov.com>
2024-09-16 21:27:30 +02:00
Jojii
e9bad464ff [Strict-Mode] Basic implementation (#4887)
* add CollectionRequestVerification

* add to api

* rebase

* improve implementation

* implement strict mode for SearchRequest+Batch

* improve code + fix Clippy

* improve error handling

* restructure StrictModeVerification trait

* generate docs

* check `enabled` option

* review remarks

* rename StrictModeConfigDiff in grpc

* use missing payload detection from issue api

* performance improvement

* decouple extractor from issues (#4945)

* some review remarks

* don't default to empty functions in StrictModeVerification trait

* update openapi

* filter_limit => query_limit

* replace discovery_max_context_size and recommend_max_examples with max_input_examples

* review remarks

* review fix: include possible index types into error message

* review remarks

---------

Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: generall <andrey@vasnetsov.com>
2024-08-29 10:49:23 +02:00
Andrey Vasnetsov
abd51d984a Implement better handling of UUID index (#4961)
* rename keyword -> string for internal data structures

* implement MatchAny and expect for filter on UUID mmap index

* implement MatchAny and expect for cardinality estimation on UUID mmap index

* refactor PayloadFieldIndex filter for handling incorrect empty query parsing case
2024-08-27 00:53:30 +02:00
Luis Cossío
0e38ab3fe7 Issues: Accept UUID index (#4953) 2024-08-26 18:27:56 -04:00
Andrey Vasnetsov
644b01ee89 Refactor resharding filter (#4799) 2024-08-02 13:48:45 +02:00
xzfc
a0ea3caccf Enable some of the pedantic clippy lints (#4715)
* Use workspace lints

* Enable lint: manual_let_else

* Enable lint: enum_glob_use

* Enable lint: filter_map_next

* Enable lint: ref_as_ptr

* Enable lint: ref_option_ref

* Enable lint: manual_is_variant_and

* Enable lint: flat_map_option

* Enable lint: inefficient_to_string

* Enable lint: implicit_clone

* Enable lint: inconsistent_struct_constructor

* Enable lint: unnecessary_wraps

* Enable lint: needless_continue

* Enable lint: unused_self

* Enable lint: from_iter_instead_of_collect

* Enable lint: uninlined_format_args

* Enable lint: doc_link_with_quotes

* Enable lint: needless_raw_string_hashes

* Enable lint: used_underscore_binding

* Enable lint: ptr_as_ptr

* Enable lint: explicit_into_iter_loop

* Enable lint: cast_lossless
2024-07-22 08:19:19 +00:00
xzfc
69b63baa07 Drop JsonPathString (#4621)
* drop some code

* Drop JsonPathString

* Fix test_remove_key

Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.

* Make json_path::path() non-generic

* Remove references to JsonPathV2

* Drop JsonPathInterface

* Move json_path::v2 code into json_path

* Drop validate_not_empty

* Drop JsonPath::head() as being unused

* Replace path() with JsonPath::new()

* Restore comments

* Move tests to json_path

* Use json() consistently in tests

* Replace many into calls with Into trait

---------

Co-authored-by: timvisee <tim@visee.me>
2024-07-11 04:06:40 +00:00
xzfc
7b8ac92208 Extend PayloadSchemaParams to every PayloadSchemaType (#4613)
* Move IntegerIndexType and TextIndexType into a common file

* Formatting

* Extend PayloadSchemaParams to every PayloadSchemaType
2024-07-05 12:09:31 +00:00
Roman Titov
d4b32012eb Rework CommitHashRing consensus message into CommitRead/CommitWrite/Finish (#4417)
* Refactor `CommitHashRing` into `CommitRead`/`CommitWrite`/`Finish`

* Add `Resharding` filter condition

* Filter "resharded" points from search, scroll by, count and retrieve request results

* fixup! Refactor `CommitHashRing` into `CommitRead`/`CommitWrite`/`Finish`

`cargo clippy --fix`

* Apply suggestions from code review

* fixup! Filter "resharded" points from search, scroll by, count and retrieve request results

Add `Condition::is_local_only` method

* fixup! Add `Resharding` filter condition

* fixup! Filter "resharded" points from search, scroll by, count and retrieve request results

Clarified a few `TODO`s

* Fix clippy suggestions

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
2024-06-19 12:14:26 +02:00
Luis Cossío
fef3e142fd proofread issue messages (#4266) 2024-05-20 11:38:46 -04:00
Luis Cossío
a4308f5308 Issues API: Unindexed fields (#4139)
* submit an issue for unindexed field

solve unindexed field issue when an index is created for the field

also log a warning when issue is submitted

fmt

fix rebase problems

get collection name without panicking

better solving of issue

remove outdated TODO

add tests, fix GET /issues

review fixes

use `/` instead of `.` as separator

fmt

prepare extractor for time-based submission

smol rename

add check in search

hook unindexed field creation in search and search_batch of rest

update after rebase on `dev`

remove submit from `struct_payload_index`

solve issues inside of handlers, not in match router

post-process at Collection level

don't clone filters

remove from openapi spec

- Add hidden env variable to adjust slow search threshold
- Remove Solution::None
- Fix UnindexedField issue extractor
- Add endpoint to openapi, but without response body spec
- Move integration test to consensus_tests to set lower threshold

* event-based integration

* use typed Code

* remove payload_index_schema hack, get through collection info

* rename `notify` to `publish`

* update after rebase

* remove breakpoint

* update after rebase

* attach current payload schema to SlowQueryEvent

* collect filters refs lazily

* review fixes

* review fixes

* use regular config for setting the threshold, use 1.2 secs as default

* fix checking against current schema for all conditions
2024-05-09 09:39:01 -04:00