color ci into green
immutable index tests
deletion test
separate filter function
are you happy fmt
add shrink_to_fit
refactor estimate_cardinality
check is points_count is zero
* Serialize clock map entries as vector tuple, cannot use struct as map key
* Log warning when storing clock map or acknowledging fails
* Add test for (de)serializing, storing and loading clock map
* Tweak serialization/deserialization format
* fixup! Tweak serialization/deserialization format
* Remove transparent from clock, we flatten it elsewhere
* Tweak `ClockMap` serde tests so that they don't require disk access
* fixup! Tweak `ClockMap` serde tests so that they don't require disk access
* fixup! Tweak `ClockMap` serde tests so that they don't require disk access
Run `cargo check` without `--all --tests` 🤦♀️
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Take clock tag as value, because it implements Copy
* Implement Display for recovery point
* Add WAL function to read from newest to oldest, including physical
* Add local shard method to resolve WAL delta based on recovery point
* Add error reporting to WAL delta resolve method
* Use lowercase error message
* Use real first physical WAL index
* Do not skip forced entries when resolving WAL delta
* Use single error branch for unsupported shard types
* Propagate WAL delta resolve method up to shard replica set
* Change physical WAL naming
* Decouple WAL delta resolve method into standalone function
* Pool larger memory batches for sparse vector search
* rework naming
* refine naming
* address great comments
* bump ADVANCE_BATCH_SIZE to use power of 2
* 16k performs bad - trying 8k
* 10k offers the best tradeoff
* Repurpose queue proxy, allow using history as well for WAL diff transfer
* Use physical WAL bounds for queue proxy range checking
* Update lib/collection/src/shards/queue_proxy_shard.rs
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Define WAL bound constraints differently
* Fix off by one error in queue proxy, rename last field to transfer_from
* Start from should be one higher
* WAL bounds are inclusive
* Fix items left calculations in queue proxy shard
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* docs, cardinality estimation, grpc not implemented yet
* Add min_should field in Filter struct
* min_should clause checks whether at least given number (min_count) of conditions are met
* modify test cases due to change in Filter struct (set min_should: None)
* add simple condition check unit test
* Impl min_should clause in REST API
* perform cardinality estimation by estimating cardinalities of intersection and combining as union
* add openapi spec with docs update
* add integration test
* Impl min_should clause in gRPC
* Cargo fmt & clippy
* Fix minor comments
* add equivalence test between min_should and must
* shortcut at min_count matches
* use `Filter::new_*` whenever possible
* Add missing min_should field
* Fix gRPC field ordering & remove deny_unknown_fields
* Empty commit
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Add initial gRPC call for requesting WAL recovery point for remote shard
* Add remote shard method to request WAL recovery point
* Add recovery point type in gRPC, use it in recovery point functions
* Add function to extend recovery point with missing clocks from clock map
* Add new gRPC type for recovery point clocks
* Remove atomic loading, because we use regular integers now
* Add clock set test for single clock
* Add clock set test for ticking a single clock guard twice
* Add test for advancing single clock to a high number
* Add test for advancing single clock to a low number
* Add test for ticking multiple clocks in various configurations
* Add clock test with some clocks that take a long time to resolve
* Test with 5000 clocks from a clock set
* Test many clocks, where each iteration, we get one more clock stuck
* Add documentation to test functions
* Reformat
* support \`start_from\`: DateTime
* add order by datetime test
* generate openapi models and grpc docs
* fixup after rebase
* allow string representation of datetime in grpc
* add TODO
* fix `.start_from()`
* use custom deserialization on datetime
* Add `ClockMap` type to track versions of update operation in local shard WAL
* `ClockMap::advance_clock_and_correct_tag`: only correct `clock_tag.clock_tick` if it's `0`
* Add `ClockMap::load` and `ClockMap::store`
* WIP: Load/restore `ClockMap` from disk/WAL when loading `LocalShard`
* WIP: Save `ClockMap` to disk when truncating WAL
* Do not apply operation with `clock_tick` that is older than clock value in `ClockMap`
* Handle `ClockMap` properly during shard snapshot operations
* Use "atomic disk write" when saving `ClockMap` to disk
* Use `into` instead of explicit `Atomic*::new`
* Return *local* shard result (instead of *remote* shard result) when updating `ForwardProxyShard`
* Refactor `ClockMap::advance_clock*`
* Refactor `ClockMap::advance_clock*` once again
* Add documentation and `must_use` attributes
* Use `atomicwrites` crate for `ClockMap::store`
* WIP: Add `force` flag to the operations forwarded by forward/queue proxy shards...
...and add `force` flag handling to the `ClockMap`
* Disable force on forward proxy, always accept old clock values
---------
Co-authored-by: timvisee <tim@visee.me>
* Extend datetime formats in REST API (#3529)
* Fix linter problem
* Support indexing and condition checking and optimize tests
* Update lib/segment/src/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/segment/src/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Update lib/segment/src/types.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* Add some more tests for wrong or unsupported datetime format
* Reformat
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
* add benches for large MatchAny
* use HashSet for MatchAny
* use fnv hash
* make fnv workspace level dependency; apply clippy
* remove SmolStr from Keyword; Improve performance
* add bench for small number of keywords
* fix openapi
* fix performance issue
* apply integer optimization; create magic number constant
* add `truncate(true)` where unspecified
* remove unused `WriteGuard` wrapper
* Don't truncate if we explicitly set the size later
---------
Co-authored-by: timvisee <tim@visee.me>
* Change max_ack_version to be wal_keep_from
This shifts the logic by one. The variable now specifies what version to
keep rather than the last version we can acknowledge.
* Update comment
* Limit ack function visibility to prevent unwanted callers
* Use better name for wal_keep_from parameter
* introduce `OrderingValue` for ordering also based on int values
* use `num-cmp` for comparing ints and floats
* Update comparison logic
---------
Co-authored-by: Albert Safin <xzfcpw@gmail.com>
* Add `ClockSet` type to generate clock tags for update operations
* Generate and assign clock tag to update operations
* Advance clock if replica returns *newer* tick for operation
* Cleanup `clock_set` module (and fix a minor bug in `Clock::advance_to`)
* Apply suggestions from code review
* Minor `ClockSet` fixes
* Make `Clock` start from `0` instead of `1`
* Rename clock value to next_tick to prevent ambiguity
The field always holds the next tick value we will be using, and our
tick functions return first and increment after. This helps making this
behavior more clear.
* add drops
* Minor `ClockSet` fixes
* Safe-guard `ClockSet` from advancing past clock `0` "prematurely"
* Use `into` instead of explicit `Atomic*::new`
* Remove unnecessary conditional in `ShardReplicaSet::update`
* Update `clock_set::Clock` documentation, rename `Clock::lock` to `Clock::try_lock`
* fixup! Update `clock_set::Clock` documentation, rename `Clock::lock` to `Clock::try_lock`
Fix a typo
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
* merge range and datetime range into one interface
* remove separate datetime_range from FieldCondition
* review fixes, remove code duplication. Thanks @xzfc
* update openapi test
* Optimize sparse vector search with batched scoring
* replace roaring bitmap by id generation
* compute intersections in Vec instead of Map
* cargo update ahash@0.8.7 --precise 0.8.5
* let rustc handle the pattern match optimization
* profiling done
* remove Option and filter out zero scores
* restore integration test assertion
* filter based on min score to avoid work
* decrease batch size and cleanup posting creation
* remove outdated tests
* remove dead test - I know where to find it if I need it
