* Unify parking_lot/arc_lock feature
* Move lib/common/{io,memory}/* -> lib/common/common/*
- Mmap-related items are grouped into `common::mmap` sub-module:
- `memory/src/chunked_utils.rs` -> `common/src/mmap/chunked.rs`
- `memory/src/madvise.rs` -> `common/src/mmap/advice.rs`
- `memory/src/mmap_ops.rs` -> `common/src/mmap/ops.rs`
- `memory/src/mmap_type_readonly.rs` -> `common/src/mmap/mmap_readonly.rs`
- `memory/src/mmap_type.rs` -> `common/src/mmap/mmap_rw.rs`
- Filesystem-related items are grouped into `common::fs` sub-module:
- `common/src/fs.rs` -> `common/src/fs/sync.rs`
- `io/src/file_operations.rs` -> `common/src/fs/ops.rs`
- `io/src/move_files.rs` -> `common/src/fs/move.rs`
- `io/src/safe_delete.rs` -> `common/src/fs/safe_delete.rs`
- `memory/src/checkfs.rs` -> `common/src/fs/check.rs`
- `memory/src/fadvise.rs` -> `common/src/fs/fadvise.rs`
- Rest is moved straight into `common`:
- `io/src/storage_version.rs` -> `common/src/storage_version.rs`
The old `io` and `memory` are now hollow crates that re-export items
from `common`. These hollow crates will be removed in next commits.
* Replace uses of `io` and `memory` with new paths in `common`
Since `io` and `memory` are just re-exports of `common`, these
replacements are no-op.
* Remove `io` and `memory` crates
* [manual] limit number of update clocks to 64 to prevent permanent overhead created by spikes
* explicit yield + fix tests
* Add explicit comment on why we yield async thread
* In resolve_wal_delta_randomized test fix clock cleanup and satisfy test
* Use tokio timeout
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Add integration test to assert all queued updates are also transferred
* Plunge update queue in stream records transfers
* Migrate existing plunger usages to new plunge helper
* Skip test if not compiled with staging flag
* Only send delay operation when staging feature is enabled
* Reformat
* Fix review remarks
* add rest and grpc interfaces
Also handle inference for this query
* follow refactor from base branch
* renaming from Ms Cooper
* get started on validation testing
* more validations
* test equivalence with query when less than 2 feedback elements
* rename feedback query to relevance_feedback query
* fix extraction of context pairs
* rename feedback vector to example
* make coderabbit happier
* upd test
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* move wal.lock.read into a spawn_blocking
* [AI] In the file with update_worker bloast the code, please try to move it out of the main function, so the function itself stay clean
* parse round floats as integers on payload indexing
* use backward conversion to validate if float is actualy integerable
* fmt
* Use then_some
---------
Co-authored-by: timvisee <tim@visee.me>
* handle `score_threshold` in Formula queries
* AI: add openapi test
prompt: Upload 4 points with a numeric payload, use the payload as the
score, and set a score threshold. We can assert which ids should be in
the result and which shouldn't
* [manual] generate snapshot download link with respect of enabled tls
* Simplify http(s) selection
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* refactor shard.create_snapshot to own everything necessary in the future for snapshotting
* do not lock shards_holder and replica_set during snapshot
* separate locks and snapshot creation all the way down
* separate locks and snapshot creation even further on the local shard level
* final touches
* minor review fixes
* initial implementation
* internal logging type
* wrap more places into auth function
* [manual refactor] use &Auth in .toc fucntion - minimise direct .access()
* [manual refactor] fmt
* [manual refactor] remove unannotated .access in creation of snapshots
* [manual refactor] remove unannotated .access in cluster telemetry
* [manual refactor] remove unannotated .access
* [manual refactor] do not log /metrics api access
* [manual refactor] do not run the service if audit logging failed to init
* [manual refactor] make auditable operation names for point and cluster updates
* [manual refactor] remove excess cloning of auth object
* [manual refactor] fmt
* [AI] instead of manual writing into the file, use tracing_appender crate
* [AI] simplify configuration to match internal crate
* fmt
* [manual refactor] cover staging options
* [AI] refactor x-forwarded-for handelling
* [manual refactor] use consts for x-forwarded-for + recover handelling for RFC case in tonic
* Update src/tonic/forwarded.rs
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* [manual] review fix: use FORWARDED in actix
* [manual] review fix: do not log strict mode checks
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use spawn_blocking for segment locking
* Wrap blocking task in AbortOnDropHandle
* Use single blocking thread for fetching all segment size information
---------
Co-authored-by: timvisee <tim@visee.me>
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Fix RST_STREAM by draining http2 channels
* remove controversial duration time for now
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Arnaud Gourlay