* initial implementation
* internal logging type
* wrap more places into auth function
* [manual refactor] use &Auth in .toc fucntion - minimise direct .access()
* [manual refactor] fmt
* [manual refactor] remove unannotated .access in creation of snapshots
* [manual refactor] remove unannotated .access in cluster telemetry
* [manual refactor] remove unannotated .access
* [manual refactor] do not log /metrics api access
* [manual refactor] do not run the service if audit logging failed to init
* [manual refactor] make auditable operation names for point and cluster updates
* [manual refactor] remove excess cloning of auth object
* [manual refactor] fmt
* [AI] instead of manual writing into the file, use tracing_appender crate
* [AI] simplify configuration to match internal crate
* fmt
* [manual refactor] cover staging options
* [AI] refactor x-forwarded-for handelling
* [manual refactor] use consts for x-forwarded-for + recover handelling for RFC case in tonic
* Update src/tonic/forwarded.rs
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* [manual] review fix: use FORWARDED in actix
* [manual] review fix: do not log strict mode checks
---------
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Use spawn_blocking for segment locking
* Wrap blocking task in AbortOnDropHandle
* Use single blocking thread for fetching all segment size information
---------
Co-authored-by: timvisee <tim@visee.me>
* weighted rrf implementation
* test
* fmt
* fix edge
* validate number of sources and number of weights
* do not partial match
* upd schema
* review fixes
* update formula
* remove calcualtions from tests
* update comment, because AI have OCD
* fmt
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Fix RST_STREAM by draining http2 channels
* remove controversial duration time for now
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* WAL replay honors applied_seq
fixes
* add integration test
* restore test after rebase
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Drop WAL from API
Flatten if statement
another approach
review remarks
switch flag back guard
unit test
* review remarks
* review remarks
* test missing records which are not truncated and not applied
* Drop wal from api: don't use applied seq number (#8030)
* Don't rely on applied seq num
* Apply suggestions from code review
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* remove unused
* Repurpose existing plunger, unset first_skipped in operation branch
* more complicated tests with empty wal
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* review remarks
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* download tar
* compute sha256 for stream download
* wip: propagate unpacking into down to the logic, todo: validation
* unpacked snapshot validation
* Minor tweaks
* Fix typo
* validation during unpack
* cancellation token
* update docstring
* remove redundant dep
* Rearrange unpack functions
- Rename `safe_unpack.rs` into `tar_unpack.rs` so it would be listed
near `tar_ext.rs` in IDEs.
- Replace calls like `ar = open_snapshot_archive(…); safe_unpack(ar, …);`
with a single call to `tar_unpack_file(…)`.
- Put calls to `Archive::new(); Archive::set_overwrite(false);` inside
`tar_unpack_reader` (was `safe_unpack`). So, now it is the only place
that does `set_overwrite`.
* Let clippy complain if tar::Archive::unpack used
* Mock snapshot download URL
Instead of downloading from storage.googleapis.com every time the test
runs, put small snapshot file to the repo.
The snapshot file is created using this command:
curl -s \
https://storage.googleapis.com/qdrant-benchmark-snapshots/test-shard.snapshot \
| tar \
--delete segments/4ea958d8-0b64-4312-9a53-0cd857e93535.tar \
--delete segments/65ac6276-8cca-4f5c-b767-9722190cee8b.tar \
> lib/storage/src/content_manager/snapshots/test-shard.snapshot
File contents:
$ tar tf lib/storage/src/content_manager/snapshots/test-shard.snapshot
wal/
wal/closed-255
newest_clocks.json
replica_state.json
shard_config.json
$ du -sh lib/storage/src/content_manager/snapshots/test-shard.snapshot
12K lib/storage/src/content_manager/snapshots/test-shard.snapshot
$ sha256sum < lib/storage/src/content_manager/snapshots/test-shard.snapshot
5d94eac5c1ede3994a28bc406120046c37370d5d45b489a0d2252531b4e3e1f2 -
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
* Implement `Optional<T>` type
Define a type with the same presumed layout as `Option<T>`, but with defined behavior.
* Make `transmute_*` functions unsafe
The functions `memory::mmap_ops::transmute_*` are inherently unsafe, but
are not marked as are. Their usage is documented, but it is not always clear
if the code is correct.
* Add `CsrHeader` to resolve another unsoundness
Tuples have no defined layout.
* Update queue info into CollectionInfo
* basic test
* fix
* cleaner
* Update lib/collection/src/shards/local_shard/mod.rs
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* fix the fix
* rename to op_num
* skip_serializing_if just in case
* first step
* test queue length with staging feature
* gate integration test based on binary feature
---------
Co-authored-by: xzfc <5121426+xzfc@users.noreply.github.com>
* benchmark against other caches
* use cycle
* use an enum for cache names
* make multithreaded bench with rayon
* don't include pregenerated keys in allocation
* improve benches
* improve keys with zipf + scan
* misc improvements
* Persist applied seq at regular interval
* Error handling
* fix brain fart
* don't reset update_count
* handles explicitly failure to load
* init from WAL instead of zero
* fixes
* try to regenerate file on corruption
* fix
* trust the WAL if higher watermark most likely from a transfer
* use last WAL index & handles file shard lifecycle
* last WAL index is inclusive
* include applied_seq in partial snapshot
* snapshot applied_seq file
* fix
* remove outdated comment
* Describe AppliedSeq structure, it is used for persisting
---------
Co-authored-by: timvisee <tim@visee.me>
* update queue dont keep ops in ram showcase
* always load operation from WAL
* revert operations buffering
cound only pending in update worker operations
fix typo
use channel size instead of wal index
* remove result expect
* decrease buffering const
* review remarks
* are you happy codespell
* introduce update_mode parameter for upsert operation to control if we want to insert, update, or upsert
* add test
* upd dockstring
* require resharding once all peers have updated version
* use service error
* fix clippy again
* wait for same version before resharding in tests
* WIP: introduce new vector store type
* handling of InRamMmap
* fmt
* feature-flag
* fmt
* Use if else
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Update lib/common/common/src/flags.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* also choose madvise for single-file in-ram-mmap
* simplify generics
* gpu fix
* fix bug
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
