* Add github action to codespell master on push and PRs
* Add rudimentary codespell config
* some skips
* fix some ambigous typos
* [DATALAD RUNCMD] run codespell throughout
=== Do not change lines below ===
{
"chain": [],
"cmd": "codespell -w",
"exit": 0,
"extra_inputs": [],
"inputs": [],
"outputs": [],
"pwd": "."
}
^^^ Do not change lines above ^^^
* Add dev branch as target for the workflow
* TableOfContent now accepts Handle, not Runtime
Runtime is left within main() and dropped there to shutdown the system
* Correct termination of consensus loop
Now consensus loop is joined by main thread and
correctly terminated between iterations of the loop
* Add clippy #[allow]
* Move complicated type into definition
* Remove duplicate on_consensus_stopped call
* wait_unwrap() is now time bounded
Drop is happening only if Arc<TableOfContent> is droped by all threads
in 30 seconds to limit shutdown time
---------
Co-authored-by: timvisee <tim@visee.me>
* Issue 1905: Configurable location for the tmp snapshot files
* Apply suggestions from code review
Co-authored-by: Tim Visée <tim+github@visee.me>
* fix code review suggestions
* clippy fix
* Propagate temp path, use configured dir for snapshot creation
* Use real temp dir in snapshot tests
* Mention default temporary snapshot file path in configuration
* Use temp everywhere rather than a mix of temp and tmp
* Use consistent naming for temporary snapshot directories
* Extract logic for temporary storage path into toc method
* Resolve clippy warnings
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* async raw scorer
* fmt
* Disable `async_raw_scorer` on non-Linux platforms
* Refactor `async_raw_scorer.rs`
* Conditionally enable `async_raw_scorer` in `segment` crate
* Add `async_scorer` config parameter to the config...
...and enable `async_raw_scorer`, if config parameter is set to `true`
* fixup! Add `async_scorer` config parameter to the config...
Fix tests
* Add basic `async_raw_scorer` test
* Extend `async_raw_scorer` tests to be more extensive
* Async uring vector storage io uring (#2041)
* replace tokio-uring with just low level io-uring
* fnt
* minor fixes
* add sync
* wip: try to use less submissions
* fmt
* fix uring size
* larger buffer
* check for overflow
* submit with re-try
* mmap owns uring context
* large disk parallelism
* rollbacK: large disk parallelism
* fix windows build
* explicitly panic on uring fail
* fix windows build again
* use async scorer in the quantization re-scoring
* refactor
* rename UringReader
* refactor buffers
* fix for windows
* error checking
* fix handing
---------
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* rename `Lookup` -> `RetrievedLookup`
* feat: expose lookup in groups
- make `with_vectors` and `with_payload` optional
- include in grpc
- move GroupId to inside of PointGroup message
- include `WithLookup` inside of `GroupRequest`, doesn't make sense to keep them separated
* chore: remove allow(dead_code) from builder
* feat: introduce `WithLookupInterface` to allow specifying only the name of the lookup collection
* fix: WithLookupInterface with collection name only brings all payload and vectors
* fix: update grpc docs
* add tests and set better defaults
* remove `RetrievedLookup` from proto
* short-circuit `fill_search_result_with_payload` when it's not needed
* cargo fmt
* proto: revert putting `GroupId` inside `PointGroup`
* update grpc docs
* Remove WithLookupInterface sugar from grpc
- set with_vectors of lookup default to false
- set with_payload of lookup default to true
* update integration tests
* use singular lookup
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Migrate from OpenSSL to Rustls
* Add TTL based certificate rotation for Rustls in actix
See: <https://github.com/qdrant/qdrant/pull/1865#issuecomment-1539752859>
* Update last update time when certificate is replaced
* Update error message
* Add option to disable certificate TTL, add TTL validation
* Cleanup
* Update comment for service.enable_tls, also enables TLS for gRPC
* Apply suggestions from code review
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Move with_buf_read into certificate_helpers, it is used there
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Remove level of scoping
* Add TLS enabled/disabled log messages
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: Roman Titov <ffuugoo@users.noreply.github.com>
* Clone current segment config to deprecated type
* Remove segment level quantization config from segment config
* Also deprecate current VectorDataConfig
* Update old segment migration to work with new refactoring
* Move index into vector data config
* Move vector data config migration logic into segment level
* Remove hnsw_config from vector data config
* Rename collection params to vector data conversions function
* Move storage type into vector data config
* Set appendable flag correctly
* Clean up and reformat
* Make segment on disk flag not optional
* Add appendable flag to segment config to replace storage type
* Remove storage type from segment config
* Deprecate storage type enum
* Use consistent variable naming
* Cleanup
* Add segment config migration for v0.5.0 to current
* Bump segment to 0.6.0
* Remove serde defaults for new storage and vector data config types
These default value configurations are not needed anymore, because these
structs are not used to deserialize old data. All current fields should
always be available in these structs. When new fields are added in new
functions, the serde default annotation must be set again.
* Cleanup
* Update OpenAPI specification
This updates the returned data structure on telemetry endpoints, as a
result of segment configuration refactoring.
* Fix quantization configuration not falling back to collection config
* Fix compiler warning when building in release mode
* Move deprecated type structs into compat module
* Update allow deprecated attributes
* Assign quantization config only in segment optimizer
* Remove unsued parameter
* Add vector storage type enum to vector data config
* Remove appendable and on_disk flags from segment and vector config
* Update OpenAPI specification
* add tests
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* wip: chunked mmap
* Fix typo
* insert and get methods
* dynamic bitvec
* clippy
* wip: vector storage
* wip: fmt
* wip: mmap chunks
* wip: mmap problems
* Share transmuted mutable reference over mmap
* option to enable appendable mmap vectors
* fmt
* rename storage status file
* update tests
* fix get deleted value range
* add recovery to vector storage tests
* add flush to tests
* fix transmute from immutable to mutable
* make transmuted pointer private
* remove unused unsafe functions
* force WAL flush if wait=true
* move wal flush into updater thread
* remove flush from update api
* Minimize pub visibility for specialized/dangerous functions
* Allocate vector with predefined capacity
* Inline format parameters
* Assert we have multiple chunks while testing, test is useless otherwise
* Remove unnecessary scope
* Remove unnecessary dereference
* Random bool has 0.5 as standard distribution, use iter::repeat_with
* Replace RemovableMmap::new with Default derive
* Rename len to num_flags
* Use Option replace as it is convention alongside take
* Add FileId enum to replace error prone manual ID rotating
* Use debug_assert_eq where applicable
* Refactor drop and set to replace
* Change default chunk size for chunked mmap vectors to 32MB
This change is made as per GitHub review, because allocating a few
storages with 128MB would take a significant amount of time and storage.
See: https://github.com/qdrant/qdrant/pull/1838#discussion_r1187215475
* Replace for-loops with iterators
* Draft: add typed mmap to improve code safety (#1860)
* Add typed mmap
* Replace some crude mmap usages with typed mmap
* Use typed mmap for deleted flags
* Simplify dynamic mmap flags a lot with new typed mmap, remove flags option
* Reformat
* Remove old mmap functions that are now unused
* Reimplement mmap locking for mmap_vectors
* Add MmapBitSlice tests
* Replace MmapChunk with new typed mmap
* Update docs
* Clean-up
* Disable alignment assertions on Windows for now
* Rename mmap lock to mlock to prevent confusion with lockable types
* one more small test
* Some review fixes
* Add aliasing note
* Add basic error handling in typed mmap constructors
* Use typed mmap error handling throughout project
* Move mmap type module to common
* Fix transmute functions being unsound
See https://github.com/qdrant/qdrant/pull/1860#discussion_r1188593854
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Tim Visée <tim+github@visee.me>
* Remove unnecessary mut
* Simplify creating option with if
* Simplify some returns
* DRY in vector name checking functions
* Use panic with attribute rather than debug_assert false
* Make update vector function plural because we can update multiple
* WIP: Start working on out-of-RAM errors handling [skip ci]
* Implement basic handling of out-of-RAM errors during Qdrant startup
* Try to fix CI fail by allowing both V1 and V2 cgroups
* Try to fix CI fail by improving cgroups handling
* Fix cgroups path detection/handling (+ some minor stylistic changes)
* fixup! Fix cgroups path detection/handling (+ some minor stylistic changes)
* Add test
* Enable low RAM test
* fixup! Add test
* free memory checks
* rm unused function
* Oom fallback script (#1809)
* add recover mode in qdrant + script for handelling OOM
* fix clippy
* reformat entrypoint.sh
* fix test
* add logging to test
* fix test
* fix test
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Minor collection optimizer cleanup
* Make optimizers better aware of available vs soft deleted points
* Fix incorrect deleted state on proxy segment for double delete
* Rename upsert_vector to upsert_point, because we work with points
* Refactor point methods for more clear and consistent naming
* Replace internal_size in IdTracker with total_point_count
* Keep track of vector deletion count on storage creation
* Add sparse index optimizer, to optimize indexes with high deletion count
* Add minimum vector count threshold to sparse index optimizer
* Add sparse index optimizer test
* Use consistent naming, write vector in full everywhere
* Simplify vacuum optimizer a bit
* Merge sparse index optimizer into vacuum optimizer
* Improve update_from in segment builder by returning early
* More accurately count vectors in segment optimizer
* Remove random from vacuum optimizer tests to make them more reliable
* Don't expose the total points in segment info, use available points
* Process review feedback
* Compare available vectors against indexed ones in vacuum optimizer
This is much better than using the number of soft-deleted vectors when
the segment was created for calculations. Not to mention that value had
other problems as well.
* Remove create_deleted_vector_count field, update vacuum test parameters
* Potentially solve out of bound panic when building index
* Review fixes:
- Propagate deleted flags into payload hnsw building
- Use `total` number of points for building HNSW instead of number of
available points
- minor refactoring of `hnsw_config` copy -> clone
- Better detection of `indexed_points` in HNSW
* fix assert condition
* Optional named vectors optimizer reveiw 2 (#1794)
* review with Ivan
* fmt
* remove available_vector_count from segment entry
* remove total_point_count from segment entry
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* rollback changes in deleted count in proxy segment
* improve vector threshold detection logic in optimized_segment_builder
* style changes
* fix propagate deleted points to vectors
* Fix typo in method name
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
