* Minor collection optimizer cleanup
* Make optimizers better aware of available vs soft deleted points
* Fix incorrect deleted state on proxy segment for double delete
* Rename upsert_vector to upsert_point, because we work with points
* Refactor point methods for more clear and consistent naming
* Replace internal_size in IdTracker with total_point_count
* Keep track of vector deletion count on storage creation
* Add sparse index optimizer, to optimize indexes with high deletion count
* Add minimum vector count threshold to sparse index optimizer
* Add sparse index optimizer test
* Use consistent naming, write vector in full everywhere
* Simplify vacuum optimizer a bit
* Merge sparse index optimizer into vacuum optimizer
* Improve update_from in segment builder by returning early
* More accurately count vectors in segment optimizer
* Remove random from vacuum optimizer tests to make them more reliable
* Don't expose the total points in segment info, use available points
* Process review feedback
* Compare available vectors against indexed ones in vacuum optimizer
This is much better than using the number of soft-deleted vectors when
the segment was created for calculations. Not to mention that value had
other problems as well.
* Remove create_deleted_vector_count field, update vacuum test parameters
* Potentially solve out of bound panic when building index
* Review fixes:
- Propagate deleted flags into payload hnsw building
- Use `total` number of points for building HNSW instead of number of
available points
- minor refactoring of `hnsw_config` copy -> clone
- Better detection of `indexed_points` in HNSW
* fix assert condition
* Optional named vectors optimizer reveiw 2 (#1794)
* review with Ivan
* fmt
* remove available_vector_count from segment entry
* remove total_point_count from segment entry
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* rollback changes in deleted count in proxy segment
* improve vector threshold detection logic in optimized_segment_builder
* style changes
* fix propagate deleted points to vectors
* Fix typo in method name
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* Exclude deleted vectors from HNSW graph building stage
* When estimating query cardinality, use available points as baseline
We should not use the total number of points in a segment, because a
portion of it may be soft deleted. Instead, we use the available
(non-deleted) points as baseline.
* Add plain search check to unfiltered HNSW search due to deleted points
* Cardinality sampling on available points, ignore deleted named vectors
* Estimate available vectors in query planner, now consider deleted points
In the query planner, we want to know the number of available points as
accurately as possible. This isn't possible because we only know the
number of deletions and vectors can be deleted in two places: as point
or as vector. These deletions may overlap. This now estimates the number
of deleted vectors based on the segment state. It assumes that point and
vector deletions have an overlap of 20%. This is an arbitrary
percentage, but reflects an almost-worst scenario.
This improves because the number of deleted points wasn't considered at
all before.
* Remove unused function from trait
* Fix bench compilation error
* Fix typo in docs
* Base whether to do plain search in HNSW upon full scan threshold
* Remove index threshold from HNSW config, only use full scan threshold
* Simplify timer aggregator assignment in HNSW search
* Remove vector storage type from cardinality function parameters
* Propagate point deletes to all its vectors
* Check for deleted vectors first, this makes early return possible
Since point deletes are now propagated to vectors, deleted points are
included in vector deletions. Because of that we can check if the vector
is deleted first so we can return early and skip the point deletion
check.
For integrity we also check if the point is deleted, if the vector was
not. That is because it may happen that point deletions are not properly
propagated to vectors.
* Don't use arbitrary vector count estimation, use vector count directly
Before we had to estimate the number of vectors (for a named vector)
because vectors could be deleted as point or vector. Point deletes are
now propagated to vector deletes, that means we can simply use the
deleted vector count which is now much more accurate.
* When sampling IDs, check deleted vecs before deleted points
* On segment consistency check, delete vectors for deleted points
* Fix vector delete state not being kept when updating storage from other
* Fix segment builder skipping deleted vectors breaking offsets
* update segment to handle optional vectors + add test (#1781)
* update segment to handle optional vectors + add test
* Only update stored record when deleting if it wasn't deleted already
* Reformat comment
---------
Co-authored-by: timvisee <tim@visee.me>
* Fix missed vector name test, these are now marked as deleted
* upd test
* upd test
* Update consensus test
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* Add QuantizationConfigDiff type
* Add quantization config diff to vector parameters
* Prefer vector config over collection config for quantization
* Update OpenAPI specification
* Validate quantization configuration quantile in 0.5-1.0 range
As per https://github.com/qdrant/qdrant/pull/1681
* Add test if check if vector specific quantization config is persisted
* Alias quantization to quantization_config in vector parameters
* Remove quantization config diff, use full vector specific config instead
* Regenerate OpenAPI specification and gRPC docs
* Fix compilation error
* Add error handling to quantization config conversions
* Fix quantization integration test, make HNSW test stricter
* Validate VectorConfig/VectorParams, remove obsolete validation
* Add HNSW config diff to vector parameters
* Validate params in collection config
* Add HNSW config to segment vector data config
* Add VectorsConfig params iterator for more elegant conversions
* Prefer vector HNSW config over collection config for building HNSW index
* Base segment vector param HNSW config on collection config
* General improvements
* Rewrite HNSW ef_construct extract function to also consider vector configs
* Update OpenAPI specification
* Add test to check if vector specific HNSW config is persisted
* review changes
* review changes
* Regenerate gRPC docs
* Fix test on Windows
* Regenerate OpenAPI specification
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* use common channel pool for healthcheck requests
* add dynamic channel pool
* fmt
* lower parallel connections num
* fmt
* fix clippy
* fix clippy
* better status error message
* smarter channel dropping conditions
* per-channel stats
* refactor pool
* rollback dockerfile
* health-check timeout
* prevent reporting last active peer as dead
* introduce update rate limit
* only rate-limit client requests
* ability to save snapshot without locking wal (#1685)
* ability to save snapshot without locking wal
* fix empty wal saving
* skip waiting on local shard in listener mode even if it is a direct request to listener shard
* snapshot recovery test + better handling of segment versions
* use latest wal
* review changes
* remove deleted flags from vector storage
* remove deleted flags from mmap
* new simple vector storage format
* are you happy clippy
* remove id_tracker from raw_scorer
* revert vector storage format changes
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* integrate quantized data to storages
* revert gitignore
* are you happy clippy
* quantize in optimizer
* provide flag
* fix segfault
* skip quantization flag, update scores
* use quantization flag
* are you happy fmt
* use quantization flag
* quantized search test
* are you happy fmt
* refactor test, refactor scorer choosing
* are you happy fmt
* run quantization on segment builder
* decrease testing parameters
* simplify segment
* update version
* remove use_quantization flag
* provide quantization config
* quantization version up
* euclid dist
* add euclid test
* saveload
* fix initialization bugs
* quantization lib version up
* fix arm build
* refactor scorer selecting
* quant lib version up
* are you happy fmt
* are you happy fmt
* are you happy clippy
* add save/load test for simple storage
* add comments
* quantiles
* quantization mmap
* remove f32
* mmap test
* fix mmap slice
* fix mmap test
* use chunks for quantization storage
* fix build
* are you happy fmt
* update quantization library
* update quantization lib
* update quantization lib
* integrate api changes
* are you happy fmt
* change quantization api
* additional checks in tests
* update quantization version
* fix unit tests
* add quantization to storage config
* use quantization for all cardinality search cases
* Integrate quantization suggestions 2 (#1520)
* review api
* wip: refactor quantization integrations
* wip: refactor quantization integrations
* wip: fmt
* include quantization into snapshot
* fmt
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* WIP: Fix `Segment::take_snapshot`
TODO:
- This commit, probably, breaks snapshotting of segments with memmapped vector storage
- `ProxySegment::take_snapshot` seems to potentially similar bug
* WIP: Fix `Segment::take_snapshot`
- Fix snapshotting of `StructPayloadIndex`
- Fix snapshotting of segments with memmapped vector storage
- Temporarily break `ProxySegment::take_snapshot`
* Fix `ProxySegment::take_snapshot`
* Remove `copy_segment_directory` test
* nitpicking
* clippy fixes
* use OperationError::service_error
* Cleanup `TinyMap` trait bounds and derive `Debug`
* Fix `test_snapshot` test
- Derive `Debug` for `NamedVectors`
* Move utility functions from `segment.rs` to `utils` module
* Contextualize `segment::utils::fs::move_all` a bit more carefully
* Fix a typo
* add backward compatibility with old snapshot formats
* fmt
* add snapshot for compatibility test
* git lfs is a piece of shit
* Nitpicking
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* add on disk key
* remove obsolete graph initialization
* remove obsolete max level
* update openapi
* graph links trait
* use mmap option
* same format for ram and mmap
* fix segment unit tests
* are you happy fmt
* are you happy clippy
* fix ci and add mmap test
* review fixes
* remove unused try-from
* fix version compatibility
* avoid loading from disk during conversion
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* use less ram for id tracker
* are you happy clippy
* use vec for internals
* use versions for internal ids
* keys test
* Use less ram for id tracker fixes (#1182)
* WIP: internal_to_version
* fmt
* fix unit tests
* add comment
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* many named vectors per point (segment-level)
* operation result for dim function
* beautifulized vector name
* fix naming bug
* segment version migration
* fmt
* add segment tests
* are you happy clippy
* fix build
* [WIP] many named vectors per point (collection-level) (#975)
* config and search
* fix placeholders for proxy segment move
* remove VectorType from collection
* are you happy fmt
* vectors in grps messages
* create collections with vectors
* segment holder fixes
* are you happy fmt
* remove default vector name placeholders
* are you happy fmt
* are you happy clippy
* fix build
* fix web api
* are you happy clippy
* are you happy fmt
* record vector&vectors
* openapi update
* fix openapi integration tests
* segment builder fix todo
* vector names for update from segment
* remove unwrap
* backward compatibility
* upd openapi
* backward compatible PointStruct
* upd openapi
* fix record back-comp
* fmt
* vector configuration backward compatibility
* fix vetor storage size estimation
* fmt
* multi-vec segment test + index test
* fmt
* api integration tests
* [WIP] Named vectors struct (#1002)
* move to separate file
* named vectors as struct
* use cow
* fix build
* keys iterator
* avoid copy in PointStruct -> get_vectors
* avoid another copy
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* allow additional params for payload field index
* fmt
* wip: full text index building
* fmt
* text search request
* text search request
* full text index persitance and loading
* fmt
* enable fts index in mapping
* clippy
* fix tests + add integration test
* review fixes: extend payload index test
* revert incedental change
* wip: u64 -> u128 + serialization tests
* breaking: use more flexible structure for saving point ids
* replace u64 external id type with enum
* update openapi definitions for uuid + fix retrieve point api + bash script tests
* implement checking stop-flag in the optimization routine
* wip: optimization cancel test
* force optimization stop during the construction of vector index
* fix clippy
* fix clippy warnings
- doc links
- explicit deref
- if instead of match for single bool condition
- combine similar match branches
* revert removal of transmute
* return Some when vec is not empty
* update segment version after completed update only
* more stable updates: check pre-existing points on update, fail recovery, WAL proper ack. check_unprocessed_points WIP
* switch to async channel
* perform update operations in a separate thread (#111)
* perform update operations in a separate thread
* ordered sending update signal
* locate a segment merging versioning bug
* rename id_mapper -> id_tracker
* per-record versioning
* clippy fixes
* cargo fmt
* rm limit of open files
* fail recovery test
* cargo fmt
* wait for worker stops befor dropping the runtime
* [CLIPPY] Fixed the warning for references of the user defined types
* [CLIPPY] Fix module naming issue
* [CLIPPY] Fix the last set of warnings and enable clippy check during CI
* Moved cargo fmt and cargo clippy into it's own action
