* add benches for large MatchAny
* use HashSet for MatchAny
* use fnv hash
* make fnv workspace level dependency; apply clippy
* remove SmolStr from Keyword; Improve performance
* add bench for small number of keywords
* fix openapi
* fix performance issue
* apply integer optimization; create magic number constant
* Move CPU count function to common, fix wrong CPU count in visited list
* Change default number of rayon threads to 8
* Use CPU budget and CPU permits for optimizer tasks to limit utilization
* Respect configured thread limits, use new sane defaults in config
* Fix spelling issues
* Fix test compilation error
* Improve breaking if there is no CPU budget
* Block optimizations until CPU budget, fix potentially getting stuck
Our optimization worker now blocks until CPU budget is available to
perform the task.
Fix potential issue where optimization worker could get stuck. This
would happen if no optimization task is started because there's no
available CPU budget. This ensures the worker is woken up again to
retry.
* Utilize n-1 CPUs with optimization tasks
* Better handle situations where CPU budget is drained
* Dynamically scale rayon CPU count based on CPU size
* Fix incorrect default for max_indexing_threads conversion
* Respect max_indexing_threads for collection
* Make max_indexing_threads optional, use none to set no limit
* Update property documentation and comments
* Property max_optimization_threads is per shard, not per collection
* If we reached shard optimization limit, skip further checks
* Add remaining TODOs
* Fix spelling mistake
* Align gRPC comment blocks
* Fix compilation errors since last rebase
* Make tests aware of CPU budget
* Use new CPU budget calculation function everywhere
* Make CPU budget configurable in settings, move static budget to common
* Do not use static CPU budget, instance it and pass it through
* Update CPU budget description
* Move heuristic into defaults
* Fix spelling issues
* Move cpu_budget property to a better place
* Move some things around
* Minor review improvements
* Use range match statement for CPU count heuristics
* Systems with 1 or 2 CPUs do not keep cores unallocated by default
* Fix compilation errors since last rebase
* Update lib/segment/src/types.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Update lib/storage/src/content_manager/toc/transfer.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename cpu_budget to optimizer_cpu_budget
* Update OpenAPI specification
* Require at least half of the desired CPUs for optimizers
This prevents running optimizations with just one CPU, which could be
very slow.
* Don't use wildcard in CPU heuristic match statements
* Rename cpu_budget setting to optimizer_cpu_budget
* Update CPU budget comments
* Spell acquire correctly
* Change if-else into match
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
* Rename max_rayon_threads to num_rayon_threads, add explanation
* Explain limit in update handler
* Remove numbers for automatic selection of indexing threads
* Inline max_workers variable
* Remove CPU budget from ShardTransferConsensus trait, it is in collection
* small allow(dead_code) => cfg(test)
* Remove now obsolete lazy_static
* Fix incorrect CPU calculation in CPU saturation test
* Make waiting for CPU budget async, don't block current thread
* Prevent deadlock on optimizer signal channel
Do not block the optimization worker task anymore to wait for CPU budget
to be available. That prevents our optimizer signal channel from being
drained, blocking incoming updates because the cannot send another
optimizer signal. Now, prevent blocking this task all together and
retrigger the optimizers separately when CPU budget is available again.
* Fix incorrect CPU calculation in optimization cancel test
* Rename CPU budget wait function to notify
* Detach API changes from CPU saturation internals
This allows us to merge into a patch version of Qdrant. We can
reintroduce the API changes in the upcoming minor release to make all of
it fully functional.
---------
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* Merge serde attributes
* Remove obsolete conversion
* Add integer type with parameters
* Make integer lookup and range parameters non-optional
* Add parameterized integer index types test
Co-authored-by: Di Zhao <diz@twitter.com>
* Cleanup
---------
Co-authored-by: Di Zhao <diz@twitter.com>
* sparse vectors with large indices support
* fix unit tests
* add tracker file to snapshot
* fix unit tests
* are you happy clippy
* fix config loading
* test with max value
* fix python tests
* fix consensus tests
* review remarks; acc test for inverted index
* remove quantization update from hnsw index
* test that hnsw was builded with presented quantization
* are you happy codespell
* prefer method over public field
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* internal sparse vector segment configs
update openapi
fix build
provide correct index type to fixtures
internal sparse vector segment configs
update openapi
fix build
load ram sparse index from file
fix spelling
add test
spelling error
use appendable flag to decide load or not
fix conflict
dont flush mmap when file is empty
* fix build
* set list of files for snapshot
* fix snapshots test
* fix codespell
* simple save-load unit test
* refactor InvertedIndex::files
* fix unit tests
* add debug_assert
* test all persistence loading cases
* sparse vectors query scorer
plain search test without internals
fix sparse_vector_index_plain_search
fix async scorer build
fix unit tests
are you happy fmt
sparse index with full query support
fix benches, use exact flag to force plain search
add alias and comment to exact flag
fix tests
recommendations TODOs fix
add simple test
add todo
refactor index
test vs comparison dense and sparse discovery
are you happy fmt
propogate error from avg negatives
reuse filtered points list
rollback api changes
better discovery test
are you happy fmt
rollback openapi
fix build
fix tests
review remark recommendations
review remarks
review remarks
codespell
integrate plain search
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
fix comment of plain search
Update lib/collection/src/recommendations.rs
Co-authored-by: Luis Cossío <luis.cossio@qdrant.com>
fix tests
* simplify search query function
* Estimate sparse vector dimension for optimizer (#3142)
* Estimate sparse vector dimension for optimizer
* add hack explanation + todo
Fix optimizer change sparse index on disk
fix index to disk
is appendable fix
add index files if necessary
fix test
* fix tests
---------
Co-authored-by: Ivan Pleshkov <pleshkov.ivan@gmail.com>
* SparseVector implements VectorIndex
* dedicated telemetry
* conflict
* easy code reviews
* simplify tracking indexed points count
* move telemetry conversion to sparse file
* move max_result_count out of inverted index trait
* unify sparse vector fixtures
* simpler conversion
* add todo regarding OOM potential
* reuse check deleted from raw scorer with TODO
* change new to open to handle mmap index
* Extend GeoPolygon to support interiors (#2315)
Per GeoJson, we should support polygon with exterior and interiors (holes on the surface) in Geo Filter by Polygon(#795). This commit extend current GeoPolygon filter to accept interiors. It includes:
1. changes to proto and internal GeoPolygon struct, and validation fn
2. add and refactor some tests
3. add integration test
* add gRPC geo_polygon validation
---------
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Reduce flakiness of `test_building_cancellation` with 5k base segment
* use fixed acceptable stopping delay, and relative cancelling delay
* Remove debug statement
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
* add enum for vector query on segment search
* rename newly introduced types
* fix: handle QueryVector on async scorer
* handle QueryVector in QuantizedVectors impl
* fix async scorer test after refactor
* rebase + refactor on queue_proxy_shard.rs
* constrain refactor propagation to segment_searcher
* fmt
* fix after rebase
* immutable map index integration
* remove wipe
* fix unit tests
* get appendable flag from config
* minor refactoring
* fix chunked mmap appendable flag
---------
Co-authored-by: generall <andrey@vasnetsov.com>
