* Use `IdTrackerEnum` type instead of `dyn IdTracker`
It would allow to be more flexible on the IdTracker trait, making it
dyn-incompatible eventually.
Coauthored with Claude Code.
* Review fixes
* Split `SegmentEntry`
Add `ImmutableSegmentEntry` for operations that can be applied to
immutable segments, and make the `SegmentEntry` as it subtrait.
* Rename to `NonAppendableSegmentEntry`
It differs semantically from `ImmutableSegmentEntry` by allowing point
deletion. Move point deletion to the trait too.
* Fix docstring
* use NonAppendableSegmentEntry where possible
---------
Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>
* feat: Add enable_hnsw option for payload field indexes
Add optional enable_hnsw parameter to all payload index types to control
whether additional HNSW graph links are built for each indexed field.
- Add enable_hnsw field to all 8 payload index param types
- Update gRPC proto definitions and conversions
- Update OpenAPI schema
- Modify HNSW graph builder to respect enable_hnsw flag
- Add enable_hnsw() helper methods to PayloadSchemaParams and PayloadFieldSchema
- Update all tests to include new field (default: None)
When enable_hnsw is true and payload_M > 0, additional HNSW links will
be built for the payload field. Default value is true for backward compatibility.
* Fix Some format problems
* fix: address comment problem
---------
Co-authored-by: EC2 Default User <ec2-user@ip-10-78-171-148.ec2.internal>
* Add progress_tracker.rs
* Pass progress tracker around
* Populate progress tracker with actual data
* Expose progress on `/collections/{name}/optimizations` endpoint
* add stopflag check into query_points of payload index
* propagate stopflag more
* fmt
* bench: compare better against no atomic
* use stoppable iter with IteratorExt
* clippy
* Add cancelled error helper
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: timvisee <tim@visee.me>
* Enable Gridstore payload index storage, rename storage types
* Respect payload_index_skip_mutable_rocksdb feature flag
* Minor cleanup
* Fix map payload index Gridstore semantics, only create files if desired
The semantics of backing storages in payload indices are inconsistent.
This aligns the Gridstore backing storage to our RocksDB backing
storage, matching the create/open/load behavior. Now, it will only
create Gridstore files on disk if we request it to do so.
This commit only implements it on the map index. In following comments
the same will be implemented in all other payload indices too.
* Apply same Gridstore semantics to other payload index types
* Rename create parameter to create_if_missing
* Fix bench compilation errors
* Use sparse vector storage fixture without RocksDB
* Fix some tests
* Replace RocksDB sparse storage with Gristore in mutable text index tests
* Use in-memory vector storage in multivector HNSW test
* Flag many more RocksDB specifics in tests
* Use database placeholder type without RocksDB flag
* Flag more tests
* Flag even more tests
* Fix imports, fix typo, and repair base test build
* Initialize dummy database if RocksDB flag is disabled
* Assert correct storage types
* Don't use old vector storage type when RocksDB is disabled
* Expos default for vector storage type only in tests
* Only expose simple segment constructor in tests
* Don't derive Default
* Fix inverted appendable flag
* take max_point_offset into consideration
* add null index even if index selector is for rocksdb
* fmt
* better cardinality estimation, check deleted inside filtered iter
* fix count indexed points, add primary conditions to estimator
* use len for count_indexed_points, adjust test
* remove deleted check from filtered iter
* cleaner removal
* Cardinality estimation measurements
* Apply hw measurements to latest changes from dev
* Clippy
* Also measure cardinality estimation for geo index
* Make measured units 'bytes'
* Use PointOffsetType instead of u32 for size calculation
* fix memory cost for check_values_any in mmap index
* fix double counting for value reading in mmap, remove hw_counter from mmap hashmap
* fmt
* fix hw measurement for text index
* Remove non necessary lifetime annotations
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* create initial strucutres
* clippy
* start field-query refactoring
* start field-query refactoring (2/N)
* start field-query refactoring (3/N): duplicate is_empty/null condiftions as field condition
* start field-query refactoring (4/N): re-instate is_empty fallback in case new index is not built yet
* filter for is_empty/is_null
* implement add/remove point
* upd schema
* open and create of null-index
* create null-index
* fix test
* Update lib/segment/src/index/query_optimization/condition_converter.rs
Co-authored-by: Tim Visée <tim+github@visee.me>
* unit test for null-index
* more unit tests
* add openapi tests
* fmt
* fix for integartion tests
* rabbit review fix
* make [null] non-empty
---------
Co-authored-by: Tim Visée <tim+github@visee.me>
* Bump Rust edition to 2024
* gen is a reserved keyword now
* Remove ref mut on references
* Mark extern C as unsafe
* Wrap unsafe function bodies in unsafe block
* Geo hash implements Copy, don't reference but pass by value instead
* Replace secluded self import with parent
* Update execute_cluster_read_operation with new match semantics
* Fix lifetime issue
* Replace map_or with is_none_or
* set_var is unsafe now
* Reformat
* rename cpu_budget -> resource_budget
* clippy
* add io budget to resources
* fmt
* move budget structures into a separate file
* add extend permit function
* dont extend existing permit
* switch from IO to CPU permit
* do not release resource before aquiring an extension
* fmt
* Review remarks
* Improve resource permit number assertion
* Make resource permit replace_with only acquire extra needed permits
* Remove obsolete drop implementation
* allocate IO budget same as CPU
* review fixes
---------
Co-authored-by: timvisee <tim@visee.me>
* bump and migrate to rand 0.9.0
also bump rand_distr to 0.5.0 to match it
* Migrate AVX2 and SSE implementations
* Remove unused thread_rng placeholders
* More random migrations
* Migrate GPU tests
* bump seed
---------
Co-authored-by: timvisee <tim@visee.me>
Co-authored-by: Arnaud Gourlay <arnaud.gourlay@gmail.com>
* Measure read io for payload storage
* Add Hardware Counter to update functions
* Fix tests and benches
* Rename (some) *_measured functions back to original
* map index: fix reachable code marked as unreachable
* plumber work to get mmap text index to interfaces
* test: add fixture for mmap text index, always create mmap segment
* various fixes
- ensure dir is created for mmap
- implement is_on_disk() for text index
- invert deleted condition for filter in mmap inverted index
* update grpc docs and openapi
* implement return of files
* review nit
* fix after rebase
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move FacetIndex into facet_index.rs
* add support for integer facets
* add support for uuid facets
* use separate internal structure
* rename FacetValue::Keyword into FacetValue::String in REST
* fix after rebase
* rename keyword -> string for internal data structures
* implement MatchAny and expect for filter on UUID mmap index
* implement MatchAny and expect for cardinality estimation on UUID mmap index
* refactor PayloadFieldIndex filter for handling incorrect empty query parsing case
* rename to FacetRequestInternal
* add rest endpoint
* fix correctness by fetching the whole list of values
* fix mmap map index variant
Also removes test for sorted output, for now
* add ytt spec
* fix clippy
* use hashmap inside of local shard
* rename operation to `facet`, add access test
* whitelist endpoint
* change api
* make limit optional
* refactor: let SegmentBuilder::update take unlocked segments
* style: split long lines
* refactor: introduce TestSegments
* test: add tests for mmap indices
* faceting in segment
* Add segment integration test
* nits
* count from filtered stream, not value->points map directly
* drop AtomicRef from fn signature
* count only unique values per point
* use entry in hashmap
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* sorting
* migrate tests and move logic into SegmentBuilder
* add test and improve implementation
* improve code
* review
* code review improvements
* add index building to test
* Do not clone ranges
* Resolve clippy warnings due to recent PR on dev
* review suggestions
* Defragmentation in api (#4684)
* add tenant config to api
* deduplicate used defragmentation keys
* rename is_tenant to is_primary
* use all values to defrag key
* rename is_primary -> is_tenant
* update schema
---------
Co-authored-by: generall <andrey@vasnetsov.com>
Co-authored-by: timvisee <tim@visee.me>
* drop some code
* Drop JsonPathString
* Fix test_remove_key
Drop failing tests:
- Deleting array indices is not idempotent, so we don't support it.
- Empty JSONPath is not supported.
* Make json_path::path() non-generic
* Remove references to JsonPathV2
* Drop JsonPathInterface
* Move json_path::v2 code into json_path
* Drop validate_not_empty
* Drop JsonPath::head() as being unused
* Replace path() with JsonPath::new()
* Restore comments
* Move tests to json_path
* Use json() consistently in tests
* Replace many into calls with Into trait
---------
Co-authored-by: timvisee <tim@visee.me>
