* Replace `Option<Segment>` with `enum LoadSegmentOutcome`
* Replace some Path/PathBuf with str/String
* Rename field Segment::{current_path -> segment_path}
* safe_delete
* Add a test for recovery after kill during Partial
* Address AI review
* simplify test
* Merge pull request #7829
* introduce a new state
* switch to ManualRecovery for user-initiated snapshot operations
* fmt
* fix test and regen api
* test fix: instead of force recovery on update failed snapshots now re…
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* move QdrantInternalService to its own file
* implement internal telemetry endpoint
cleaner diff on src/main.rs
upd response in proto
* use try_from conversion
* rename request/response in proto
* Use RwLock for pending changes in MmapBitSliceBufferedUpdateWrapper
* Use RwLock for pending operations in DatabaseColumnScheduledDeleteWrapper
* Use RwLock for pending updates in DatabaseColumnScheduledUpdateWrapper
* Drop alive guard before reconciliation, we don't touch files after
* Remove redundant clone
* Update comments
* in MmapSliceBufferedUpdateWrapper
* in MmapBitsliceBufferedUpdateWrapper
* in MutableIdTracker's versions updates
* in MutableIdTracker's mapping updates
* clone updates only when non-empty
* only lock for reconciling pending changes
* simpler reconciling
* use Mutex as argument to ensure we only lock within reconciliation
* Add test to reproduce broken WAL delta after aborting stream records
* Add staging env var to slow down stream records transfers for test
* Tweak test formatting and utilities a bit
* Add comment to test, link to PR describing bug
* Update test so it still succeeds with patched behavior
* Fix broken WAL delta after stream records abort (#7791)
* Make set_replica_state async
* Add function called when active state of local replica changes
* Add snapshot for newest clocks
* Bump newest clocks snapshot on replica deactivation
* Use newest clocks snapshot during recovery
* Add enum for specifying whether to take or clear clocks snapshot
* Store clock snapshot inside clock map, removing extra file
This greatly simplifies state handling. It also prevent any kind of
desynchronization because all newest clocks are always persisted
atomically.
* Immediately persist clocks after taking snapshot
* Always update snapshot, only take if missing
* Take clock snapshots through each shard flavor, including proxies
* Propagate dedicated functions for taking and clearing clocks snapshot
* Only persist clocks immediately if changed on snapshot/clear
* Simplify recovery point logic, always take clocks snapshot if exists
* Remove unwrap
* Fix typo
* Fix doc comment
* Transfer driver is async, use Tokio sleep
* Reduce visibility
* easy fix with version check
* Link to PR
* Fix get transfers with source/target pairs, also fix dead replica aborts
* Use new source/target functions elsewhere
* Flip shard and peer arguments
* Fix incorrect check, from should be to
* Inline more format arguments
---------
Co-authored-by: timvisee <tim@visee.me>
* Enrich Gridstore model testing
* better
* no clear on Windows
* less windows
* sigh
* change strategy
* never give up
* x100 scale down for Windows
* last touch
* decrease for others too
* remove Iter test - the quadratic cost is not worth it
* Revert "remove Iter test - the quadratic cost is not worth it"
This reverts commit f0e98cb33b.
* try with limited Iter
* less logging as it is hurting crasher
* tracker.mapping_len() is slow
* hard limit good enough for now
* Implement `__repr__` for `PyJsonPath`
* Implement `__repr__` for `PyFilter`
* Implement `__repr__` for `PyQuery`
* Implement `__repr__` for `PyQueryRequest`
* Implement `__repr__` for `PySearchRequest`
* Implement `__repr__` for `PyFilter` using `pyclass_repr` attribute
* Implement `__repr__` for `PyQuery` using `pyclass_repr` attribute
* Implement `__repr__` for `PyQueryRequest` using `pyclass_repr` attribute
* Implement `__repr__` for `PySearchRequest` using `pyclass_repr` attribute
* Refactor `PyUpdateOperation` constructors
* Add default parameters to `PyVectorDataConfig::new`
* Add `Repr` trait and `WriteExt` helper
* Implement `__repr__` for config types
* fixup! Implement `__repr__` for config types
Use `Copy` instead of `Clone`
* fixup! Implement `__repr__` for config types
Add basic test
* Implement `__repr__` for `PyPointId`
* Implement `__repr__` for `PyVector`
* Implement `__repr__` for `PyVectorInternal`
* Implement `__repr__` for `PyPayload`
* Implement `__repr__` for `PyValue`
* Implement `__repr__` for `PyPoint`
* Implement `__repr__` for `PyPointVectors`
* Implement `__repr__` for `PyRecord`
* Move `PyScoredPoint` into a separate file
* Implement `__repr__` for `PyScoredPoint`
* Cleanup examples
* fixup! Implement `__repr__` for `PyScoredPoint`
* Move `PyOrderValue` into separate file
* Add `PyScoredPoint::order_value`
* Implement `pyclass_repr` proc-macro attribute
* Implement `__repr__` for config types using `pyclass_repr` attribute
* Implement `__repr__` for `PySparseVector` using `pyclass_repr` attribute
* Implement `__repr__` for `PyPoint` using `pyclass_repr` attribute
* Implement `__repr__` for `PyPointVectors` using `pyclass_repr` attribute
* Implement `__repr__` for `PyRecord` using `pyclass_repr` attribute
* Implement `__repr__` for `PyScoredPoint` using `pyclass_repr` attribute
* Minor fixes and cleanups
* fixup! Minor fixes and cleanups
* rollback copy for quantization config
* rollback copy for quantization config
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* introduce IsAliveLock
* add tests
* rename to mark_dead
* update gridstore comment
* Update some comments
* Add must_use attribute to lock_if_alive
* Rename mark_dead to blocking_mark_dead
* make the handle take a `Weak` reference
* make dropping explicit
* use `Mutex::lock_arc` instead
* less nesting
* clippy
---------
Co-authored-by: timvisee <tim@visee.me>
* Separate set and unset lists for each Gridstore pointer update
This makes the list of pointer updates much easier to grasp. The
simplification is desired because this exact structure has been a cause
for bugs multiple times now.
* Add more aggressive debug assertions
* Use consistent terminology
* Correct removal of set
* Simplify drain function
* Add safe guard to ignore empty pointer updates
* Rework tracker getter, explicitly branch variants
* Patch transmute, require Sized type
* rename and adjust comments
- set/unset is now current/to_free
- adjusted descriptions and comments for this nomenclature too
* clippy
* clippyyy
not 4, not 2, but 3 spaces
---------
Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
