mirror of
https://github.com/qdrant/qdrant.git
synced 2026-08-06 18:10:58 -05:00
7edf599d73
* Exclude deleted vectors from HNSW graph building stage * When estimating query cardinality, use available points as baseline We should not use the total number of points in a segment, because a portion of it may be soft deleted. Instead, we use the available (non-deleted) points as baseline. * Add plain search check to unfiltered HNSW search due to deleted points * Cardinality sampling on available points, ignore deleted named vectors * Estimate available vectors in query planner, now consider deleted points In the query planner, we want to know the number of available points as accurately as possible. This isn't possible because we only know the number of deletions and vectors can be deleted in two places: as point or as vector. These deletions may overlap. This now estimates the number of deleted vectors based on the segment state. It assumes that point and vector deletions have an overlap of 20%. This is an arbitrary percentage, but reflects an almost-worst scenario. This improves because the number of deleted points wasn't considered at all before. * Remove unused function from trait * Fix bench compilation error * Fix typo in docs * Base whether to do plain search in HNSW upon full scan threshold * Remove index threshold from HNSW config, only use full scan threshold * Simplify timer aggregator assignment in HNSW search * Remove vector storage type from cardinality function parameters * Propagate point deletes to all its vectors * Check for deleted vectors first, this makes early return possible Since point deletes are now propagated to vectors, deleted points are included in vector deletions. Because of that we can check if the vector is deleted first so we can return early and skip the point deletion check. For integrity we also check if the point is deleted, if the vector was not. That is because it may happen that point deletions are not properly propagated to vectors. * Don't use arbitrary vector count estimation, use vector count directly Before we had to estimate the number of vectors (for a named vector) because vectors could be deleted as point or vector. Point deletes are now propagated to vector deletes, that means we can simply use the deleted vector count which is now much more accurate. * When sampling IDs, check deleted vecs before deleted points * On segment consistency check, delete vectors for deleted points * Fix vector delete state not being kept when updating storage from other * Fix segment builder skipping deleted vectors breaking offsets * update segment to handle optional vectors + add test (#1781) * update segment to handle optional vectors + add test * Only update stored record when deleting if it wasn't deleted already * Reformat comment --------- Co-authored-by: timvisee <tim@visee.me> * Fix missed vector name test, these are now marked as deleted * upd test * upd test * Update consensus test --------- Co-authored-by: Andrey Vasnetsov <andrey@vasnetsov.com>