upstream-archive/qdrant
1
0
Fork 0
mirror of https://github.com/qdrant/qdrant.git synced 2026-07-31 15:11:35 -05:00
Code Issues Packages Projects Releases Wiki Activity
3,332 Commits 343 Branches 116 Tags
b94d02ad30d1d2c2065b0d64c94de9d7d7560879
Commit Graph

142 Commits

Author SHA1 Message Date
Andrey Vasnetsov
e76132b6e7 enable sparse snapshots (#5183) 2024-10-06 17:50:54 +02:00
xzfc
7932286709 Update tar_ext::BuilderExt (#5112) 
* tar_ext::BuilderExt: generalize and add drop workaround

- Generalize: instead of taking `std::fs::File`, take
  `impl Write + Seek`.
- Add a workaround for drop: `tar::Builder<W>::drop` might panic if the
  underlying writer panics when using in async context.

* Drop unnecessary write_fn usage

* tar_ext::BuilderExt: support both seekable and streaming variants

* tar_ext::BuilderExt: support both owned/borrowed variants
 2024-10-04 14:10:09 +00:00
dependabot[bot]
86de9fadc2 build(deps): bump flate2 from 1.0.33 to 1.0.34 (#5157) 
Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.33 to 1.0.34.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.33...1.0.34)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-10-01 10:42:16 +02:00
dependabot[bot]
52db80a292 build(deps): bump ordered-float from 4.2.2 to 4.3.0 (#5154) 
Bumps [ordered-float](https://github.com/reem/rust-ordered-float) from 4.2.2 to 4.3.0.
- [Release notes](https://github.com/reem/rust-ordered-float/releases)
- [Commits](https://github.com/reem/rust-ordered-float/compare/v4.2.2...v4.3.0)

---
updated-dependencies:
- dependency-name: ordered-float
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-10-01 09:19:46 +02:00
Arnaud Gourlay
d527ab9cd0 Promote atomicwrites to workspace (#5135) 2024-09-24 09:44:12 +02:00
dependabot[bot]
f6246b0e44 build(deps): bump atomicwrites from 0.4.3 to 0.4.4 (#5130) 
Bumps [atomicwrites](https://github.com/untitaker/rust-atomicwrites) from 0.4.3 to 0.4.4.
- [Commits](https://github.com/untitaker/rust-atomicwrites/compare/0.4.3...0.4.4)

---
updated-dependencies:
- dependency-name: atomicwrites
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-09-23 21:24:50 +02:00
Tim Visée
e804720551 Bump dev version to 1.11.6-dev (#5122) 2024-09-23 17:38:01 +02:00
xzfc
e7c445b15e Do not return error if madvise(PopulateRead) fails (#5123) 2024-09-23 17:08:41 +02:00
Tim Visée
bcd9bfd5ac Bump dev version to 1.11.5-dev (#5099) 2024-09-17 17:06:23 +02:00
dependabot[bot]
6c8da1669f build(deps): bump anyhow from 1.0.87 to 1.0.89 (#5094) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.87 to 1.0.89.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.87...1.0.89)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-09-17 07:29:52 +02:00
Andrey Vasnetsov
3bb6628bee mmap_to_type_unbounded -> mmap_prefix_to_type_unbounded (#5088) 2024-09-16 18:42:43 +02:00
Vamshi Maskuri
e0ab73fccb fix oom issues and add logs to monitor memory usage under high contention (#4994) 
* fix oom issues and add logs to monitor memory usage under high contention

---------

Signed-off-by: Vamshi Maskuri <117595548+varshith257@users.noreply.github.com>
 2024-09-15 00:06:00 +02:00
Andrey Vasnetsov
0585b70ba6 disable sparse tar arcive creation for backward compatibility (#5081) 2024-09-14 23:33:00 +02:00
Andrey Vasnetsov
e8d5cfc63c Allow explicit populate of mmap (#4923) 
* expose mmap populate

* expose mmap populate in open_read_mmap

* FOR TEST, REVERSE IT: make InRamChunkedMmap default

* enable populate advise on unix

* fix clippy

* unix -> linux

* Update lib/collection/src/config.rs

* clippy fixes

* resolve conflicts

* fmt

* Runtime check for PopulateRead

---------

Co-authored-by: xzfc <xzfcpw@gmail.com>
 2024-09-14 20:53:07 +02:00
xzfc
a6400138f8 Direct snapshot creation (#5061) 2024-09-12 19:39:15 +00:00
Andrey Vasnetsov
b342cbca6e initial structures for reading mmap text index (#5029) 
* initial structures for reading mmap text index

* creation of text index mmap

* WIP create mmap inverted index

* use same file writing technique as in MmapHashmap

* Save points_to_tokens_count in two files

* clippy

* fill slice with iterator

* review fixes

* fix after review

* implement loading of MmapInvertedIndex

* WIP: prepare for converting ChunkReader into trait

* fmt

* WIP: prepare ChunkReader to be a trai

* WIP: replace ChunkReaderImpl with trait

* implement chunk reader for mmap view

* rollback ChunkReader trait and remove CompressedMmapPostingListView

* move ImmutableInvertedIndex and MutableInvertedIndex into separate files

* impl ops for MmapInvertedIndex

* make mmap hashmap value param generic (#5042)

* make mmap hashmap value param generic

* test + fixes

* Update lib/common/common/src/mmap_hashmap.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* Update lib/common/common/src/mmap_hashmap.rs

Co-authored-by: Tim Visée <tim+github@visee.me>

* fixups

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>

* test and fix immutable to mmap conversion

* fix case 4 and refactor MmapBitSlice::create

* more tests

* fmt

---------

Co-authored-by: Luis Cossío <luis.cossio@outlook.com>
Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: xzfc <xzfcpw@gmail.com>
 2024-09-10 12:02:05 +02:00
dependabot[bot]
21e96329fc build(deps): bump anyhow from 1.0.86 to 1.0.87 (#5047) 
Bumps [anyhow](https://github.com/dtolnay/anyhow) from 1.0.86 to 1.0.87.
- [Release notes](https://github.com/dtolnay/anyhow/releases)
- [Commits](https://github.com/dtolnay/anyhow/compare/1.0.86...1.0.87)

---
updated-dependencies:
- dependency-name: anyhow
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-09-10 07:45:06 +02:00
dependabot[bot]
b6d528ccb0 build(deps): bump dashmap from 6.0.1 to 6.1.0 (#5055) 
Bumps [dashmap](https://github.com/xacrimon/dashmap) from 6.0.1 to 6.1.0.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/compare/v6.0.1...v6.1.0)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-09-10 07:21:21 +02:00
Andrey Vasnetsov
7447fff238 debug asserts for malformed hashmmap (#5030) 
* debug asserts for malformed hashmmap

* Apply suggestions from code review

* Reformat using cargo fmt

---------

Co-authored-by: Tim Visée <tim+github@visee.me>
Co-authored-by: timvisee <tim@visee.me>
 2024-09-09 10:46:06 +02:00
Tim Visée
8b0aa6f549 Allow shard transfer from/to the same peer if from/to shard ID is different (#5034) 2024-09-09 09:15:42 +02:00
Tim Visée
6f8dd8176a Bump dev version to 1.11.4-dev (#4986) 2024-08-29 17:07:04 +02:00
Tim Visée
2eff2c7227 Bump dev version to 1.11.3-dev (#4975) 2024-08-28 16:14:02 +02:00
Tim Visée
6f11e748c3 Bump dev version to 1.11.2-dev (#4965) 2024-08-27 11:47:56 +02:00
dependabot[bot]
5581e0b588 build(deps): bump flate2 from 1.0.31 to 1.0.33 (#4956) 
Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.31 to 1.0.33.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Changelog](https://github.com/rust-lang/flate2-rs/blob/main/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/flate2-rs/compare/1.0.31...1.0.33)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-08-27 10:52:55 +02:00
Arnaud Gourlay
c8844388d7 Update Validator 0.18 (#4894) 
* Update Validator 0.18

* fix new test error message

* clearer geo error message

* fix error message

* fix unit tests

* Put spaces back

---------

Co-authored-by: timvisee <tim@visee.me>
 2024-08-15 15:22:51 +02:00
Andrey Vasnetsov
707134b43e Refactor chunked mmaps (#4888) 
* remove dependency on segment in chunked mmap

* move chunked utils to memory subcrate
 2024-08-15 12:04:39 +02:00
xzfc
0e6f86bdfb InRamChunkedMmap: optimize using madvise (#4881) 
* refactor: introduce AdviceSetting for open_read_mmap/open_write_mmap

* perf: use Advice::Normal in a few places

* perf: use MADV_DONTNEED after reading

* perf: use POSIX_FADV_DONTNEED after reading
 2024-08-14 13:41:37 +00:00
Andrey Vasnetsov
44130a462e Mmap subcrate refactoring (#4886) 
* make mmap_type independent from segment structures

* make bitvec and thiserror workspace dependencies

* move mmap_type into common/memory subcrate

* fmt
 2024-08-14 11:04:56 +02:00
Tim Visée
537bc70fc1 Bump dev version to 1.11.1-dev (#4880) 2024-08-13 12:41:14 +02:00
Jojii
2cb03610d8 On-disk Uuid hybrid-index (#4825) 
* Migrate uuid index to mmap

* add uuid numeric index

* select correct index + tenants

* Only serialize if necessary

* select correct index

* reset Cargo.toml

* review: use only map index for uuid for now

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-08-07 19:45:42 +02:00
Andrey Vasnetsov
e77b6332dd Use mmap lock as default vector storage (#4828) 
* add force_ram parameter to chuncked mmap vector storage

* enable mlocked mmap vector storage on unix by default

* regen openapi

* add mlock on creation of chunck

* minor unrelated renaming

* rollback changes in LockedChunkedMmap

* fmt

* make AppendableMmapDenseVectorStorage generic of storage type

* make AppendableMmapMultiDenseVectorStorage generic of storage type

* implement initialization of InRamChunkedMmap

* implement MultiDenseAppendableInRam and variations

* enable InRamChunkedMmap for multivectors

* use same CHUNK_SIZE for mmap and regular chuncked vectors

* enable InRamChunkedMmap by default

* fix tests

* rollback usage of InRamChunkedMmap by default

* review changes

* add assertion on chunk_capacity [skip-ci]
 2024-08-07 16:34:58 +02:00
dependabot[bot]
9b195bd6fc build(deps): bump flate2 from 1.0.30 to 1.0.31 (#4835) 
Bumps [flate2](https://github.com/rust-lang/flate2-rs) from 1.0.30 to 1.0.31.
- [Release notes](https://github.com/rust-lang/flate2-rs/releases)
- [Commits](https://github.com/rust-lang/flate2-rs/commits)

---
updated-dependencies:
- dependency-name: flate2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-08-06 09:28:41 +02:00
Luis Cossío
ba43d16880 Support timeout in Facets (#4792) 
* nits in segments_searcher

* implement timeout into segment faceting

* Add timeout to internal service api

* refactor iterator_ext, and add test
 2024-08-02 12:57:20 -04:00
Ivan Pleshkov
3a610d97f6 Mmap map index (#4779) 
* define mmap map index

add point to values

add mmap hash map

are you happy fmt

use new mmap hashmap methods

build index

saturating_sub

are you happy clippy

fix tests build

* integrate facets for mmap index

* mmap tests

* fix ci

* review remarks

* review remarks
 2024-08-02 13:01:19 +02:00
Ivan Pleshkov
a4c4c0e02c Mmap hashmap iterate over keys (#4764) 
* mmap hashmap iterate over keys

* fix tests

* Use associated consts rather than static methods

---------

Co-authored-by: Albert Safin <xzfcpw@gmail.com>
 2024-07-29 18:24:26 +02:00
dependabot[bot]
849f6499ff build(deps): bump thiserror from 1.0.62 to 1.0.63 (#4730) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.62 to 1.0.63.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.62...1.0.63)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-23 08:39:51 +02:00
xzfc
a0ea3caccf Enable some of the pedantic clippy lints (#4715) 
* Use workspace lints

* Enable lint: manual_let_else

* Enable lint: enum_glob_use

* Enable lint: filter_map_next

* Enable lint: ref_as_ptr

* Enable lint: ref_option_ref

* Enable lint: manual_is_variant_and

* Enable lint: flat_map_option

* Enable lint: inefficient_to_string

* Enable lint: implicit_clone

* Enable lint: inconsistent_struct_constructor

* Enable lint: unnecessary_wraps

* Enable lint: needless_continue

* Enable lint: unused_self

* Enable lint: from_iter_instead_of_collect

* Enable lint: uninlined_format_args

* Enable lint: doc_link_with_quotes

* Enable lint: needless_raw_string_hashes

* Enable lint: used_underscore_binding

* Enable lint: ptr_as_ptr

* Enable lint: explicit_into_iter_loop

* Enable lint: cast_lossless
 2024-07-22 08:19:19 +00:00
xzfc
cf815d0a9f Fixup for MmapHashMap (#4718) 2024-07-22 08:15:46 +00:00
dependabot[bot]
e353ce169f build(deps): bump thiserror from 1.0.61 to 1.0.62 (#4667) 
Bumps [thiserror](https://github.com/dtolnay/thiserror) from 1.0.61 to 1.0.62.
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/1.0.61...1.0.62)

---
updated-dependencies:
- dependency-name: thiserror
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-07-16 12:15:43 +02:00
xzfc
643532db2b Update MmapHashMap (#4648) 
* Update MmapHashMap

* review: use PointOffsetType instead of u32

* Don't scare muggles

* Improve comments

* Review fixes

---------

Co-authored-by: generall <andrey@vasnetsov.com>
 2024-07-15 19:52:30 +00:00
Tim Visée
a1ab6d36d2 Bump dev version to 1.10.2-dev (#4660) 2024-07-11 16:41:37 +02:00
xzfc
90ce6cbd01 Add MmapHashMap (#4612) 
* Promote some deps to workspace deps

* Add MmapHashMap

* Review fixups

* Use tempfile for atomicity
 2024-07-08 17:44:52 +00:00
Arnaud Gourlay
e1b2af1a86 Promote log dep to workspace (#4605) 2024-07-03 12:35:04 +02:00
Tim Visée
737f0c5f5c Bump dev version to 1.10.1-dev (#4603) 2024-07-03 09:50:56 +02:00
tellet-q
c88f10c19f Add test for OOD during indexing (#4267) 
* Add test for OOD during indexing

* Start indexing right after 1st OOD message

* Only send search request after insert loop

* Fail early when encountering out-of-storage during optimization (#4578)

* fs4@0.8.4

* fail early on low storage

* move `dir_size` to `common`

* move the ood bailout to `SegmentOptimizer::optimized_segment_builder`

* drop dead code

* move dir_size to common subcrate

---------

Co-authored-by: generall <andrey@vasnetsov.com>

---------

Co-authored-by: xhjkl <xhjkl@users.noreply.github.com>
Co-authored-by: generall <andrey@vasnetsov.com>
 2024-06-29 22:40:18 +02:00
Ivan Pleshkov
291add4d37 Atomic create_and_ensure_length (#4583) 
* atomic create_and_ensure_length

* fix tests; review remarks

* review remarks

* create false flag
 2024-06-29 00:50:54 +02:00
Tim Visée
1c38b0849d Bump dev version to 1.9.8-dev (#4559) 2024-06-26 08:56:50 +02:00
dependabot[bot]
3f29115f99 build(deps): bump lazy_static from 1.4.0 to 1.5.0 (#4547) 
Bumps [lazy_static](https://github.com/rust-lang-nursery/lazy-static.rs) from 1.4.0 to 1.5.0.
- [Release notes](https://github.com/rust-lang-nursery/lazy-static.rs/releases)
- [Commits](https://github.com/rust-lang-nursery/lazy-static.rs/compare/1.4.0...1.5.0)

---
updated-dependencies:
- dependency-name: lazy_static
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-25 09:52:31 +02:00
dependabot[bot]
472096a5a2 build(deps): bump dashmap from 5.5.3 to 6.0.1 (#4546) 
Bumps [dashmap](https://github.com/xacrimon/dashmap) from 5.5.3 to 6.0.1.
- [Release notes](https://github.com/xacrimon/dashmap/releases)
- [Commits](https://github.com/xacrimon/dashmap/compare/v.5.5.3...v6.0.1)

---
updated-dependencies:
- dependency-name: dashmap
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-06-25 09:26:11 +02:00
Tim Visée
a359b6ed6a Fix typo (#4481) 2024-06-18 10:10:31 +02:00