* test(consensus): de-flake replicate_points_stream_transfer_updates override case
With override_points=True the background writer re-upserts points
9990-9999, re-rolling their city payload. Points flipping away from
"London" legitimately drop out of the filtered count on both shards, so
asserting dest_filtered_count >= original snapshot count is not a valid
invariant. On a slow CI runner the sleep(1)+kill() stopped the writer
right after the overrides, before new inserts could compensate, making
a net-negative flip likely (observed: 4954 >= 4959 failure).
Replace the blind kill with a bounded workload (60 points) joined
cleanly before the ~10s transfer of 10k points can finish, assert the
writer exit code, and allow the filtered count to drop by up to the
number of overridden points.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(consensus): guard that writer finishes while transfer is running
The exact count consistency check requires every concurrent write to go
through the transfer proxy. Make that precondition explicit: if the
transfer ever finishes before the writer, fail with a clear message
instead of a confusing count mismatch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(consensus): make replicate_points update consistency checks exact
Assign the city payload deterministically by point ID parity so filter
membership can never change under concurrent overwrites. All assertions
become exact ID-set comparisons with no slack: random city re-rolls made
count-based checks unsound, since the forward proxy filters forwarded
updates by post-update state and a point flipping out of the filter
legitimately goes stale or missing on the destination.
Replace the background writer process (sleep/kill/join choreography)
with synchronous wait=true upserts issued while the transfer streams the
initial points. Leave low point IDs unoccupied and insert into them
during the transfer: the stream cursor passes them immediately, so these
points can only reach the destination through live update forwarding,
which the previous layout (writes at the stream tail) never verified.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
The test relied on concurrent background upserts adding *new* matching
points to the destination shard during the streamed transfer, asserting
the destination count was strictly greater than the original snapshot.
Whether new matching points land in the destination before the transfer
completes is timing-dependent (especially under pytest-xdist load), so
the strict `>` assertion is racy and occasionally fails with equal
counts (e.g. `assert 5031 > 5031`).
Relax to `>=` and keep the strict `dest == src` consistency check, which
is the actual invariant being verified.
Co-authored-by: Cursor <cursoragent@cursor.com>
* ci: parallelize consensus tests with pytest-xdist
Enable pytest-xdist for consensus_tests to run tests across multiple
workers in parallel, significantly reducing CI wall time (~20min → ~5-7min).
Changes:
- Add `-n auto --dist=loadfile` to the consensus test pytest invocation
- Remove hardcoded port_seed from tests that don't need fixed ports for
restart/rejoin (test_order_by, test_consensus_compaction,
test_named_vector_crud, test_listener_node)
- Give test_cluster_rejoin its own PORT_SEED=15000 to avoid port
conflicts with auth tests (PORT_SEED=10000)
- Derive restart ports from killed PeerProcess objects instead of
hardcoded arithmetic where possible
- Add xdist_group("auth") marker to auth test files to ensure they
run on the same worker (they share PORT_SEED=10000)
Made-with: Cursor
* fix: remove remaining hardcoded port_seed=20000 causing parallel test conflicts
8 test files were using port_seed=20000 as a positional argument to
start_cluster(), which was missed in the initial change. When running
in parallel with pytest-xdist, multiple workers would try to bind to
the same port range (20000-20x02), causing port conflicts and cascading
test failures.
Also remove port_seed=23000 from test_snapshot_recovery_kill.py since
it doesn't need fixed ports for restart.
Made-with: Cursor
* fix: use saved port for restart in test_two_follower_nodes_down
The test was restarting killed peers on hardcoded ports (20200/20100)
that previously matched port_seed=20000. After switching to random
ports, the restart ports no longer match the original peer ports,
causing raft state URI mismatches and peer startup failures.
Save the p2p_port from the killed PeerProcess and reuse it for restart.
Made-with: Cursor
* Reuse p2p ports when restarting killed peers in consensus tests
When a peer is killed and restarted with random ports, it gets a new
consensus URI. The cluster needs a Raft operation to update this URI,
which under CPU contention from parallel test workers can exceed the
30-second timeout. Fix by capturing each peer's p2p_port before killing
and reusing it on restart, so the URI stays the same and no consensus
update is needed.
Made-with: Cursor
* A few improvements for parallel runs (#8731)
* fix: make auth tests' PORT_SEED per-worker to avoid port collisions
* ci: improve failure visibility for parallel consensus tests
* Three small changes to make hangs, interleaved output, and coverage runs behave predictably under pytest-xdist
* fix: two test bugs surfaced by parallel runs and revert drop PR_SET_PDEATHSIG helper
* fix: wait for count convergence in test_triple_replication
* fix: clean leaked peer processes at test start
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Cursor Agent <agent@cursor.com>
Co-authored-by: tellet-q <166374656+tellet-q@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Respect payload and hashring filters while replicating point
* Merge points instead of replacing for filtered stream records transfer
* cleanup
* Update utils
* Allow filters in transfer
* drop hashring filter since its not required when destination has single shard
* test comment
* Pass actual filter
* use exact=True
* Forward updates that satisfy the filter
* make test long enough for extra points to be upserted
* minor improvements
* fix test
* try with only inserting new points
* Temporarily forward points that match before or after
* Fix failing test
* Only do inserts
* Dont merge points and update old points in test
* Fix the bug in integration test
* fix read_batch_with_hashring fn name and docs
* changes after rebase
* add validation check
* apply suggestions
* Always trim clock tag when forwarding to a different shard ID
* review fixes
---------
Co-authored-by: generall <andrey@vasnetsov.com>
* Allow replicating points between shards
* Add integration test for replicating points between shards
* Drop filter from ForwardProxyShard to for now
* Clean up test
* fix assert failures
* Fix integration test
* Drop reliance on FilteredStreamRecords
* minor improvements
