Scott Cutler 860ee2a8a4 ggml-cuda: replace event-based watchdog with per-GPU ring buffer ...

Completely rework the GGML_CUDA_AR_WATCHDOG system:

- Replace the shared debug_buf + event-polling + queue design with
  per-GPU ring buffers in pinned host memory
- Kernel writes a debug record only on spin-limit bailout: claims a
  ring slot via atomicAdd (single-GPU host atomics work on RTX 5090),
  writes fields, fences, sets completion flag, then all threads exit
- Watchdog thread simply polls ring head counters every 1ms and prints
  any new complete records — no CUDA event queries, no mutex, no queue
- Zero overhead on the dispatch path (no queue posting, no memset)
- Watchdog shutdown returns within ~1ms (atomic bool, no drain)
- On bailout the kernel skips Phase 3 entirely and exits cleanly

Verified: 20/20 prefill soak test clean at ~1112 t/s, no hangs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
P32, tensors <= 256 KB. Notes in NOTES-allreduce.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-04-22 19:54:03 -07:00

..

cmake

cmake: improve NCCL detection for source-tree builds, add static/dynamic switch

2026-04-21 19:01:11 -07:00

include

CUDA: manage NCCL communicators in context (#21891)

2026-04-15 15:58:40 +02:00

src

ggml-cuda: replace event-based watchdog with per-GPU ring buffer

2026-04-22 19:54:03 -07:00

.gitignore

vulkan : cmake integration (#8119)

2024-07-13 18:12:39 +02:00

CMakeLists.txt

ggml-cuda: add AllReduce hang watchdog (GGML_CUDA_AR_WATCHDOG)

2026-04-21 19:50:17 -07:00