linear_dtype in comfy_quant metadata can be used to set if the int4 op does
the matrix multiplication in int8 or int4, the default is int4 on GPUs that
support it with fallback to int8 for GPUs that don't.
* main: implement --vram-headroom
Implement --vram-headroom for dynamic vram as a hybrid debug/diagnostic
option that can be used for people who still report shared VRAM spills.
They can trial and error the setting to maintain a bit more headroom to
avoid shared VRAM spills.
* main: implement --reserve-vram
Implement --reserve-vram as extra headroom on the simple method which
is semantically as close as possible to the stated functionality and
formet behaviour of non-dynamic VRAM.
* memory_management: Add direct to read GPU mode
Make destination optional (or make it optionally GPU) and use aimdo
to file_read direct to GPU.
* ops: Remove stream pin buffers and use aimdo reads
This consumed too much RAM and its better to just take the hit on
the CPU syncing back the stream on a short ring buffer. Aimdo
implements this so just rip the stream pin buffer from comfy.
* model_management: all active pin registration movement
Its better to just let the active model load past the pin limit as
pins and let the pins move around. The saves the HDD and SATA
people disk traffic while only costing a few GPU syncs.
* utils: use aimdo file handle
This opens on windows with more favourable flags
* mp: only count the model proper for loaded_ram and vram
Exclude live loras from the numbers to avoid the case where the reported
loaded memory exceeds the size of the model.
This causes me confusion in the Kijai visualizer when it looked fully
loaded but was hitting disk due to this accounding disrepency.
* utils: add bit reverse utility
useful for max scattering something ordered.
* pinned_memory: Implement offload balancing
Use a max scatter alogorithm to prioritize pins of the same size such
that when doing a little bit of offloading it gets scattered, allowing
the prefetcher to more evenly swollow the offload.
* comfy-aimdo 0.4.7
Aimdo 0.4.7 implement VRAM buffer exhaustion predection to avoid
early speculative load of weights that definately wont fix once the
inference gets further in.
* model-prefetch: consolidate pin ensures on the sync point
This could happen mid prefetch block, cause a sync of the entire
block and lose overlap. Get ahead of the problem with a free down
at the natural compute stream sync point.
* mm: Put a 2GB min on the pin ceiling
This is reasonably bad if it starts causing swap pressure, moreso than
during normal ram-cache proceedings. Clamp it.
* add --fast-disk
