* Speed up the startup prune's owned-prefix check
mark_contents_missing_outside_prefixes tested every live AssetContent row
against every owned prefix with Path.is_relative_to, which walks the path's
parents on each call: rows x prefixes x depth. At 9k rows and 150 prefixes
that was 9.5s (15.5s with deeper output paths).
path_prefix_matcher normalizes the prefixes once and checks each row with a
normcase'd, separator-bounded str.startswith over a tuple, keeping
is_relative_to's component bounds and platform case rules. The same case
now takes about 0.04s, flat in prefix count and depth.
* Move path_prefix_matcher next to the SQL containment predicate
path_utils needs the same check for per-file tagging, and scanner_changes
already imports path_utils, so the matcher moves to app.assets.helpers
beside sql_path_under_prefix.
* Speed up asset tag derivation during scans
get_backend_system_tags_from_path checked every scanned file against every
model base with Path.is_relative_to, which dominated the fast scan: 37s of
build_asset_specs for 100k output files. Using path_prefix_matcher, the
separator-bounded string check the startup prune already uses, brings that
to about 7s.
* Keep path_prefix_matcher's parity for paths starting with two slashes
abspath keeps exactly two leading slashes, and pathlib treats that '//' as an
anchor of its own, so Path('//server/f').is_relative_to('/') is False, but
the string check accepted it. Prefixes are now split once by whether their
anchor is '//', and a candidate is only checked against prefixes with the
same anchor. The per-row check is still a single str.startswith.
* Build tag-derivation prefix matchers once per folder config
get_backend_system_tags_from_path built a path_prefix_matcher for input,
output, temp and every model category on each call, so most of its cost was
normalizing the same prefixes again for every scanned file. cached_prefix_matcher
memoizes construction on the raw prefix tuple, which the callers pass as
absolute folder paths; a folder-config change is a new key.
100k files, 58 model bases: tag derivation 81.8 -> 27.7 us/file, and
build_asset_specs CPU 11.55 -> 6.40s.
* State cached_prefix_matcher's absolute-prefix precondition
The docstring presented absolute prefixes as a property of the callers. It
is a precondition the caller must meet: folder_paths stores what it is given,
and a relative prefix is resolved once, on first use, and then frozen.
---------
Co-authored-by: guill <jacob.e.segal@gmail.com>
process_qwen2vl_images() computed the patch grid from height/width only
but flatten_patches kept every input channel, so a 4-channel image
(e.g. Qwen-Image-2.1's RGBA VAEDecode output fed back as a reference
image) inflated the patch count by channels/3 and crashed with a shape
mismatch against the position embeddings. Drop extra channels up front
so only RGB reaches the patch embed.
Add device-aware torch-npu stream creation, current-stream lookup, and synchronization so the existing async offload path can run on Ascend NPU devices. Keep the feature opt-in and cover stream rotation and disabled behavior in the unit-test suite.
* chore(assets): drop the unused asset_meta table from migration 0007
* fix(assets): drop asset_meta when downgrading a database that already created it
* docs(assets): clarify asset schema docstring
* test(assets): assert alembic and ORM index parity for the surviving asset tables
* test(assets): cover asset system state index parity
* refactor(tests): hoist migration-0007 test imports to module scope
* fix(assets): guard the hashing dependency and chain the real import error
* fix(assets): always resume background scanning when prompt handling fails
* fix(api): derive the assets feature flag from the selected manager
* fix(assets): paginate enrichment by id cursor so failures cannot starve or overflow the query
* refactor(assets): extract prompt_worker so its resume contract is testable in-process
* refactor(assets): test the blake3 import guard in-process instead of via subprocess
* fix(assets): advance the enrichment cursor only past rows the batch attempted
* fix(assets): track the scan pause across prompt worker iterations
* chore(assets): address review follow-ups in the hashing guard, feature flags, and pagination pin
* docs(assets): describe enrichment rows as attempted rather than selected
The cursor holds at the last row a batch actually attempted, so a pause ends a batch early and the rows behind it are selected again when the scan resumes. Only the attempt is capped at once per pass.
* test(api): derive the expected assets flag from the manager under test
The assertion asked for the flag with no argument, so it read the parameter default rather than anything the manager reported - in a test whose subject is the two agreeing. Passing the manager's own state keeps it honest if the setup ever yields an enabled manager.
* test(assets): let a broken prompt worker import fail instead of skipping
The fixture wrapped importlib.import_module in a bare except that called
pytest.skip, so a circular import, a missing dependency or a syntax error in
app/prompt_worker.py would retire all four resume-contract tests while CI
stayed green. The whole premise of extracting the module is that main.py can
import it, so an import failure has to be a collection error.
The CPU guard is genuinely load-bearing — comfy.model_management selects its
device at import time and a CUDA build with no driver raises there — so it is
kept, but as a precondition rather than an exception handler, matching the
args.cpu-before-import convention already used by the comfy_test and
comfy_api_nodes_test modules. Nothing is caught now.
* docs(assets): name the unattempted rows instead of the ones behind the cursor
The cursor moves forward through ascending ids, so 'the rows behind it' reads as the rows already passed - the opposite of what is selected again.
* fix(assets): only absorb duplicate-path races when seeding scanned assets
* fix(db): copy the legacy database inside the process lock
* fix(assets): drop watch-list entries on stat errors instead of aborting the scan
* fix(assets): clean up temp uploads on validation failures
Multipart parsing writes the uploaded bytes to a temporary file before it
validates the remaining form fields, so a request rejected after its file
part had already been read left the temp file and its uuid directory on
disk.
Routing those removals through delete_temp_file_if_exists also changes the
success path. The previous helper returned early when the temp file was
already gone, so it never reached the parent rmdir; the shared helper
attempts the rmdir unconditionally. Moving the upload to its destination
leaves the temp path absent, so a successful upload now also discards its
empty uuid directory, closing a pre-existing leak.
* fix(assets): keep updated_at stable on no-op renames
* docs(assets): make module docstrings and the rebuild warning truthful
* fix(assets): correct event-log status snapshots and failure telemetry
* test(assets): make the keyset tie-breaker and temp-exclusion tests falsifiable
* chore: comment cleanup
Comment-Gate: 3 quarantined
* fix(assets): keep unreadable filesystem metadata from failing a whole scan batch
* fix(assets): remove temporary uploads on non-UploadError failures
* fix(assets): preserve successful specs when a scan batch fault propagates
* test(db): drop the inert legacy-copy patch from the path preparation tests
prepare_file_db_path no longer copies the legacy database - that moved inside the process lock in _init_file_db - so patching copy_legacy_default_db here did nothing. Leaving it implied a side effect the function does not have, and would have masked one if it were reintroduced.
* fix(assets): report specs committed before a batch fault and preserve the fault itself
* fix(assets): distinguish partial batch insert failures
* refactor(assets): collapse the duplicated batch fault deferral into seed_asset_specs
* fix(assets): reject duplicate file parts instead of stranding the first upload
* refactor(assets): reap empty upload directories without importing the API layer
* fix(assets): emit the invalid-mtime event once per scan
Every other per-file emit on this scan path is gated -- mark_emitted(
"stat_failed:enrich"), "hash_discarded_modified", "hash_failed",
"enrich_failed" -- but scanner.invalid_mtime fired per file, so a restored
archive or a FAT volume of pre-epoch mtimes put one structured event per file
into the stream the closed vocabulary exists to keep parseable.
Counted and emitted once, carrying the count. seed_asset_specs receives no
_ScanProgress object and neither does insert_asset_specs above it, so routing
this through mark_emitted would mean changing both signatures plus the seeder
call site; the count form needs neither and the event is now strictly more
informative than N identical fieldless lines. The per-file logging.warning is
unchanged, and the emit stays inside seed_asset_specs so the static call-site
manifest still matches.
test_seed_skips_negative_fresh_mtime_with_warning_and_telemetry now pins the
full list of invalid_mtime lines to exactly ["... count=1"] instead of
asserting one such line exists -- a strictly stronger assertion, and the only
change the new field required.
* fix(assets): keep spec construction failures from wedging the watch list
get_name_and_tags_from_asset_path raises ValueError by contract when a path
stops resolving to a configured root, and it sat outside the guard, as did
compute_loader_path and mimetypes.guess_type. An escape skipped the
_WATCH_LIST[:] = remaining write at the end, so drained entries stayed on the
list and were re-attempted every tick while entries past the fault never
reached the increment _WATCH_SCAN_RETRIES needs to retire them. The list
wedged permanently.
Spec construction is now inside a guard that drops just the offending entry,
and the list write moved into a finally so no future escape can skip it. The
loop walks an iterator rather than the list, so the finally can put back the
entries it never reached instead of discarding them.
New event name rather than reusing one: scanner.watch_seed_failed is emitted
only when seed_asset_specs returns an error, and widening it to also mean
"never got as far as seeding" would make it lie -- a consumer treating it as a
database-health signal would get false positives from what is really a path
layout problem. scanner.watch_spec_failed is registered in ALLOWED_EVENTS and
in the static call-site manifest.
* refactor(assets): export the live-path conflict check as public API
scanner.py reached past the package's own re-export surface to import
_is_live_path_conflict directly out of records.py. The underscore said
module-private while the import said otherwise, and records.py deliberately
publishes its public names through app.assets.database.queries -- which the
same import block three lines above was already using.
The use is correct and unchanged; only the name and the route change. Renamed
to is_live_path_conflict, listed in the package __init__ import and __all__
alongside its siblings, and scanner.py now takes it from the package like
everything else it imports from there.
* docs(db): restore the rationale for locking before migration
Commit 1dbcdcd7 and the comment-cleanup pass 8205022f reduced this to "All
database reads and writes, including the legacy import, run under the lock",
dropping the part that did the work: upstream master locks after migrating and
justifies it with "Alembic uses its own connection, so we must wait until it's
done before locking -- otherwise our own lock blocks the migration". That is
false, the lock is on a separate <db>.lock file, and the surviving sentence
said nothing to stop a contributor "fixing" the ordering back.
Restored and adapted rather than pasted: the legacy copy and the db_exists
probe now happen inside the lock, which the original text predates, so both
are named in the list of things the ordering makes mutually exclusive.
* fix(assets): stop a scan on memory exhaustion instead of deferring it
MemoryError is an Exception, so the per-spec and per-batch handlers stored it alongside ordinary faults and carried on - allocating for every remaining spec and then every remaining batch while the process was already out of memory. Both handlers now let it through, and the scan records a failure and stops.
* docs(assets): document the prune failure response and its None result
The route gained a 500 PRUNE_FAILED branch and the seeder method gained a None return, both so a prune that did not run cannot be reported as a clean one. Neither contract was written down.
* test(assets): assert the surviving spec count after a propagated fault
* docs(db): shorten the lock-ordering comment while keeping its rationale
* docs(tests): drop the cross-module justification from the import-order comment
* test(assets): restore the cpu flag after the guarded prompt worker import
* test(assets): restore the cpu flag even when the prompt worker import fails
* fix(assets): drop asset_meta in a new migration instead of editing 0007
0007 shipped in v0.36.0, v0.37.0 and v0.37.1, so editing it would leave two
installs at that revision with different schemas depending on when they
upgraded. Restore 0007 to its released form and drop the unused asset_meta
table in 0008 instead.
Nothing reads or writes asset_meta; asset metadata lives in the JSON columns
on assets. 0008 downgrades by recreating the table and its four indexes
exactly as 0007 creates them.
* refactor(assets): keep prompt_worker in main.py
Custom nodes may reference main.prompt_worker, and tests can already
import main (test_db_init_locking does), so the function stays where it
was. Its body keeps the resume-on-failure handling and the pause flag
that persists across loop iterations; the tests now call
main.prompt_worker.
* fix(assets): report the selected manager through the existing feature flags
* fix(assets): record a failed prune in the scan status
A failed prune left the scan's error list empty, so the run looked clean.
Record it instead and let discovery continue as before.
* test(assets): test the feature flag API directly instead of copying the startup write
Both parity tests wrote SERVER_FEATURE_FLAGS["assets"] themselves, so they
passed whatever startup did. Pin the one real claim - a no-argument
get_server_features() reports the flag - in the feature flags tests, and drop
the disabled case, which default_asset_manager already covers.
* Bring #16486's watch-list batching and seed logging into this branch
The merge before this commit is `git merge -X ours origin/master`: in
conflicting hunks it keeps this branch's side. This commit ports what
#16486 changed in those hunks.
- tick_watch_list takes no session. It collects settled entries and seeds
them in one batch through insert_asset_specs, keeping this branch's
per-entry stat and spec handling and the finally that always rewrites
the watch list. A failed seed is reported once per batch, since the
batch only returns its first error.
- seed_asset_specs no longer warns again for a skipped spec;
observe_asset_specs already logged why.
- Tests call tick_watch_list() without a session, bind the write session
where they seed for real, and fake insert_asset_specs with its
(created, error) return. The mid-drain fault now comes from stat,
because seeding runs after the loop.
* Note where the assets core flag is set and why prompt_worker catches BaseException
---------
Co-authored-by: guill <jacob.e.segal@gmail.com>
* Take the SQLite write lock up front for scan and output-registration writes
The scanner's seeding and reference sync, and executed-output registration,
write through a separate engine whose transactions open with BEGIN
IMMEDIATE, and the database runs in WAL mode. On those paths stat, hashing
and metadata extraction now happen before the write transaction opens, and
reference-sync results are applied only to rows unchanged since they were
observed. busy_timeout stays at pysqlite's 5s default.
A fast-scan batch now commits as one transaction, so an unexpected error
partway through discards the whole batch; the next scan recreates it.
Enrichment, verification, uploads and tagging still write through the
existing sessions.
Migration backups use SQLite's backup API, and a legacy database is
checkpointed before it is relocated, since in WAL mode committed pages can
live in the -wal file that a plain file copy misses.
* Skip relocating a legacy database whose WAL cannot be checkpointed
The checkpoint can report busy without raising; moving the file then would
leave committed pages behind in the -wal. Also keep the source's file mode
on SQLite backups, as the plain copy did.
* Replace run_write_txn with a create_write_session factory
Write paths open the write engine's session the same way the rest of the
code opens create_session(), and commit explicitly. Executed-output
registration reads the new record's fields before committing, so expiry
does not reload them in a second write transaction.
* Document the scanner's pre-transaction observation types
* Document that write sessions must not nest
* Warn that a nested write session looks like lock contention
* Seed a hashed spec with the stat its hash was verified against
* Bound the SQLite backup so a locked destination cannot hang startup
* Time out a backup only while it is blocked
* Commit each drained entry before hashing the next
drain_pending_verifications and drain_transition_queue ran every entry in
one session, so an entry that wrote (marking a vanished file missing, say)
left a transaction open while the next entry's file was stat'ed and
hashed. Commit at the top of each entry instead, so the hash runs with no
transaction open.
* Seed settled watch-list entries through insert_asset_specs
tick_watch_list seeded each settled file through seed_asset_specs in the
caller's session, where the enrich phase still held drain_pending's
writes open, and each seed ran in a deferred savepoint that reads before
it writes. Collect the settled specs and hand them to insert_asset_specs,
which stats and hashes before opening one write session. The seeder
commits the pending verifications before ticking, so no transaction is
open while the watch list stats or waits for the write lock.
* Read upload metadata before the claim transaction
_create_upload_record read the file for system metadata after the
content claim had opened a write transaction. Callers now extract the
metadata before opening their session (or, when reusing content, before
claiming it) and pass it in. The claim's own stat re-check stays inside
the transaction: that is what makes the claim sound.
* Keep the upgrade error when restoring the backup also fails
If restoring the pre-upgrade backup raised, that exception replaced the
upgrade's, and the backup's location was never logged. Log the upgrade
error first, log where the pre-upgrade copy is kept if the restore or its
cleanup fails, and re-raise the upgrade error either way.
* Note the drains' session requirement and word the restore log for either failure
The drains' per-entry commit only leaves no transaction open on a
create_session() session. The restore log now covers a failed backup
removal as well as a failed restore. The watch-list admission test
patches insert_asset_specs, the seam tick_watch_list now calls.
* Log the real error when a seed spec cannot be read
observe_asset_specs treated every OSError as a vanished file, so a
permission error or an I/O error on a file that still exists was
reported only as "Skipping vanished asset during scan". A missing file
is still handled as before; any other OSError is now also logged through
_log_scan_error before the spec is skipped. The vanished-path test's
fake now raises FileNotFoundError, the error a vanished file produces.
* Report an unreadable seed spec once, without also calling it vanished
* Skip a pending verification whose row changed while it was hashed
Committing per entry means no lock is held while the file is hashed, so
another writer can retire or replace the row in that time. The drain
would then store a hash on a missing row, or split it and attach a
record to the other writer's row. Re-read the row after hashing and skip
it unless it is still live with the hash, size and mtime it was loaded
with, as apply_reference_observations does.
* Remove the stored upload file in the reupload claim test
The test cleaned up its temp files but left the first upload's stored
file in the output directory.
* Take the SQLite write lock up front for scan and output-registration writes
The scanner's seeding and reference sync, and executed-output registration,
write through a separate engine whose transactions open with BEGIN
IMMEDIATE, and the database runs in WAL mode. On those paths stat, hashing
and metadata extraction now happen before the write transaction opens, and
reference-sync results are applied only to rows unchanged since they were
observed. busy_timeout stays at pysqlite's 5s default.
A fast-scan batch now commits as one transaction, so an unexpected error
partway through discards the whole batch; the next scan recreates it.
Enrichment, verification, uploads and tagging still write through the
existing sessions.
Migration backups use SQLite's backup API, and a legacy database is
checkpointed before it is relocated, since in WAL mode committed pages can
live in the -wal file that a plain file copy misses.
* Skip relocating a legacy database whose WAL cannot be checkpointed
The checkpoint can report busy without raising; moving the file then would
leave committed pages behind in the -wal. Also keep the source's file mode
on SQLite backups, as the plain copy did.
* Replace run_write_txn with a create_write_session factory
Write paths open the write engine's session the same way the rest of the
code opens create_session(), and commit explicitly. Executed-output
registration reads the new record's fields before committing, so expiry
does not reload them in a second write transaction.
* Document the scanner's pre-transaction observation types
* Document that write sessions must not nest
* Warn that a nested write session looks like lock contention
* Seed a hashed spec with the stat its hash was verified against
* Bound the SQLite backup so a locked destination cannot hang startup
* Time out a backup only while it is blocked
Upscale models loaded via Spandrel expect exactly 3 input channels, so a 4-channel IMAGE crashed in the model's first conv. Split off the alpha channel before upscaling, then resize it to the output resolution and concatenate it back so transparency survives the upscale.
Fixes#16499
The AR decode loop captured the per-layer graphs without decode buffers, so
Llama2_ fell back to x = x.clone() and every replay read the address of the
first step after it had been freed. The output was garbage audio codes with no
error: the decoded track is NaN, saved as a constant -32768 by the FLAC encoder.
YuE2 and the generic generate() already pin the hidden state and rotary tensors
through decode_buffers; do the same here, gated the way generate() gates it.
Fixes#16002 (the silent-noise report) and #16222.
dante01yoon