9 Commits

Include renames

Author	SHA1	Message	Date
Simon Pinfoldandguill	02ba106f97	fix(assets): harden the asset catalogue's failure paths (#16393) ... * chore(assets): drop the unused asset_meta table from migration 0007 * fix(assets): drop asset_meta when downgrading a database that already created it * docs(assets): clarify asset schema docstring * test(assets): assert alembic and ORM index parity for the surviving asset tables * test(assets): cover asset system state index parity * refactor(tests): hoist migration-0007 test imports to module scope * fix(assets): guard the hashing dependency and chain the real import error * fix(assets): always resume background scanning when prompt handling fails * fix(api): derive the assets feature flag from the selected manager * fix(assets): paginate enrichment by id cursor so failures cannot starve or overflow the query * refactor(assets): extract prompt_worker so its resume contract is testable in-process * refactor(assets): test the blake3 import guard in-process instead of via subprocess * fix(assets): advance the enrichment cursor only past rows the batch attempted * fix(assets): track the scan pause across prompt worker iterations * chore(assets): address review follow-ups in the hashing guard, feature flags, and pagination pin * docs(assets): describe enrichment rows as attempted rather than selected The cursor holds at the last row a batch actually attempted, so a pause ends a batch early and the rows behind it are selected again when the scan resumes. Only the attempt is capped at once per pass. * test(api): derive the expected assets flag from the manager under test The assertion asked for the flag with no argument, so it read the parameter default rather than anything the manager reported - in a test whose subject is the two agreeing. Passing the manager's own state keeps it honest if the setup ever yields an enabled manager. * test(assets): let a broken prompt worker import fail instead of skipping The fixture wrapped importlib.import_module in a bare except that called pytest.skip, so a circular import, a missing dependency or a syntax error in app/prompt_worker.py would retire all four resume-contract tests while CI stayed green. The whole premise of extracting the module is that main.py can import it, so an import failure has to be a collection error. The CPU guard is genuinely load-bearing — comfy.model_management selects its device at import time and a CUDA build with no driver raises there — so it is kept, but as a precondition rather than an exception handler, matching the args.cpu-before-import convention already used by the comfy_test and comfy_api_nodes_test modules. Nothing is caught now. * docs(assets): name the unattempted rows instead of the ones behind the cursor The cursor moves forward through ascending ids, so 'the rows behind it' reads as the rows already passed - the opposite of what is selected again. * fix(assets): only absorb duplicate-path races when seeding scanned assets * fix(db): copy the legacy database inside the process lock * fix(assets): drop watch-list entries on stat errors instead of aborting the scan * fix(assets): clean up temp uploads on validation failures Multipart parsing writes the uploaded bytes to a temporary file before it validates the remaining form fields, so a request rejected after its file part had already been read left the temp file and its uuid directory on disk. Routing those removals through delete_temp_file_if_exists also changes the success path. The previous helper returned early when the temp file was already gone, so it never reached the parent rmdir; the shared helper attempts the rmdir unconditionally. Moving the upload to its destination leaves the temp path absent, so a successful upload now also discards its empty uuid directory, closing a pre-existing leak. * fix(assets): keep updated_at stable on no-op renames * docs(assets): make module docstrings and the rebuild warning truthful * fix(assets): correct event-log status snapshots and failure telemetry * test(assets): make the keyset tie-breaker and temp-exclusion tests falsifiable * chore: comment cleanup Comment-Gate: 3 quarantined * fix(assets): keep unreadable filesystem metadata from failing a whole scan batch * fix(assets): remove temporary uploads on non-UploadError failures * fix(assets): preserve successful specs when a scan batch fault propagates * test(db): drop the inert legacy-copy patch from the path preparation tests prepare_file_db_path no longer copies the legacy database - that moved inside the process lock in _init_file_db - so patching copy_legacy_default_db here did nothing. Leaving it implied a side effect the function does not have, and would have masked one if it were reintroduced. * fix(assets): report specs committed before a batch fault and preserve the fault itself * fix(assets): distinguish partial batch insert failures * refactor(assets): collapse the duplicated batch fault deferral into seed_asset_specs * fix(assets): reject duplicate file parts instead of stranding the first upload * refactor(assets): reap empty upload directories without importing the API layer * fix(assets): emit the invalid-mtime event once per scan Every other per-file emit on this scan path is gated -- mark_emitted( "stat_failed:enrich"), "hash_discarded_modified", "hash_failed", "enrich_failed" -- but scanner.invalid_mtime fired per file, so a restored archive or a FAT volume of pre-epoch mtimes put one structured event per file into the stream the closed vocabulary exists to keep parseable. Counted and emitted once, carrying the count. seed_asset_specs receives no _ScanProgress object and neither does insert_asset_specs above it, so routing this through mark_emitted would mean changing both signatures plus the seeder call site; the count form needs neither and the event is now strictly more informative than N identical fieldless lines. The per-file logging.warning is unchanged, and the emit stays inside seed_asset_specs so the static call-site manifest still matches. test_seed_skips_negative_fresh_mtime_with_warning_and_telemetry now pins the full list of invalid_mtime lines to exactly ["... count=1"] instead of asserting one such line exists -- a strictly stronger assertion, and the only change the new field required. * fix(assets): keep spec construction failures from wedging the watch list get_name_and_tags_from_asset_path raises ValueError by contract when a path stops resolving to a configured root, and it sat outside the guard, as did compute_loader_path and mimetypes.guess_type. An escape skipped the _WATCH_LIST[:] = remaining write at the end, so drained entries stayed on the list and were re-attempted every tick while entries past the fault never reached the increment _WATCH_SCAN_RETRIES needs to retire them. The list wedged permanently. Spec construction is now inside a guard that drops just the offending entry, and the list write moved into a finally so no future escape can skip it. The loop walks an iterator rather than the list, so the finally can put back the entries it never reached instead of discarding them. New event name rather than reusing one: scanner.watch_seed_failed is emitted only when seed_asset_specs returns an error, and widening it to also mean "never got as far as seeding" would make it lie -- a consumer treating it as a database-health signal would get false positives from what is really a path layout problem. scanner.watch_spec_failed is registered in ALLOWED_EVENTS and in the static call-site manifest. * refactor(assets): export the live-path conflict check as public API scanner.py reached past the package's own re-export surface to import _is_live_path_conflict directly out of records.py. The underscore said module-private while the import said otherwise, and records.py deliberately publishes its public names through app.assets.database.queries -- which the same import block three lines above was already using. The use is correct and unchanged; only the name and the route change. Renamed to is_live_path_conflict, listed in the package __init__ import and __all__ alongside its siblings, and scanner.py now takes it from the package like everything else it imports from there. * docs(db): restore the rationale for locking before migration Commit 1dbcdcd7 and the comment-cleanup pass 8205022f reduced this to "All database reads and writes, including the legacy import, run under the lock", dropping the part that did the work: upstream master locks after migrating and justifies it with "Alembic uses its own connection, so we must wait until it's done before locking -- otherwise our own lock blocks the migration". That is false, the lock is on a separate <db>.lock file, and the surviving sentence said nothing to stop a contributor "fixing" the ordering back. Restored and adapted rather than pasted: the legacy copy and the db_exists probe now happen inside the lock, which the original text predates, so both are named in the list of things the ordering makes mutually exclusive. * fix(assets): stop a scan on memory exhaustion instead of deferring it MemoryError is an Exception, so the per-spec and per-batch handlers stored it alongside ordinary faults and carried on - allocating for every remaining spec and then every remaining batch while the process was already out of memory. Both handlers now let it through, and the scan records a failure and stops. * docs(assets): document the prune failure response and its None result The route gained a 500 PRUNE_FAILED branch and the seeder method gained a None return, both so a prune that did not run cannot be reported as a clean one. Neither contract was written down. * test(assets): assert the surviving spec count after a propagated fault * docs(db): shorten the lock-ordering comment while keeping its rationale * docs(tests): drop the cross-module justification from the import-order comment * test(assets): restore the cpu flag after the guarded prompt worker import * test(assets): restore the cpu flag even when the prompt worker import fails * fix(assets): drop asset_meta in a new migration instead of editing 0007 0007 shipped in v0.36.0, v0.37.0 and v0.37.1, so editing it would leave two installs at that revision with different schemas depending on when they upgraded. Restore 0007 to its released form and drop the unused asset_meta table in 0008 instead. Nothing reads or writes asset_meta; asset metadata lives in the JSON columns on assets. 0008 downgrades by recreating the table and its four indexes exactly as 0007 creates them. * refactor(assets): keep prompt_worker in main.py Custom nodes may reference main.prompt_worker, and tests can already import main (test_db_init_locking does), so the function stays where it was. Its body keeps the resume-on-failure handling and the pause flag that persists across loop iterations; the tests now call main.prompt_worker. * fix(assets): report the selected manager through the existing feature flags * fix(assets): record a failed prune in the scan status A failed prune left the scan's error list empty, so the run looked clean. Record it instead and let discovery continue as before. * test(assets): test the feature flag API directly instead of copying the startup write Both parity tests wrote SERVER_FEATURE_FLAGS["assets"] themselves, so they passed whatever startup did. Pin the one real claim - a no-argument get_server_features() reports the flag - in the feature flags tests, and drop the disabled case, which default_asset_manager already covers. * Bring #16486's watch-list batching and seed logging into this branch The merge before this commit is `git merge -X ours origin/master`: in conflicting hunks it keeps this branch's side. This commit ports what #16486 changed in those hunks. - tick_watch_list takes no session. It collects settled entries and seeds them in one batch through insert_asset_specs, keeping this branch's per-entry stat and spec handling and the finally that always rewrites the watch list. A failed seed is reported once per batch, since the batch only returns its first error. - seed_asset_specs no longer warns again for a skipped spec; observe_asset_specs already logged why. - Tests call tick_watch_list() without a session, bind the write session where they seed for real, and fake insert_asset_specs with its (created, error) return. The mid-drain fault now comes from stat, because seeding runs after the loop. * Note where the assets core flag is set and why prompt_worker catches BaseException --------- Co-authored-by: guill <jacob.e.segal@gmail.com>	2026-09-24 15:51:10 -07:00
Simon Pinfold	f427c3a285	Move file reads out of the remaining asset write transactions (#16486) ... * Take the SQLite write lock up front for scan and output-registration writes The scanner's seeding and reference sync, and executed-output registration, write through a separate engine whose transactions open with BEGIN IMMEDIATE, and the database runs in WAL mode. On those paths stat, hashing and metadata extraction now happen before the write transaction opens, and reference-sync results are applied only to rows unchanged since they were observed. busy_timeout stays at pysqlite's 5s default. A fast-scan batch now commits as one transaction, so an unexpected error partway through discards the whole batch; the next scan recreates it. Enrichment, verification, uploads and tagging still write through the existing sessions. Migration backups use SQLite's backup API, and a legacy database is checkpointed before it is relocated, since in WAL mode committed pages can live in the -wal file that a plain file copy misses. * Skip relocating a legacy database whose WAL cannot be checkpointed The checkpoint can report busy without raising; moving the file then would leave committed pages behind in the -wal. Also keep the source's file mode on SQLite backups, as the plain copy did. * Replace run_write_txn with a create_write_session factory Write paths open the write engine's session the same way the rest of the code opens create_session(), and commit explicitly. Executed-output registration reads the new record's fields before committing, so expiry does not reload them in a second write transaction. * Document the scanner's pre-transaction observation types * Document that write sessions must not nest * Warn that a nested write session looks like lock contention * Seed a hashed spec with the stat its hash was verified against * Bound the SQLite backup so a locked destination cannot hang startup * Time out a backup only while it is blocked * Commit each drained entry before hashing the next drain_pending_verifications and drain_transition_queue ran every entry in one session, so an entry that wrote (marking a vanished file missing, say) left a transaction open while the next entry's file was stat'ed and hashed. Commit at the top of each entry instead, so the hash runs with no transaction open. * Seed settled watch-list entries through insert_asset_specs tick_watch_list seeded each settled file through seed_asset_specs in the caller's session, where the enrich phase still held drain_pending's writes open, and each seed ran in a deferred savepoint that reads before it writes. Collect the settled specs and hand them to insert_asset_specs, which stats and hashes before opening one write session. The seeder commits the pending verifications before ticking, so no transaction is open while the watch list stats or waits for the write lock. * Read upload metadata before the claim transaction _create_upload_record read the file for system metadata after the content claim had opened a write transaction. Callers now extract the metadata before opening their session (or, when reusing content, before claiming it) and pass it in. The claim's own stat re-check stays inside the transaction: that is what makes the claim sound. * Keep the upgrade error when restoring the backup also fails If restoring the pre-upgrade backup raised, that exception replaced the upgrade's, and the backup's location was never logged. Log the upgrade error first, log where the pre-upgrade copy is kept if the restore or its cleanup fails, and re-raise the upgrade error either way. * Note the drains' session requirement and word the restore log for either failure The drains' per-entry commit only leaves no transaction open on a create_session() session. The restore log now covers a failed backup removal as well as a failed restore. The watch-list admission test patches insert_asset_specs, the seam tick_watch_list now calls. * Log the real error when a seed spec cannot be read observe_asset_specs treated every OSError as a vanished file, so a permission error or an I/O error on a file that still exists was reported only as "Skipping vanished asset during scan". A missing file is still handled as before; any other OSError is now also logged through _log_scan_error before the spec is skipped. The vanished-path test's fake now raises FileNotFoundError, the error a vanished file produces. * Report an unreadable seed spec once, without also calling it vanished * Skip a pending verification whose row changed while it was hashed Committing per entry means no lock is held while the file is hashed, so another writer can retire or replace the row in that time. The drain would then store a hash on a missing row, or split it and attach a record to the other writer's row. Re-read the row after hashing and skip it unless it is still live with the hash, size and mtime it was loaded with, as apply_reference_observations does. * Remove the stored upload file in the reupload claim test The test cleaned up its temp files but left the first upload's stored file in the output directory.	2026-09-23 17:23:56 -07:00
Simon Pinfold	9d3bfc393b	Take the SQLite write lock up front for asset scan and output-registration writes (#16480) ... * Take the SQLite write lock up front for scan and output-registration writes The scanner's seeding and reference sync, and executed-output registration, write through a separate engine whose transactions open with BEGIN IMMEDIATE, and the database runs in WAL mode. On those paths stat, hashing and metadata extraction now happen before the write transaction opens, and reference-sync results are applied only to rows unchanged since they were observed. busy_timeout stays at pysqlite's 5s default. A fast-scan batch now commits as one transaction, so an unexpected error partway through discards the whole batch; the next scan recreates it. Enrichment, verification, uploads and tagging still write through the existing sessions. Migration backups use SQLite's backup API, and a legacy database is checkpointed before it is relocated, since in WAL mode committed pages can live in the -wal file that a plain file copy misses. * Skip relocating a legacy database whose WAL cannot be checkpointed The checkpoint can report busy without raising; moving the file then would leave committed pages behind in the -wal. Also keep the source's file mode on SQLite backups, as the plain copy did. * Replace run_write_txn with a create_write_session factory Write paths open the write engine's session the same way the rest of the code opens create_session(), and commit explicitly. Executed-output registration reads the new record's fields before committing, so expiry does not reload them in a second write transaction. * Document the scanner's pre-transaction observation types * Document that write sessions must not nest * Warn that a nested write session looks like lock contention * Seed a hashed spec with the stat its hash was verified against * Bound the SQLite backup so a locked destination cannot hang startup * Time out a backup only while it is blocked	2026-09-23 15:17:11 -07:00
3 people	19e1058f4c	feat(assets): split asset records from content (#16295) ... * review-stack 1/4: code (37 files, +3217/-3958) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: 7007d18582 Merge-base: 783545f689 * review-stack 2/4: tests-removed (24 files, +274/-8220) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file deleted, or modified with deleted/(added+deleted) >= 0.9 Question: For each dropped assertion: obsolete by a ruling, or covered by a tests-new test? Source tip: 7007d18582 Merge-base: 783545f689 * review-stack 3/4: tests-changed (13 files, +1043/-1218) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: remaining modified test files (incl. conftest.py / helpers) Question: Did the edits weaken an existing check? Source tip: 7007d18582 Merge-base: 783545f689 * review-stack 4/4: tests-new (46 files, +8601/-0) Review-and-land stack for synap5e/feat/asset-record-content-split, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: test file added Question: Is the code layer well covered? Source tip: 7007d18582 Merge-base: 783545f689 * review-stack 5/6: code (13 files, +351/-104) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: path not under tests-unit/ or tests/ Question: Is the logic change right? Source tip: eca2c74bff Merge-base: 20d59d2a5f * review-stack 6/6: tests (8 files, +753/-238) Review-and-land stack for synap5e/feat/assets-di, generated by review-stack.py. Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: every changed file under tests-unit/ or tests/ (added, modified, or deleted) Question: Is the code layer well covered, and did any edit weaken an existing check? Source tip: eca2c74bff Merge-base: 20d59d2a5f * review-stack 7/8: ported-fixes (42 files, +1361/-180) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: the 11 base-branch fix/docs commits 595cd6e4..94d7185b cherry-picked across the DI refactor (7efdd1d7 excluded, superseded by layer 8) Question: was each base fix ported faithfully across the DI refactor? Source tip: 6841881069284803b902b4a9e33bdcda13126771 Merge-base: 7fdfb40f4b * review-stack 8/8: defensive-parity (4 files, +36/-3) Review-and-land stack for synap5e/feat/assets-di-v2, generated by review-stack.py conventions (hand-built continuation layer; see the PR body). Once approved, merges DOWN into the layer below (a fast-forward); only the bottom layer squash-merges into the real base. See ~/adocs/review-stack.md. Rule: match-or-improve master's dependency defenses — NoAssets selection when DB deps unavailable (7efdd1d7's outcome via the DI seam), requirements warning before assets imports, blake3 in the guarded dependency set Question: does each degradation path now match or improve master's behavior? Source tip: ebc2cfeebc Merge-base: 7fdfb40f4b * fix(assets): only discard content rows this operation actually inserted CR-9: Enumerated all six create_content call sites. Only scanner seeding and the three ingest registration paths track IDs for failure cleanup. * fix(assets): reject hash-only uploads with FEATURE_DISABLED when hashing is off CodeRabbit finding CR-2: reject hash-only multipart uploads before create_from_hash when hashing is disabled. * fix(assets): seed persists the stat it verified CR-7: persist the fresh seed-time restat instead of walk-time spec values. * fix(assets): route database lock failures to the lock guidance CR-16: route file-lock startup failures through the existing lock guidance and exit path. * fix(assets): drop the inaccurate temp-cleanup claim from the shutdown warning References CR-10. * fix(assets): walk the output root after execution so undeclared outputs register promptly Custom nodes that write files into the output directory without declaring them in output_ui only became assets when the next full walk happened - a frontend GET /object_info or a restart. Headless and API-only sessions never trigger either, so those files never converged into the asset database. The post-execution hook now requests a FULL scan of the output root instead of an enrich-only pass. The seeder's pending-request queue was generalised from enrich-specific to carrying a scan phase, so the request starts immediately when the seeder is idle and coalesces (escalating to FULL on a phase mismatch) when a scan is already running. queue_output_enrichment is renamed to queue_output_scan across the protocol, the NoAssets no-op and the call site. References FIX-6. * chore(assets): remove seeder paths orphaned by the output-scan change 45c2f96e rerouted both former enrich call sites to start()/enqueue_scan(), leaving two seeder methods that look live but are not. Review round F2 raised this along with four smaller items; the user's disposition was to fix all six here. - Delete start_enrich: zero callers repo-wide after 45c2f96e. - Delete enqueue_enrich: no production callers; its ~18 call sites in tests/test_asset_seeder.py move to enqueue_scan(phase=ScanPhase.ENRICH) with their semantics unchanged. The deletion forces the half-done class renames (TestEnqueueEnrich* -> TestEnqueueScan*, consistent with the already-renamed TestPendingScanDrain) and restores the module docstring that was dropped rather than reworded. - Document at manager.queue_output_scan that ScanPhase.FULL per debounce window is the deliberate, user-ratified trade, so it is not optimised back to ENRICH without revisiting the decision. - Document that SeedAssetSpec.size_bytes/mtime_ns are walk-time diagnostics only - production persists the seed-time restat since CR-7. - Export create_content_reporting_insert from the queries facade and fold scanner.py's direct-module import into the existing facade block. - Harden test_queue_output_scan_does_not_duplicate_declared_output against a vacuous pass: it now asserts the seeder finished without errors and that an undeclared sibling written into the same directory WAS registered by the same scan, proving the walk actually ran. No production behaviour changes beyond the two deletions. References F2-cleanup. * chore: comment cleanup Comment-Gate: 18 quarantined * fix(assets): preserve pause across the seeder's pending-scan drain pause() runs before every prompt, while pending-scan enqueue and resume only run inside the debounced gc-interval gate. If the active scan finishes just after the next prompt's pause, its finally block resets the seeder to idle and the pending drain starts a replacement with the run gate open, so resume becomes a no-op. Capture pausedness under the lock before resetting to idle, then start the drained scan already paused. Setting the state and gate before launching the thread avoids the start-then-reclear window and lets resume release the existing scan checkpoints. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * test(assets): pin job_id absence for scan-discovered assets Owner ruling, recorded 2026-09-03 in the stack-9-hardening planning notepad: scan-discovered assets — including undeclared outputs found by the post-execution walk — carry job_id = None, always; only emission-time registration (output_ui declaration) attributes a job; attributing walk finds to the most recent prompt would be a temporal-correlation guess that is wrong exactly when prompts interleave; None is honest provenance. Do NOT add proximity-based attribution heuristics to the scanner. Ratified against Jacob Segal's cross-job-attribution concern (2026-09-08 review meeting) — a wrongly-attributed asset could mean one user's cloud job sees another user's asset. Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 10/10] assets-tests (#16218) * test(execution): run the battery with assets enabled and assert asset-system health at teardown * test(execution): cover list-shaped outputs registering assets Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent) Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> * [review-stack 11/11] review-fixes (#16261) * fix(assets): only exit on database file-lock timeout when assets are enabled * test(assets): pin live_contents_under_prefixes path-filtering semantics * perf(assets): push live-content prefix filtering into SQL * test(assets): declare per-entry intent in the path-prefix corpus * test(assets): normalize POSIX-literal path expectations for Windows * test(assets): force observable stat changes and close-before-mutate on Windows-sensitive rewrites * test(assets): force an observable mtime change in the hash-mode split test --------- Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai> Co-authored-by: guill <jacob.e.segal@gmail.com>	2026-09-13 12:04:51 -07:00
Constantineandguill	5f0c4e18cb	Fix default database path for custom user directory (#14539) ... * fix: respect user directory for default database * refactor: use None default for --database-url instead of argv scan Per review: default --database-url to None and treat a non-None value as explicit at resolution time. Removes the sys.argv scan and the database_url_explicit attribute. database_default_path now serves directly as the legacy copy source. Adds regression tests for the unchanged no-flag default path and explicit URLs at the legacy location. * refactor: rename legacy database to .bak after copy Per review: after copying the legacy install-dir database to the effective user directory, rename the original to comfyui.db.bak so a later launch without --user-directory cannot silently fall back to a diverged copy, while keeping the file around for recovery. Also hoist the database_default_path import to module scope. * fix: guard legacy migration on existing .bak and rename before copy Per review: bail out of the legacy migration when comfyui.db.bak already exists, so only the first run migrates and later launches with a fresh --user-directory cannot grab a database another instance is using. Rename before copy so os.replace fails fast if the legacy DB is held open by a running instance. --------- Co-authored-by: guill <jacob.e.segal@gmail.com>	2026-08-24 16:19:03 -07:00
3 people	2bd4d82b4f	feat(assets): align local API with cloud spec (#12863) ... * feat(assets): align local API with cloud spec Unify response models, add missing fields, and align input schemas with the cloud OpenAPI spec at cloud.comfy.org/openapi. - Replace AssetSummary/AssetDetail/AssetUpdated with single Asset model - Add is_immutable, metadata (system_metadata), prompt_id fields - Support mime_type and preview_id in update endpoint - Make CreateFromHashBody.name optional, add mime_type, require >=1 tag - Add id/mime_type/preview_id to upload, relax tags to optional - Rename total_tags → tags in tag add/remove responses - Add GET /api/assets/tags/refine histogram endpoint - Add DB migration for system_metadata and prompt_id columns Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix review issues: tags validation, size nullability, type annotation, hash mismatch check, and add tag histogram tests - Remove contradictory min_length=1 from CreateFromHashBody.tags default - Restore size field to int|None=None for proper null semantics - Add Union type annotation to _build_asset_response result param - Add hash mismatch validation on idempotent upload path (409 HASH_MISMATCH) - Add unit tests for list_tag_histogram service function Amp-Thread-ID: https://ampcode.com/threads/T-019cd993-f43c-704e-b3d7-6cfc3d4d4a80 Co-authored-by: Amp <amp@ampcode.com> * Add preview_url to /assets API response using /api/view endpoint For input and output assets, generate a preview_url pointing to the existing /api/view endpoint using the asset's filename and tag-derived type (input/output). Handles subdirectories via subfolder param and URL-encodes filenames with spaces, unicode, and special characters. This aligns the OSS backend response with the frontend AssetCard expectation for thumbnail rendering. Amp-Thread-ID: https://ampcode.com/threads/T-019cda3f-5c2c-751a-a906-ac6c9153ac5c Co-authored-by: Amp <amp@ampcode.com> * chore: remove unused imports from asset_reference queries Amp-Thread-ID: https://ampcode.com/threads/T-019cda7d-cb21-77b4-a51b-b965af60208c Co-authored-by: Amp <amp@ampcode.com> * feat: resolve blake3 hashes in /view endpoint via asset database Amp-Thread-ID: https://ampcode.com/threads/T-019cda7d-cb21-77b4-a51b-b965af60208c Co-authored-by: Amp <amp@ampcode.com> * Register uploaded images in asset database when --enable-assets is set Add register_file_in_place() service function to ingest module for registering already-saved files without moving them. Call it from the /upload/image endpoint to return asset metadata in the response. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Exclude None fields from asset API JSON responses Add exclude_none=True to model_dump() calls across asset routes to keep response payloads clean by omitting unset optional fields. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Add comment explaining why /view resolves blake3 hashes Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Move blake3 hash resolution to asset_management service Extract resolve_hash_to_path() into asset_management.py and remove _resolve_blake3_to_path from server.py. Also revert loopback origin check to original logic. Amp-Thread-ID: https://ampcode.com/threads/T-019ce023-3384-7560-bacf-de40b0de0dd2 Co-authored-by: Amp <amp@ampcode.com> * Require at least one tag in UploadAssetSpec Enforce non-empty tags at the Pydantic validation layer so uploads with no tags are rejected with a 400 before reaching ingest. Adds test_upload_empty_tags_rejected to cover this case. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Add owner_id check to resolve_hash_to_path Filter asset references by owner visibility so the /view endpoint only resolves hashes for assets the requesting user can access. Adds table-driven tests for owner visibility cases. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Make ReferenceData.created_at and updated_at required Remove None defaults and type: ignore comments. Move fields before optional fields to satisfy dataclass ordering. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Fix double commit in create_from_hash Move mime_type update into _register_existing_asset so it shares a single transaction with reference creation. Log a warning when the hash is not found instead of silently returning None. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Add exclude_none=True to create/upload responses Align with get/update/list endpoints for consistent JSON output. Amp-Thread-ID: https://ampcode.com/threads/T-019ce377-8bde-7048-bc28-a9df063409f9 Co-authored-by: Amp <amp@ampcode.com> * Change preview_id to reference asset by reference ID, not content ID Clients receive preview_id in API responses but could not dereference it through public routes (which use reference IDs). Now preview_id is a self-referential FK to asset_references.id so the value is directly usable in the public API. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Filter soft-deleted and missing refs from visibility queries list_references_by_asset_id and list_tags_with_usage were not filtering out deleted_at/is_missing refs, allowing /view?filename=blake3:... to serve files through hidden references and inflating tag usage counts. Add list_all_file_paths_by_asset_id for orphan cleanup which intentionally needs unfiltered access. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Pass preview_id and mime_type through all asset creation fast paths The duplicate-content upload path and hash-based creation paths were silently dropping preview_id and mime_type. This wires both fields through _register_existing_asset, create_from_hash, and all route call sites so behavior is consistent regardless of whether the asset content already exists. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Remove unimplemented client-provided ID from upload API The `id` field on UploadAssetSpec was advertised for idempotent creation but never actually honored when creating new references. Remove it rather than implementing the feature. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Make asset mime_type immutable after first ingest Prevents cross-tenant metadata mutation when multiple references share the same content-addressed Asset row. mime_type can now only be set when NULL (first ingest); subsequent attempts to change it are silently ignored. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use resolved content_type from asset lookup in /view endpoint The /view endpoint was discarding the content_type computed by resolve_hash_to_path() and re-guessing from the filename, which produced wrong results for extensionless files or mismatched extensions. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Merge system+user metadata into filter projection Extract rebuild_metadata_projection() to build AssetReferenceMeta rows from {**system_metadata, **user_metadata}, so system-generated metadata is queryable via metadata_filter and user keys override system keys. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Standardize tag ordering to alphabetical across all endpoints Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Derive subfolder tags from path in register_file_in_place Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Reject client-provided id, fix preview URLs, rename tags→total_tags - Reject 'id' field in multipart upload with 400 UNSUPPORTED_FIELD instead of silently ignoring it - Build preview URL from the preview asset's own metadata rather than the parent asset's - Rename 'tags' to 'total_tags' in TagsAdd/TagsRemove response schemas for clarity Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: SQLite migration 0003 FK drop fails on file-backed DBs (MB-2) Add naming_convention to Base.metadata so Alembic batch-mode reflection can match unnamed FK constraints created by migration 0002. Pass naming_convention and render_as_batch=True through env.py online config. Add migration roundtrip tests (upgrade/downgrade/cycle from baseline). Amp-Thread-ID: https://ampcode.com/threads/T-019ce466-1683-7471-b6e1-bb078223cda0 Co-authored-by: Amp <amp@ampcode.com> * Fix missing tag count for is_missing references and update test for total_tags field - Allow is_missing=True references to be counted in list_tags_with_usage when the tag is 'missing', so the missing tag count reflects all references that have been tagged as missing - Add update_is_missing_by_asset_id query helper for bulk updates by asset - Update test_add_and_remove_tags to use 'total_tags' matching the API schema Amp-Thread-ID: https://ampcode.com/threads/T-019ce482-05e7-7324-a1b0-a56a929cc7ef Co-authored-by: Amp <amp@ampcode.com> * Remove unused imports in scanner.py Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Rename prompt_id to job_id on asset_references Rename the column in the DB model, migration, and service schemas. The API response emits both job_id and prompt_id (deprecated alias) for backward compatibility with the cloud API. Amp-Thread-ID: https://ampcode.com/threads/T-019cef41-60b0-752a-aa3c-ed7f20fda2f7 Co-authored-by: Amp <amp@ampcode.com> * Add index on asset_references.preview_id for FK cascade performance Amp-Thread-ID: https://ampcode.com/threads/T-019cef45-a4d2-7548-86d2-d46bcd3db419 Co-authored-by: Amp <amp@ampcode.com> * Add clarifying comments for Asset/AssetReference naming and preview_id Amp-Thread-ID: https://ampcode.com/threads/T-019cef49-f94e-7348-bf23-9a19ebf65e0d Co-authored-by: Amp <amp@ampcode.com> * Disallow all-null meta rows: add CHECK constraint, skip null values on write - convert_metadata_to_rows returns [] for None values instead of an all-null row - Remove dead None branch from _scalar_to_row - Simplify null filter in common.py to just check for row absence - Add CHECK constraint ck_asset_reference_meta_has_value to model and migration 0003 Amp-Thread-ID: https://ampcode.com/threads/T-019cef4e-5240-7749-bb25-1f17fcf9c09c Co-authored-by: Amp <amp@ampcode.com> * Remove dead None guards on result.asset in upload handler register_file_in_place guarantees a non-None asset, so the 'if result.asset else None' checks were unreachable. Amp-Thread-ID: https://ampcode.com/threads/T-019cef5b-4cf8-723c-8a98-8fb8f333c133 Co-authored-by: Amp <amp@ampcode.com> * Remove mime_type from asset update API Clients can no longer modify mime_type after asset creation via the PUT /api/assets/{id} endpoint. This reduces the risk of mime_type spoofing. The internal update_asset_hash_and_mime function remains available for server-side use (e.g., enrichment). Amp-Thread-ID: https://ampcode.com/threads/T-019cef5d-8d61-75cc-a1c6-2841ac395648 Co-authored-by: Amp <amp@ampcode.com> * Fix migration constraint naming double-prefix and NULL in mixed metadata lists - Use fully-rendered constraint names in migration 0003 to avoid the naming convention doubling the ck_ prefix on batch operations. - Add table_args to downgrade so SQLite batch mode can find the CHECK constraint (not exposed by SQLite reflection). - Fix model CheckConstraint name to use bare 'has_value' (convention auto-prefixes). - Skip None items when converting metadata lists to rows, preventing all-NULL rows that violate the has_value check constraint. Amp-Thread-ID: https://ampcode.com/threads/T-019cef87-94f9-7172-a6af-c6282290ce4f Co-authored-by: Amp <amp@ampcode.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> Co-authored-by: Amp <amp@ampcode.com>	2026-03-16 12:34:04 -07:00
Luke Mino-Altherr	29b24cb517	refactor(assets): modular architecture + async two-phase scanner & background seeder (#12621)	2026-03-07 20:37:25 -05:00
Jedrzej Kosinski	1dc3da6314	Add most basic Asset support for models (#11315) ... * Brought over minimal elements from PR 10045 to reproduce seed_assets and register_assets_system without adding anything to the DB or server routes yet, for now making everything sync (can introduce async once everything is cleaned up and brought over) * Added db script to insert assets stuff, cleaned up some code; assets (models) now get added/rescanned * Added support for 5 http endpoints for assets * Replaced Optional with | None in schemas_in.py and schemas_out.py * Remove two routes that will not be relevant yet in this PR: HEAD /api/assets/hash/<hash> and PUT /api/assets/<id>/preview * Remove some functions the two deleted endpoints were using * Don't show assets scan message upon calling /object_info endpoint * removed unsued import to satisfy ruff * Simplified hashing function tpye hint and _hash_file_obj * Satisfied ruff	2026-01-08 22:21:51 -05:00
pythongosssss	50c605e957	Add support for sqlite database (#8444) ... * Add support for sqlite database * fix	2025-06-11 16:43:39 -04:00