mirror of
https://github.com/pionxzh/chatgpt-exporter.git
synced 2026-07-23 09:00:51 -05:00
462836d1b2
* feat(export): batch export in 100-conv waves and date-range filter Closes #347 (feature request on upstream pionxzh/chatgpt-exporter). Changes: - src/constants.ts: add EXPORT_OPERATION_BATCH = 100 (fixed, non-tunable) - src/utils/download.ts: add PartInfo type, part-suffix support to buildZipFileName, and new buildJsonBatchFileName for official JSON - src/exporter/{html,json,markdown}.ts: accept optional partIndex / totalParts params; each exportAllTo* uses PartInfo to suffix its output filename when multiple batches are produced - src/ui/ExportDialog.tsx: - Remove hardcoded EXPORT_LIMIT = 100 selection cap; users may now select all loaded conversations (up to exportAllLimit from settings) - Add DateFilter component with from/to date inputs that filter the conversation list by create_time (inclusive, end-of-day for dateTo) - Wave-based export pipeline: splits selected conversations into chunks of EXPORT_OPERATION_BATCH (100), fetches each chunk via RequestQueue, downloads the completed batch, pauses 400 ms, then continues with the next chunk — keeping memory bounded and API pressure low - Progress bar now shows global count across all batches plus "Batch X/N" indicator when multiple batches are active - Local-file source also batched in 100-conversation waves with the same 400 ms inter-batch delay - Shift-click range selection and Select All no longer impose a hard cap (handled by the batch pipeline instead) - src/locales/en.json: add Date Filter Label/Hint, Selected count, Export batch info, Exporting batch strings; update Export All Limit Description to mention 100-conversation waves Made-with: Cursor * fix(export): correct date filter for ISO string timestamps + add update_time field The list endpoint returns create_time/update_time as ISO 8601 strings (e.g. "2025-07-10T14:53:58.103234Z"), not Unix numbers. The previous filter compared a string to a number, always producing NaN comparisons and filtering out every conversation. - Add toMs() helper that handles both ISO strings and legacy Unix numbers - Fix filter comparisons to use toMs() and compare in milliseconds - Update ApiConversationItem type: create_time is number|string, update_time is optional number|string - Add filterField state ('create_time' | 'update_time') with a Created/Updated selector in the DateFilter UI so users can filter by either timestamp - Add i18n strings for the new selector labels Made-with: Cursor * feat: redesign date filter layout with presets, auto-load conversations, batch label on export - Date filter redesigned to two-row layout: field selector + quick preset buttons (7d / 30d / 90d / This year / Clear) on row 1; From–To date inputs on row 2, eliminating the wrapping issue - Auto-load all conversations when dialog opens (default project null instead of undefined) so search box, Last 100 button, and counter are immediately visible without needing to manually pick a project - Export button shows batch count when selection > 100, e.g. "Export · 3 files" with a tooltip explaining "3 separate downloads, 100 conversations each" - Progress display shows "Batch X/Y · completed/total" for multi-batch exports - Remove redundant "Export from API" static text (project selector implies it) - Add locale strings: Date Preset 7d/30d/90d/Year, Clear filter, Batch progress Made-with: Cursor * fix: show batch download count inline below export button Made-with: Cursor * feat: filter chips, date column, All conversations, starred indicator - Filter chips via # trigger: type # in search box to open a popover with available filters; click to add as a removable chip: ★ Starred only | 💾 Saved only (skip temp) | ⏱ Long conversation (active ≥ N days, editable inline) | 🤖 GPT/custom AI | 💬 Regular chats - Date column: each conversation row now shows a compact relative date (Today / Yesterday / Xd ago / Mon DD / MMM YYYY) on the right edge - Starred indicator: ★ shown inline for starred conversations - Project dropdown: new "📂 All conversations" option fetches the main list plus every project's conversations, deduplicated; uses new fetchAllConversationsAll() API function - ApiConversationItem: add is_starred, is_temporary_chat, gizmo_id fields from the list endpoint payload - CSS: SelectItem flex layout, new SelectChip / SelectChips / SelectFilterPopover / SelectItemMeta classes Made-with: Cursor * feat: add Load More button to fetch conversations beyond the initial limit - Export fetchConversationsPage from api.ts for single-page incremental loading - Add onHasMore callback to fetchAllConversations so callers know when the fetch was stopped by the user-configured limit (vs the API having no more data) - DialogContent tracks hasMore / totalAvailable / loadingMore state - A "Load N more" button appears below the conversation list when more pages exist; shows "N remaining" count once the API total is known - Resets cleanly on project change Made-with: Cursor * feat: comprehensive filter chip system with chat class, origin, status, and mode logic - Add conversation_origin, pinned_time, is_archived, is_do_not_remember to ApiConversationItem - Replace simple chip flags with typed FilterChipDef union: chat_class (Regular/GPT/Project), origin (dynamic from data), status (starred/temporary/pinned), duration_gte, recency_lte - Each chip has an include/exclude mode toggle (click badge to flip) - AND/OR logic toggle appears in chip bar when 2+ chips are active - Text search supports * and ? wildcards alongside chips - Picker grouped by category with dynamic Origin values from loaded conversations - Project classification cross-references fetched project IDs against gizmo_id - Remove ProjectSelect dropdown: always auto-load all conversations on open - Chat class chip: Regular (no gizmo), GPT (store gizmo), Project (matched gizmo) Made-with: Cursor * refactor(filters): redesign chips as multi-select with two-stage picker - Chat class, Origin, Project, Status are now single chips that open a sub-view where users tick multiple values (Regular + GPT, etc.) - New Project chip lets users pick specific project(s) by name - Date filter moved from top-of-dialog into the chip bar as a full-width date-range chip (Created/Updated selector + date inputs + presets) - Picker uses two-stage UX: root list → sub-view with checkboxes + Apply - Clicking a chip's value label reopens the picker for in-place editing - AND/OR logic toggle only appears for 2+ non-date chips - Remove standalone DateFilter component and related state from DialogContent Made-with: Cursor * fix(filters): guard gizmo_id null check for project chip Made-with: Cursor * fix: prevent export dialog close mid-run and add resume-from-offset control - Block ESC, outside-click, and X button while an export is in progress; module-level exportingRef bridges DialogContent to ExportDialog without lifting state. - Stale-fetch prevention: fetchGenRef generation counter discards callbacks from a previous dialog mount after remount, eliminating the console errors seen when reopening the dialog mid-run. - Cleanup on unmount: clear all three RequestQueues when DialogContent unmounts so orphaned background work stops immediately. - Add "→ 100 from #N" resume control in the toolbar: a number input sets the starting offset and the arrow button selects the next 100 convs from that position, letting users continue a partial export (e.g. set offset to 200 to resume after 2 finished batches). Made-with: Cursor * perf: use Web Worker for sleep() to bypass background-tab timer throttle Chrome throttles setTimeout in non-focused tabs to >=1 s minimum, which causes the export queue (200 ms between requests * 100 convs = 20 s/batch) to stall entirely when the user switches away from the tab. Fix: run all timers inside a single persistent Web Worker. Workers are exempt from background-tab throttling, so exports run at full speed regardless of tab focus. A graceful fallback to regular setTimeout is retained for environments where blob-URL Worker creation fails (CSP etc.) so the change is entirely non-breaking. Made-with: Cursor * fix: proper 429 backoff — wait Retry-After secs, cap retries, skip when exhausted Previously the queue retried forever with a max backoff of 1600 ms, causing the 429 storm seen in the console (the same request hammering the API continuously with no meaningful pause). Changes: - api.ts: export RateLimitError with retryAfterMs read from Retry-After header (defaults to 30 s when header is absent or unparseable) - queue.ts: on RateLimitError, sleep max(retryAfterMs, 30 s) before retrying; give up after 8 consecutive 429s and skip the request; on regular errors give up after 5 retries and skip; add rate_limited status so the UI can show what is happening - ExportDialog.tsx: show "Rate limited — waiting Xs…" with an amber progress bar when status is rate_limited; clear progress bar to blue once the pause ends Made-with: Cursor * fix: global queue pause on 429 instead of per-item backoff The previous design retried each item individually with a 30s wait, so if all 100 conversations in a batch were rate-limited the theoretical worst case was 100 x 8 x 30s = 24,000s. New design: - On the first 429 the entire queue is frozen (pauseUntil timestamp). Every subsequent process() call checks pauseUntil and waits out the remaining time before making any new request — one shared wait for all queued items, not N separate waits. - The pause length is max(Retry-After header, 60s * pauseCount) so it backs off exponentially if the first pause was not long enough: pause 1 = 60s, pause 2 = 120s, pause 3 = 180s, etc. - After MAX_GLOBAL_PAUSES (5) the queue aborts rather than looping forever (total max wait ~15 min before giving up). - The failed item is always returned to the front of the queue to be retried after the pause clears rather than being counted as a skip. - Removed per-item rateRetries counter (no longer needed). Made-with: Cursor * feat: add Test API button and rate-limit header logging - probeApi(): make one minimal request (1 conversation) to check whether the API is currently accepting traffic. Returns ok/rate_limited status plus any X-RateLimit-* or Retry-After headers in the response. - fetchApi(): logs all known rate-limit response headers to console on every call (success or failure) so we can discover which headers ChatGPT actually sends. - ExportDialog: "Test API" button next to the upload icon. Shows "Testing... / API ready / Rate limited · wait Xs / Error" inline. Hovering the button after a successful test shows any rate-limit headers as a tooltip (useful for debugging). Made-with: Cursor * fix: add Cancel button during export; replace useless dimmed X Previously the X button was grayed out and non-functional while an export was running, with no way to stop it short of refreshing the page. Changes: - Cancel button (red, top-right) appears while processing. Clicking it sets cancelledRef, calls queue.stop() on all three queues, and lets the in-flight request finish before cleanly stopping the loop. - The on('done') handler checks cancelledRef: if set, it skips the download and the next-batch kickoff and just resets processing state. - cancelledRef is cleared at the start of each new export so back-to-back exports work correctly. - The dimmed X is kept (slightly more transparent) as a visual reminder that the dialog cannot be closed mid-export; its tooltip now says "Click Cancel to stop the export". Made-with: Cursor * feat: sortable column headers, both date columns, unambiguous date format Problems fixed: - Date column only showed create_time with no label — confusing when the user had filtered or sorted by update_time. - Dates < 1 year omitted the year ("Jun 17" vs "Jun 17, 2025"), making it impossible to tell which year a conversation belonged to. - No way to sort the list — order was whatever the API returned. Changes: - formatConvDate: always includes the year (removes the < 365-day branch). Still shows "Today" / "Yesterday" for very recent items. - ConversationSelect: two date columns (Created, Updated). The active-sort column is visually highlighted (bold, blue) so the user always knows what order the list is in. - Sortable header row (SelectListHeader): click Title / Created / Updated to sort; click again to toggle asc/desc. Arrow indicator (↑/↓/↕) shows current state. Defaults to Created ↓ (newest first) matching the original API order. - Sorting applied in the filtered memo after chip/text filtering. Made-with: Cursor --------- Co-authored-by: Pionxzh <spigbbbbb@gmail.com>