mirror of
https://github.com/pionxzh/chatgpt-exporter.git
synced 2026-07-23 09:00:51 -05:00
feat: batched export, smart rate-limit backoff, filter chips, and dialog UX improvements (#348)
* feat(export): batch export in 100-conv waves and date-range filter Closes #347 (feature request on upstream pionxzh/chatgpt-exporter). Changes: - src/constants.ts: add EXPORT_OPERATION_BATCH = 100 (fixed, non-tunable) - src/utils/download.ts: add PartInfo type, part-suffix support to buildZipFileName, and new buildJsonBatchFileName for official JSON - src/exporter/{html,json,markdown}.ts: accept optional partIndex / totalParts params; each exportAllTo* uses PartInfo to suffix its output filename when multiple batches are produced - src/ui/ExportDialog.tsx: - Remove hardcoded EXPORT_LIMIT = 100 selection cap; users may now select all loaded conversations (up to exportAllLimit from settings) - Add DateFilter component with from/to date inputs that filter the conversation list by create_time (inclusive, end-of-day for dateTo) - Wave-based export pipeline: splits selected conversations into chunks of EXPORT_OPERATION_BATCH (100), fetches each chunk via RequestQueue, downloads the completed batch, pauses 400 ms, then continues with the next chunk — keeping memory bounded and API pressure low - Progress bar now shows global count across all batches plus "Batch X/N" indicator when multiple batches are active - Local-file source also batched in 100-conversation waves with the same 400 ms inter-batch delay - Shift-click range selection and Select All no longer impose a hard cap (handled by the batch pipeline instead) - src/locales/en.json: add Date Filter Label/Hint, Selected count, Export batch info, Exporting batch strings; update Export All Limit Description to mention 100-conversation waves Made-with: Cursor * fix(export): correct date filter for ISO string timestamps + add update_time field The list endpoint returns create_time/update_time as ISO 8601 strings (e.g. "2025-07-10T14:53:58.103234Z"), not Unix numbers. The previous filter compared a string to a number, always producing NaN comparisons and filtering out every conversation. - Add toMs() helper that handles both ISO strings and legacy Unix numbers - Fix filter comparisons to use toMs() and compare in milliseconds - Update ApiConversationItem type: create_time is number|string, update_time is optional number|string - Add filterField state ('create_time' | 'update_time') with a Created/Updated selector in the DateFilter UI so users can filter by either timestamp - Add i18n strings for the new selector labels Made-with: Cursor * feat: redesign date filter layout with presets, auto-load conversations, batch label on export - Date filter redesigned to two-row layout: field selector + quick preset buttons (7d / 30d / 90d / This year / Clear) on row 1; From–To date inputs on row 2, eliminating the wrapping issue - Auto-load all conversations when dialog opens (default project null instead of undefined) so search box, Last 100 button, and counter are immediately visible without needing to manually pick a project - Export button shows batch count when selection > 100, e.g. "Export · 3 files" with a tooltip explaining "3 separate downloads, 100 conversations each" - Progress display shows "Batch X/Y · completed/total" for multi-batch exports - Remove redundant "Export from API" static text (project selector implies it) - Add locale strings: Date Preset 7d/30d/90d/Year, Clear filter, Batch progress Made-with: Cursor * fix: show batch download count inline below export button Made-with: Cursor * feat: filter chips, date column, All conversations, starred indicator - Filter chips via # trigger: type # in search box to open a popover with available filters; click to add as a removable chip: ★ Starred only | 💾 Saved only (skip temp) | ⏱ Long conversation (active ≥ N days, editable inline) | 🤖 GPT/custom AI | 💬 Regular chats - Date column: each conversation row now shows a compact relative date (Today / Yesterday / Xd ago / Mon DD / MMM YYYY) on the right edge - Starred indicator: ★ shown inline for starred conversations - Project dropdown: new "📂 All conversations" option fetches the main list plus every project's conversations, deduplicated; uses new fetchAllConversationsAll() API function - ApiConversationItem: add is_starred, is_temporary_chat, gizmo_id fields from the list endpoint payload - CSS: SelectItem flex layout, new SelectChip / SelectChips / SelectFilterPopover / SelectItemMeta classes Made-with: Cursor * feat: add Load More button to fetch conversations beyond the initial limit - Export fetchConversationsPage from api.ts for single-page incremental loading - Add onHasMore callback to fetchAllConversations so callers know when the fetch was stopped by the user-configured limit (vs the API having no more data) - DialogContent tracks hasMore / totalAvailable / loadingMore state - A "Load N more" button appears below the conversation list when more pages exist; shows "N remaining" count once the API total is known - Resets cleanly on project change Made-with: Cursor * feat: comprehensive filter chip system with chat class, origin, status, and mode logic - Add conversation_origin, pinned_time, is_archived, is_do_not_remember to ApiConversationItem - Replace simple chip flags with typed FilterChipDef union: chat_class (Regular/GPT/Project), origin (dynamic from data), status (starred/temporary/pinned), duration_gte, recency_lte - Each chip has an include/exclude mode toggle (click badge to flip) - AND/OR logic toggle appears in chip bar when 2+ chips are active - Text search supports * and ? wildcards alongside chips - Picker grouped by category with dynamic Origin values from loaded conversations - Project classification cross-references fetched project IDs against gizmo_id - Remove ProjectSelect dropdown: always auto-load all conversations on open - Chat class chip: Regular (no gizmo), GPT (store gizmo), Project (matched gizmo) Made-with: Cursor * refactor(filters): redesign chips as multi-select with two-stage picker - Chat class, Origin, Project, Status are now single chips that open a sub-view where users tick multiple values (Regular + GPT, etc.) - New Project chip lets users pick specific project(s) by name - Date filter moved from top-of-dialog into the chip bar as a full-width date-range chip (Created/Updated selector + date inputs + presets) - Picker uses two-stage UX: root list → sub-view with checkboxes + Apply - Clicking a chip's value label reopens the picker for in-place editing - AND/OR logic toggle only appears for 2+ non-date chips - Remove standalone DateFilter component and related state from DialogContent Made-with: Cursor * fix(filters): guard gizmo_id null check for project chip Made-with: Cursor * fix: prevent export dialog close mid-run and add resume-from-offset control - Block ESC, outside-click, and X button while an export is in progress; module-level exportingRef bridges DialogContent to ExportDialog without lifting state. - Stale-fetch prevention: fetchGenRef generation counter discards callbacks from a previous dialog mount after remount, eliminating the console errors seen when reopening the dialog mid-run. - Cleanup on unmount: clear all three RequestQueues when DialogContent unmounts so orphaned background work stops immediately. - Add "→ 100 from #N" resume control in the toolbar: a number input sets the starting offset and the arrow button selects the next 100 convs from that position, letting users continue a partial export (e.g. set offset to 200 to resume after 2 finished batches). Made-with: Cursor * perf: use Web Worker for sleep() to bypass background-tab timer throttle Chrome throttles setTimeout in non-focused tabs to >=1 s minimum, which causes the export queue (200 ms between requests * 100 convs = 20 s/batch) to stall entirely when the user switches away from the tab. Fix: run all timers inside a single persistent Web Worker. Workers are exempt from background-tab throttling, so exports run at full speed regardless of tab focus. A graceful fallback to regular setTimeout is retained for environments where blob-URL Worker creation fails (CSP etc.) so the change is entirely non-breaking. Made-with: Cursor * fix: proper 429 backoff — wait Retry-After secs, cap retries, skip when exhausted Previously the queue retried forever with a max backoff of 1600 ms, causing the 429 storm seen in the console (the same request hammering the API continuously with no meaningful pause). Changes: - api.ts: export RateLimitError with retryAfterMs read from Retry-After header (defaults to 30 s when header is absent or unparseable) - queue.ts: on RateLimitError, sleep max(retryAfterMs, 30 s) before retrying; give up after 8 consecutive 429s and skip the request; on regular errors give up after 5 retries and skip; add rate_limited status so the UI can show what is happening - ExportDialog.tsx: show "Rate limited — waiting Xs…" with an amber progress bar when status is rate_limited; clear progress bar to blue once the pause ends Made-with: Cursor * fix: global queue pause on 429 instead of per-item backoff The previous design retried each item individually with a 30s wait, so if all 100 conversations in a batch were rate-limited the theoretical worst case was 100 x 8 x 30s = 24,000s. New design: - On the first 429 the entire queue is frozen (pauseUntil timestamp). Every subsequent process() call checks pauseUntil and waits out the remaining time before making any new request — one shared wait for all queued items, not N separate waits. - The pause length is max(Retry-After header, 60s * pauseCount) so it backs off exponentially if the first pause was not long enough: pause 1 = 60s, pause 2 = 120s, pause 3 = 180s, etc. - After MAX_GLOBAL_PAUSES (5) the queue aborts rather than looping forever (total max wait ~15 min before giving up). - The failed item is always returned to the front of the queue to be retried after the pause clears rather than being counted as a skip. - Removed per-item rateRetries counter (no longer needed). Made-with: Cursor * feat: add Test API button and rate-limit header logging - probeApi(): make one minimal request (1 conversation) to check whether the API is currently accepting traffic. Returns ok/rate_limited status plus any X-RateLimit-* or Retry-After headers in the response. - fetchApi(): logs all known rate-limit response headers to console on every call (success or failure) so we can discover which headers ChatGPT actually sends. - ExportDialog: "Test API" button next to the upload icon. Shows "Testing... / API ready / Rate limited · wait Xs / Error" inline. Hovering the button after a successful test shows any rate-limit headers as a tooltip (useful for debugging). Made-with: Cursor * fix: add Cancel button during export; replace useless dimmed X Previously the X button was grayed out and non-functional while an export was running, with no way to stop it short of refreshing the page. Changes: - Cancel button (red, top-right) appears while processing. Clicking it sets cancelledRef, calls queue.stop() on all three queues, and lets the in-flight request finish before cleanly stopping the loop. - The on('done') handler checks cancelledRef: if set, it skips the download and the next-batch kickoff and just resets processing state. - cancelledRef is cleared at the start of each new export so back-to-back exports work correctly. - The dimmed X is kept (slightly more transparent) as a visual reminder that the dialog cannot be closed mid-export; its tooltip now says "Click Cancel to stop the export". Made-with: Cursor * feat: sortable column headers, both date columns, unambiguous date format Problems fixed: - Date column only showed create_time with no label — confusing when the user had filtered or sorted by update_time. - Dates < 1 year omitted the year ("Jun 17" vs "Jun 17, 2025"), making it impossible to tell which year a conversation belonged to. - No way to sort the list — order was whatever the API returned. Changes: - formatConvDate: always includes the year (removes the < 365-day branch). Still shows "Today" / "Yesterday" for very recent items. - ConversationSelect: two date columns (Created, Updated). The active-sort column is visually highlighted (bold, blue) so the user always knows what order the list is in. - Sortable header row (SelectListHeader): click Title / Created / Updated to sort; click again to toggle asc/desc. Arrow indicator (↑/↓/↕) shows current state. Defaults to Created ↓ (newest first) matching the original API order. - Sorting applied in the filtered memo after chip/text filtering. Made-with: Cursor --------- Co-authored-by: Pionxzh <spigbbbbb@gmail.com>
This commit is contained in:
1814
dist/chatgpt.user.js
vendored
1814
dist/chatgpt.user.js
vendored
File diff suppressed because it is too large
Load Diff
148
src/api.ts
148
src/api.ts
@@ -291,7 +291,24 @@ export type ApiConversationWithId = ApiConversation & {
|
||||
export interface ApiConversationItem {
|
||||
id: string
|
||||
title: string
|
||||
create_time: number
|
||||
/** ISO 8601 string from the list endpoint (e.g. "2025-07-10T14:53:58.103234Z") */
|
||||
create_time: number | string
|
||||
/** ISO 8601 string from the list endpoint */
|
||||
update_time?: number | string
|
||||
/** True when the user has starred/pinned this conversation */
|
||||
is_starred?: boolean | null
|
||||
/** True for temporary chats that are not saved to history */
|
||||
is_temporary_chat?: boolean
|
||||
/** Non-null when the conversation belongs to a custom GPT or project */
|
||||
gizmo_id?: string | null
|
||||
/** How the conversation was initiated, e.g. "apple" for Siri/iOS, null for web/app */
|
||||
conversation_origin?: string | null
|
||||
/** ISO 8601 timestamp if the conversation is pinned, null otherwise */
|
||||
pinned_time?: string | null
|
||||
/** True when this conversation is archived */
|
||||
is_archived?: boolean
|
||||
/** True when memory is disabled for this conversation */
|
||||
is_do_not_remember?: boolean | null
|
||||
}
|
||||
|
||||
export interface ApiConversations {
|
||||
@@ -548,7 +565,20 @@ async function fetchProjectConversations(project: string, cursor: string | numbe
|
||||
}
|
||||
}
|
||||
|
||||
export async function fetchAllConversations(project: string | null = null, maxConversations = 1000, onBatch?: (batch: ApiConversationItem[]) => void): Promise<ApiConversationItem[]> {
|
||||
/**
|
||||
* Fetch a single page of conversations starting at `offset`.
|
||||
* Useful for "load more" functionality in the UI — call this after the initial
|
||||
* full load to append additional pages without re-fetching everything.
|
||||
*/
|
||||
export async function fetchConversationsPage(
|
||||
project: string | null,
|
||||
offset: number,
|
||||
limit: number,
|
||||
): Promise<ApiConversations> {
|
||||
return fetchConversations(offset, limit, project)
|
||||
}
|
||||
|
||||
export async function fetchAllConversations(project: string | null = null, maxConversations = 1000, onBatch?: (batch: ApiConversationItem[]) => void, onHasMore?: (hasMore: boolean) => void): Promise<ApiConversationItem[]> {
|
||||
const conversations: ApiConversationItem[] = []
|
||||
const limit = project === null ? 100 : 50 // gizmos api uses a smaller limit
|
||||
let offset = 0
|
||||
@@ -585,7 +615,34 @@ export async function fetchAllConversations(project: string | null = null, maxCo
|
||||
}
|
||||
}
|
||||
// Ensure we don't return more than the requested limit if the last batch pushed us over
|
||||
return conversations.slice(0, maxConversations)
|
||||
const result = conversations.slice(0, maxConversations)
|
||||
// Let the caller know whether the fetch was cut off by the user limit vs the API having no more data
|
||||
onHasMore?.(result.length >= maxConversations)
|
||||
return result
|
||||
}
|
||||
|
||||
/**
|
||||
* Fetch conversations from every source: the main conversation list (no-project)
|
||||
* plus each project's own list. Deduplicates by ID so a conversation that
|
||||
* appears in both won't be exported twice.
|
||||
*/
|
||||
export async function fetchAllConversationsAll(
|
||||
projects: ApiProjectInfo[],
|
||||
maxConversations = 1000,
|
||||
onBatch?: (batch: ApiConversationItem[]) => void,
|
||||
): Promise<void> {
|
||||
const seen = new Set<string>()
|
||||
const notify = (items: ApiConversationItem[]) => {
|
||||
const novel = items.filter(c => !seen.has(c.id))
|
||||
for (const c of novel) seen.add(c.id)
|
||||
if (novel.length > 0) onBatch?.(novel)
|
||||
}
|
||||
// Main conversation list first (no-project or all, depending on the API)
|
||||
await fetchAllConversations(null, maxConversations, notify)
|
||||
// Then each project's conversations
|
||||
for (const project of projects) {
|
||||
await fetchAllConversations(project.id, maxConversations, notify)
|
||||
}
|
||||
}
|
||||
|
||||
export async function archiveConversation(chatId: string): Promise<boolean> {
|
||||
@@ -608,6 +665,45 @@ export async function deleteConversation(chatId: string): Promise<boolean> {
|
||||
return success
|
||||
}
|
||||
|
||||
/**
|
||||
* Thrown when the API responds with 429 Too Many Requests.
|
||||
* Carries the wait time from the `Retry-After` header (or a safe default).
|
||||
*/
|
||||
export class RateLimitError extends Error {
|
||||
/** Milliseconds to wait before retrying */
|
||||
readonly retryAfterMs: number
|
||||
constructor(retryAfterHeader: string | null) {
|
||||
super('Too Many Requests (429)')
|
||||
this.name = 'RateLimitError'
|
||||
const secs = retryAfterHeader != null ? Number.parseInt(retryAfterHeader, 10) : Number.NaN
|
||||
// Default to 30 s if the header is missing or unparseable
|
||||
this.retryAfterMs = Number.isFinite(secs) && secs > 0 ? secs * 1000 : 30_000
|
||||
}
|
||||
}
|
||||
|
||||
/** Header names ChatGPT might use for rate-limit signalling */
|
||||
const RATE_LIMIT_HEADERS = [
|
||||
'retry-after',
|
||||
'x-ratelimit-limit-requests',
|
||||
'x-ratelimit-remaining-requests',
|
||||
'x-ratelimit-reset-requests',
|
||||
'x-ratelimit-limit-tokens',
|
||||
'x-ratelimit-remaining-tokens',
|
||||
'x-ratelimit-reset-tokens',
|
||||
]
|
||||
|
||||
function logRateLimitHeaders(response: Response) {
|
||||
const found: Record<string, string> = {}
|
||||
for (const h of RATE_LIMIT_HEADERS) {
|
||||
const val = response.headers.get(h)
|
||||
if (val != null) found[h] = val
|
||||
}
|
||||
if (Object.keys(found).length > 0) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.info('[Exporter] Rate-limit headers:', found)
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchApi<T>(url: string, options?: RequestInit): Promise<T> {
|
||||
const accessToken = await getAccessToken()
|
||||
const accountId = await getTeamAccountId()
|
||||
@@ -621,12 +717,58 @@ async function fetchApi<T>(url: string, options?: RequestInit): Promise<T> {
|
||||
...options?.headers,
|
||||
},
|
||||
})
|
||||
|
||||
logRateLimitHeaders(response)
|
||||
|
||||
if (!response.ok) {
|
||||
if (response.status === 429) {
|
||||
throw new RateLimitError(response.headers.get('Retry-After'))
|
||||
}
|
||||
throw new Error(response.statusText)
|
||||
}
|
||||
return response.json()
|
||||
}
|
||||
|
||||
/**
|
||||
* Lightweight probe: fetch exactly 1 conversation to check whether the API
|
||||
* is currently accepting requests. Returns a status object that the UI can
|
||||
* display before the user starts a large export.
|
||||
*/
|
||||
export async function probeApi(): Promise<{
|
||||
ok: boolean
|
||||
retryAfterMs?: number
|
||||
rateLimitHeaders: Record<string, string>
|
||||
}> {
|
||||
const accessToken = await getAccessToken()
|
||||
const accountId = await getTeamAccountId()
|
||||
const url = conversationsApi(0, 1)
|
||||
|
||||
const response = await fetch(url, {
|
||||
headers: {
|
||||
'Authorization': `Bearer ${accessToken}`,
|
||||
'X-Authorization': `Bearer ${accessToken}`,
|
||||
...(accountId ? { 'Chatgpt-Account-Id': accountId } : {}),
|
||||
},
|
||||
})
|
||||
|
||||
const rateLimitHeaders: Record<string, string> = {}
|
||||
for (const h of RATE_LIMIT_HEADERS) {
|
||||
const val = response.headers.get(h)
|
||||
if (val != null) rateLimitHeaders[h] = val
|
||||
}
|
||||
|
||||
if (!response.ok) {
|
||||
if (response.status === 429) {
|
||||
const secs = response.headers.get('retry-after')
|
||||
const ms = secs ? Number.parseInt(secs, 10) * 1000 : 60_000
|
||||
return { ok: false, retryAfterMs: ms, rateLimitHeaders }
|
||||
}
|
||||
return { ok: false, rateLimitHeaders }
|
||||
}
|
||||
|
||||
return { ok: true, rateLimitHeaders }
|
||||
}
|
||||
|
||||
async function _fetchSession(): Promise<ApiSession> {
|
||||
const response = await fetch(sessionApi)
|
||||
if (!response.ok) {
|
||||
|
||||
@@ -21,3 +21,10 @@ export const KEY_EXPORT_ALL_LIMIT = 'exporter:export_all_limit'
|
||||
|
||||
export const KEY_OAI_LOCALE = 'oai/apps/locale'
|
||||
export const KEY_OAI_HISTORY_DISABLED = 'oai/apps/historyDisabled'
|
||||
|
||||
/**
|
||||
* Fixed batch size for both the per-conversation API fetch loop and each
|
||||
* generated download archive. Not user-configurable—kept at 100 to stay
|
||||
* well within ChatGPT web-session rate limits.
|
||||
*/
|
||||
export const EXPORT_OPERATION_BATCH = 100
|
||||
|
||||
@@ -11,6 +11,7 @@ import { standardizeLineBreaks } from '../utils/text'
|
||||
import { dateStr, getColorScheme, timestamp, unixTimestampToISOString } from '../utils/utils'
|
||||
import type { ApiConversationWithId, ConversationNodeMessage, ConversationResult } from '../api'
|
||||
import type { ExportMeta } from '../ui/SettingContext'
|
||||
import type { PartInfo } from '../utils/download'
|
||||
|
||||
export async function exportToHtml(fileNameFormat: string, metaList: ExportMeta[]) {
|
||||
if (!checkIfConversationStarted()) {
|
||||
@@ -31,7 +32,7 @@ export async function exportToHtml(fileNameFormat: string, metaList: ExportMeta[
|
||||
return true
|
||||
}
|
||||
|
||||
export async function exportAllToHtml(fileNameFormat: string, apiConversations: ApiConversationWithId[], metaList?: ExportMeta[], projectName?: string) {
|
||||
export async function exportAllToHtml(fileNameFormat: string, apiConversations: ApiConversationWithId[], metaList?: ExportMeta[], projectName?: string, partIndex?: number, totalParts?: number) {
|
||||
const userAvatar = await getUserAvatar()
|
||||
|
||||
const zip = new JSZip()
|
||||
@@ -63,7 +64,10 @@ export async function exportAllToHtml(fileNameFormat: string, apiConversations:
|
||||
level: 9,
|
||||
},
|
||||
})
|
||||
downloadFile(buildZipFileName('html', projectName), 'application/zip', blob)
|
||||
const partInfo: PartInfo | undefined = (partIndex != null && totalParts != null)
|
||||
? { part: partIndex, total: totalParts }
|
||||
: undefined
|
||||
downloadFile(buildZipFileName('html', projectName, partInfo), 'application/zip', blob)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -3,9 +3,10 @@ import { fetchConversation, getCurrentChatId, processConversation } from '../api
|
||||
import i18n from '../i18n'
|
||||
import { checkIfConversationStarted } from '../page'
|
||||
import { convertToOoba, convertToTavern } from '../utils/conversion'
|
||||
import { buildZipFileName, downloadFile, getFileNameWithFormat, normalizeProjectName } from '../utils/download'
|
||||
import { buildJsonBatchFileName, buildZipFileName, downloadFile, getFileNameWithFormat } from '../utils/download'
|
||||
import type { ApiConversationWithId } from '../api'
|
||||
import type { ExportMeta } from '../ui/SettingContext'
|
||||
import type { PartInfo } from '../utils/download'
|
||||
|
||||
export async function exportToJson(fileNameFormat: string) {
|
||||
if (!checkIfConversationStarted()) {
|
||||
@@ -61,17 +62,17 @@ export async function exportToOoba(fileNameFormat: string) {
|
||||
return true
|
||||
}
|
||||
|
||||
export async function exportAllToOfficialJson(_fileNameFormat: string, apiConversations: ApiConversationWithId[], _metaList?: ExportMeta[], projectName?: string) {
|
||||
export async function exportAllToOfficialJson(_fileNameFormat: string, apiConversations: ApiConversationWithId[], _metaList?: ExportMeta[], projectName?: string, partIndex?: number, totalParts?: number) {
|
||||
const partInfo: PartInfo | undefined = (partIndex != null && totalParts != null)
|
||||
? { part: partIndex, total: totalParts }
|
||||
: undefined
|
||||
const content = conversationToJson(apiConversations)
|
||||
const baseName = projectName
|
||||
? `chatgpt-export-project-${normalizeProjectName(projectName)}`
|
||||
: 'chatgpt-export'
|
||||
downloadFile(`${baseName}.json`, 'application/json', content)
|
||||
downloadFile(buildJsonBatchFileName(projectName, partInfo), 'application/json', content)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
export async function exportAllToJson(fileNameFormat: string, apiConversations: ApiConversationWithId[], _metaList?: ExportMeta[], projectName?: string) {
|
||||
export async function exportAllToJson(fileNameFormat: string, apiConversations: ApiConversationWithId[], _metaList?: ExportMeta[], projectName?: string, partIndex?: number, totalParts?: number) {
|
||||
const zip = new JSZip()
|
||||
const filenameMap = new Map<string, number>()
|
||||
const conversations = apiConversations.map(x => ({
|
||||
@@ -104,7 +105,10 @@ export async function exportAllToJson(fileNameFormat: string, apiConversations:
|
||||
level: 9,
|
||||
},
|
||||
})
|
||||
downloadFile(buildZipFileName('json', projectName), 'application/zip', blob)
|
||||
const partInfo: PartInfo | undefined = (partIndex != null && totalParts != null)
|
||||
? { part: partIndex, total: totalParts }
|
||||
: undefined
|
||||
downloadFile(buildZipFileName('json', projectName, partInfo), 'application/zip', blob)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -10,6 +10,7 @@ import { standardizeLineBreaks } from '../utils/text'
|
||||
import { dateStr, timestamp, unixTimestampToISOString } from '../utils/utils'
|
||||
import type { ApiConversationWithId, Citation, ConversationNodeMessage, ConversationResult } from '../api'
|
||||
import type { ExportMeta } from '../ui/SettingContext'
|
||||
import type { PartInfo } from '../utils/download'
|
||||
|
||||
export async function exportToMarkdown(fileNameFormat: string, metaList: ExportMeta[]) {
|
||||
if (!checkIfConversationStarted()) {
|
||||
@@ -28,7 +29,7 @@ export async function exportToMarkdown(fileNameFormat: string, metaList: ExportM
|
||||
return true
|
||||
}
|
||||
|
||||
export async function exportAllToMarkdown(fileNameFormat: string, apiConversations: ApiConversationWithId[], metaList?: ExportMeta[], projectName?: string) {
|
||||
export async function exportAllToMarkdown(fileNameFormat: string, apiConversations: ApiConversationWithId[], metaList?: ExportMeta[], projectName?: string, partIndex?: number, totalParts?: number) {
|
||||
const zip = new JSZip()
|
||||
const filenameMap = new Map<string, number>()
|
||||
const conversations = apiConversations.map(x => processConversation(x))
|
||||
@@ -58,7 +59,10 @@ export async function exportAllToMarkdown(fileNameFormat: string, apiConversatio
|
||||
level: 9,
|
||||
},
|
||||
})
|
||||
downloadFile(buildZipFileName('markdown', projectName), 'application/zip', blob)
|
||||
const partInfo: PartInfo | undefined = (partIndex != null && totalParts != null)
|
||||
? { part: partIndex, total: totalParts }
|
||||
: undefined
|
||||
downloadFile(buildZipFileName('markdown', projectName, partInfo), 'application/zip', blob)
|
||||
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -42,9 +42,60 @@
|
||||
"Select Project": "Select Project",
|
||||
"(no project)": "(no project)",
|
||||
"Export All Limit": "Export All Limit",
|
||||
"Export All Limit Description": "Set the maximum number of conversations to load in the 'Export All' dialog.",
|
||||
"Export All Limit Description": "Set the maximum number of conversations to load. Exports run in waves of 100 conversations to stay within API rate limits.",
|
||||
"Select a source to load conversations": "Select a project above to load conversations.",
|
||||
"Search": "Search",
|
||||
"Last 100": "Last 100",
|
||||
"No results": "No results"
|
||||
"No results": "No results",
|
||||
"Date From": "From",
|
||||
"Date To": "To",
|
||||
"Date Filter Label": "Date",
|
||||
"Date Filter Hint": "Filters by the chosen timestamp field. Leave blank for no date restriction.",
|
||||
"Date Filter Field Created": "Created",
|
||||
"Date Filter Field Updated": "Updated",
|
||||
"Date Preset 7d": "7d",
|
||||
"Date Preset 30d": "30d",
|
||||
"Date Preset 90d": "90d",
|
||||
"Date Preset Year": "This year",
|
||||
"Clear filter": "Clear",
|
||||
"Selected count": "{{count}} selected",
|
||||
"Export batch info": "Exports in batches of 100 per download",
|
||||
"Exporting batch": "Exporting batch {{current}} of {{total}}",
|
||||
"Export batches button": "Export ({{n}} downloads)",
|
||||
"Batch progress": "Batch {{current}}/{{total}}",
|
||||
"All conversations": "All conversations",
|
||||
"Filters hint": "type # to add filters",
|
||||
"Load more conversations": "Load {{n}} more",
|
||||
"Load more conversations remaining": "Load {{n}} more · {{remaining}} left",
|
||||
"Chip group chat class": "Chat class",
|
||||
"Chip group origin": "Origin",
|
||||
"Chip group status": "Status",
|
||||
"Chip group duration": "Duration",
|
||||
"Chip group recency": "Recency",
|
||||
"Chip cc regular label": "💬 Regular",
|
||||
"Chip cc regular desc": "Standard chats with no GPT or Project",
|
||||
"Chip cc gpt label": "🤖 GPT",
|
||||
"Chip cc gpt desc": "Used a GPT Store AI",
|
||||
"Chip cc project label": "📂 Project",
|
||||
"Chip cc project desc": "Started inside one of your Projects",
|
||||
"Chip origin web label": "🌐 Web / App",
|
||||
"Chip origin web desc": "Web app or mobile app",
|
||||
"Chip status starred label": "⭐ Starred",
|
||||
"Chip status starred desc": "Conversations you have starred",
|
||||
"Chip status temporary label": "💬 Temporary",
|
||||
"Chip status temporary desc": "Temporary chats not saved to history",
|
||||
"Chip status pinned label": "📌 Pinned",
|
||||
"Chip status pinned desc": "Conversations you have pinned",
|
||||
"Chip duration label": "⏱ Long conversation",
|
||||
"Chip duration desc": "Active span ≥ N days (adjustable on chip)",
|
||||
"Chip recency label": "📅 Recently active",
|
||||
"Chip recency desc": "Updated within N days (adjustable on chip)",
|
||||
"Chip mode include": "include",
|
||||
"Chip mode exclude": "exclude",
|
||||
"Chip logic and": "AND",
|
||||
"Chip logic or": "OR",
|
||||
"Chip duration prefix": "⏱ active ≥",
|
||||
"Chip duration suffix": "d",
|
||||
"Chip recency prefix": "📅 updated <",
|
||||
"Chip recency suffix": "d"
|
||||
}
|
||||
|
||||
@@ -206,8 +206,21 @@
|
||||
}
|
||||
|
||||
.SelectItem {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.SelectItem .CheckBoxLabel {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.SelectItem .LabelText {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.SelectItem label, .SelectItem input {
|
||||
@@ -218,6 +231,365 @@
|
||||
vertical-align: middle;
|
||||
}
|
||||
|
||||
.SelectItemMeta {
|
||||
flex-shrink: 0;
|
||||
font-size: 0.7rem;
|
||||
color: #9ca3af;
|
||||
white-space: nowrap;
|
||||
font-variant-numeric: tabular-nums;
|
||||
min-width: 6.5rem;
|
||||
text-align: right;
|
||||
}
|
||||
.SelectItemMetaActive {
|
||||
color: #6b7280;
|
||||
font-weight: 600;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectItemMetaActive { color: #d1d5db; }
|
||||
}
|
||||
|
||||
/* ── Sortable column header row ── */
|
||||
.SelectListHeader {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 0 16px;
|
||||
border: 1px solid #6f6e77;
|
||||
border-bottom: none;
|
||||
background: #f9fafb;
|
||||
user-select: none;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectListHeader { background: #1f2937; }
|
||||
}
|
||||
.SelectListHeaderCell {
|
||||
flex-shrink: 0;
|
||||
font-size: 0.68rem;
|
||||
font-weight: 600;
|
||||
color: #9ca3af;
|
||||
letter-spacing: 0.03em;
|
||||
text-transform: uppercase;
|
||||
background: transparent;
|
||||
border: none;
|
||||
padding: 5px 4px;
|
||||
cursor: pointer;
|
||||
white-space: nowrap;
|
||||
min-width: 6.5rem;
|
||||
text-align: right;
|
||||
}
|
||||
.SelectListHeaderCell:hover { color: #374151; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectListHeaderCell:hover { color: #e5e7eb; }
|
||||
}
|
||||
.SelectListHeaderCellTitle {
|
||||
flex: 1;
|
||||
text-align: left;
|
||||
padding-left: 28px; /* align with checkbox label */
|
||||
}
|
||||
.SelectListHeaderCellActive {
|
||||
color: #2563eb;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectListHeaderCellActive { color: #60a5fa; }
|
||||
}
|
||||
|
||||
.SelectChips {
|
||||
display: flex;
|
||||
flex-wrap: wrap;
|
||||
gap: 5px;
|
||||
padding: 6px 12px;
|
||||
border: 1px solid #6f6e77;
|
||||
border-bottom: none;
|
||||
}
|
||||
|
||||
.SelectChip {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
gap: 4px;
|
||||
padding: 2px 8px;
|
||||
border-radius: 9999px;
|
||||
font-size: 0.72rem;
|
||||
background: #dbeafe;
|
||||
color: #1d4ed8;
|
||||
border: 1px solid #bfdbfe;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectChip {
|
||||
background: #1e3a5f;
|
||||
color: #93c5fd;
|
||||
border-color: #1e40af;
|
||||
}
|
||||
}
|
||||
|
||||
.SelectChipRemove {
|
||||
cursor: pointer;
|
||||
opacity: 0.6;
|
||||
line-height: 1;
|
||||
padding: 0 2px;
|
||||
}
|
||||
.SelectChipRemove:hover {
|
||||
opacity: 1;
|
||||
color: #dc2626;
|
||||
}
|
||||
|
||||
.SelectFilterPopover {
|
||||
position: absolute;
|
||||
top: 100%;
|
||||
left: 0;
|
||||
right: 0;
|
||||
z-index: 50;
|
||||
background: white;
|
||||
border: 1px solid #6f6e77;
|
||||
border-radius: 0 0 6px 6px;
|
||||
box-shadow: 0 4px 12px rgba(0,0,0,0.15);
|
||||
overflow-y: auto;
|
||||
max-height: 280px;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectFilterPopover {
|
||||
background: #1f2937;
|
||||
border-color: #374151;
|
||||
}
|
||||
}
|
||||
|
||||
.SelectFilterOption {
|
||||
width: 100%;
|
||||
text-align: left;
|
||||
padding: 8px 14px;
|
||||
background: transparent;
|
||||
border: none;
|
||||
cursor: pointer;
|
||||
color: inherit;
|
||||
}
|
||||
|
||||
.SelectFilterOption:hover {
|
||||
background: #f3f4f6;
|
||||
}
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectFilterOption:hover {
|
||||
background: #374151;
|
||||
}
|
||||
}
|
||||
|
||||
.SelectFilterOption strong {
|
||||
display: block;
|
||||
font-size: 0.82rem;
|
||||
}
|
||||
|
||||
.SelectFilterOption small {
|
||||
display: block;
|
||||
font-size: 0.7rem;
|
||||
color: #9ca3af;
|
||||
margin-top: 1px;
|
||||
}
|
||||
|
||||
/* Group header inside the filter picker popover */
|
||||
.SelectFilterGroupHeader {
|
||||
padding: 6px 14px 2px;
|
||||
font-size: 0.67rem;
|
||||
font-weight: 700;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.07em;
|
||||
color: #9ca3af;
|
||||
pointer-events: none;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
/* AND / OR logic toggle pill in the chip bar */
|
||||
.SelectChipLogic {
|
||||
display: inline-flex;
|
||||
align-items: center;
|
||||
padding: 2px 7px;
|
||||
border-radius: 9999px;
|
||||
font-size: 0.67rem;
|
||||
font-weight: 700;
|
||||
letter-spacing: 0.05em;
|
||||
background: #e0f2fe;
|
||||
color: #0369a1;
|
||||
border: 1px solid #bae6fd;
|
||||
cursor: pointer;
|
||||
white-space: nowrap;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.SelectChipLogic:hover { background: #bae6fd; }
|
||||
.SelectChipLogicOr { background: #fef3c7; color: #b45309; border-color: #fde68a; }
|
||||
.SelectChipLogicOr:hover { background: #fde68a; }
|
||||
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectChipLogic { background: #0c4a6e; color: #7dd3fc; border-color: #075985; }
|
||||
.SelectChipLogicOr { background: #451a03; color: #fcd34d; border-color: #78350f; }
|
||||
}
|
||||
|
||||
/* Include / exclude mode badge inside each chip */
|
||||
.SelectChipMode {
|
||||
font-size: 0.6rem;
|
||||
padding: 1px 4px;
|
||||
border-radius: 3px;
|
||||
background: rgba(29, 78, 216, 0.1);
|
||||
color: inherit;
|
||||
cursor: pointer;
|
||||
opacity: 0.75;
|
||||
white-space: nowrap;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
.SelectChipMode:hover { opacity: 1; }
|
||||
.SelectChipModeExclude { background: rgba(220, 38, 38, 0.15); }
|
||||
|
||||
/* Chip rendered in exclude mode */
|
||||
.SelectChipExclude {
|
||||
background: #fee2e2;
|
||||
color: #b91c1c;
|
||||
border-color: #fecaca;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectChipExclude { background: #450a0a; color: #f87171; border-color: #7f1d1d; }
|
||||
}
|
||||
|
||||
/* Number input inside duration / recency chips */
|
||||
.SelectChip input[type="number"] {
|
||||
width: 2.2rem;
|
||||
background: transparent;
|
||||
color: inherit;
|
||||
font-size: inherit;
|
||||
text-align: center;
|
||||
border: none;
|
||||
outline: none;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
/* ── Picker: back button (stage 2 header) ── */
|
||||
.SelectFilterBack {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
width: 100%;
|
||||
padding: 7px 14px 6px;
|
||||
font-size: 0.78rem;
|
||||
font-weight: 600;
|
||||
color: #6b7280;
|
||||
text-align: left;
|
||||
background: transparent;
|
||||
border: none;
|
||||
border-bottom: 1px solid #e5e7eb;
|
||||
cursor: pointer;
|
||||
}
|
||||
.SelectFilterBack:hover { color: #374151; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectFilterBack { color: #9ca3af; border-color: #374151; }
|
||||
.SelectFilterBack:hover { color: #e5e7eb; }
|
||||
}
|
||||
|
||||
/* ── Picker: checkbox-style option in sub-view ── */
|
||||
.SelectFilterCheckboxBtn {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
width: 100%;
|
||||
padding: 7px 14px;
|
||||
background: transparent;
|
||||
border: none;
|
||||
cursor: pointer;
|
||||
color: inherit;
|
||||
text-align: left;
|
||||
}
|
||||
.SelectFilterCheckboxBtn:hover { background: #f3f4f6; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectFilterCheckboxBtn:hover { background: #374151; }
|
||||
}
|
||||
.SelectFilterCheckIcon {
|
||||
flex-shrink: 0;
|
||||
width: 1rem;
|
||||
text-align: center;
|
||||
font-size: 0.85rem;
|
||||
color: #9ca3af;
|
||||
}
|
||||
.SelectFilterCheckboxBtnChecked .SelectFilterCheckIcon { color: #2563eb; }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectFilterCheckboxBtnChecked .SelectFilterCheckIcon { color: #60a5fa; }
|
||||
}
|
||||
|
||||
/* ── Picker: apply/update button at bottom of sub-view ── */
|
||||
.SelectFilterApplyBtn {
|
||||
display: block;
|
||||
width: calc(100% - 28px);
|
||||
margin: 6px 14px;
|
||||
padding: 5px 0;
|
||||
background: #2563eb;
|
||||
color: white;
|
||||
border: none;
|
||||
border-radius: 4px;
|
||||
font-size: 0.8rem;
|
||||
font-weight: 600;
|
||||
cursor: pointer;
|
||||
text-align: center;
|
||||
}
|
||||
.SelectFilterApplyBtn:disabled { opacity: 0.4; cursor: not-allowed; }
|
||||
.SelectFilterApplyBtn:not(:disabled):hover { background: #1d4ed8; }
|
||||
|
||||
/* ── Chip: clickable value label that opens edit picker ── */
|
||||
.SelectChipValueBtn {
|
||||
background: transparent;
|
||||
border: none;
|
||||
color: inherit;
|
||||
font-size: inherit;
|
||||
cursor: pointer;
|
||||
padding: 0;
|
||||
text-decoration: underline;
|
||||
text-decoration-style: dotted;
|
||||
text-underline-offset: 2px;
|
||||
}
|
||||
.SelectChipValueBtn:hover { opacity: 0.75; }
|
||||
|
||||
/* ── Date chip: full-width row in chip bar ── */
|
||||
.SelectChipDateRow {
|
||||
flex-basis: 100%;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
flex-wrap: wrap;
|
||||
gap: 5px;
|
||||
padding: 5px 8px;
|
||||
background: #eff6ff;
|
||||
border-radius: 6px;
|
||||
border: 1px solid #bfdbfe;
|
||||
color: #1d4ed8;
|
||||
font-size: 0.75rem;
|
||||
}
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectChipDateRow { background: #1e3a5f; color: #93c5fd; border-color: #1e40af; }
|
||||
}
|
||||
.SelectChipDateRow select,
|
||||
.SelectChipDateRow input[type="date"] {
|
||||
background: transparent;
|
||||
border: 1px solid currentColor;
|
||||
border-radius: 3px;
|
||||
color: inherit;
|
||||
font-size: 0.72rem;
|
||||
padding: 1px 4px;
|
||||
outline: none;
|
||||
}
|
||||
.SelectChipDateRow input[type="date"] { min-width: 6rem; }
|
||||
.SelectChipDatePresets {
|
||||
display: flex;
|
||||
gap: 3px;
|
||||
margin-left: auto;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
.SelectChipDatePresets button {
|
||||
padding: 1px 5px;
|
||||
border-radius: 3px;
|
||||
border: 1px solid currentColor;
|
||||
background: transparent;
|
||||
color: inherit;
|
||||
font-size: 0.67rem;
|
||||
cursor: pointer;
|
||||
}
|
||||
.SelectChipDatePresets button:hover { background: rgba(37, 99, 235, 0.12); }
|
||||
@media (prefers-color-scheme: dark) {
|
||||
.SelectChipDatePresets button:hover { background: rgba(147, 197, 253, 0.12); }
|
||||
}
|
||||
|
||||
@keyframes contentShow {
|
||||
from {
|
||||
opacity: 0;
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -28,11 +28,31 @@ export function normalizeProjectName(projectName: string) {
|
||||
.replace(/^-+|-+$/g, '')
|
||||
}
|
||||
|
||||
export function buildZipFileName(format: string, projectName?: string) {
|
||||
export interface PartInfo {
|
||||
part: number
|
||||
total: number
|
||||
}
|
||||
|
||||
function partSuffix(partInfo?: PartInfo): string {
|
||||
if (!partInfo || partInfo.total <= 1) return ''
|
||||
const pad = (n: number) => String(n).padStart(2, '0')
|
||||
return `-part-${pad(partInfo.part)}-of-${pad(partInfo.total)}`
|
||||
}
|
||||
|
||||
export function buildZipFileName(format: string, projectName?: string, partInfo?: PartInfo) {
|
||||
const suffix = partSuffix(partInfo)
|
||||
if (projectName) {
|
||||
return `chatgpt-export-${format}-project-${normalizeProjectName(projectName)}.zip`
|
||||
return `chatgpt-export-${format}-project-${normalizeProjectName(projectName)}${suffix}.zip`
|
||||
}
|
||||
return `chatgpt-export-${format}.zip`
|
||||
return `chatgpt-export-${format}${suffix}.zip`
|
||||
}
|
||||
|
||||
export function buildJsonBatchFileName(projectName?: string, partInfo?: PartInfo) {
|
||||
const suffix = partSuffix(partInfo)
|
||||
if (projectName) {
|
||||
return `chatgpt-export-project-${normalizeProjectName(projectName)}${suffix}.json`
|
||||
}
|
||||
return `chatgpt-export${suffix}.json`
|
||||
}
|
||||
|
||||
export function getFileNameWithFormat(format: string, ext: string, {
|
||||
|
||||
@@ -1,25 +1,52 @@
|
||||
import EventEmitter from 'mitt'
|
||||
import { RateLimitError } from '../api'
|
||||
import { sleep } from './utils'
|
||||
|
||||
type RequestFn<T> = () => Promise<T>
|
||||
|
||||
/** Public shape callers pass to `add()` */
|
||||
interface RequestObject<T> {
|
||||
name: string
|
||||
request: RequestFn<T>
|
||||
}
|
||||
|
||||
/** Internal shape with per-item retry counters */
|
||||
interface InternalRequestObject<T> extends RequestObject<T> {
|
||||
retries: number // general error retries
|
||||
rateRetries: number // 429-specific retries
|
||||
}
|
||||
|
||||
export type RequestStatus = 'processing' | 'retrying' | 'rate_limited'
|
||||
|
||||
interface ProgressEvent {
|
||||
total: number
|
||||
completed: number
|
||||
currentName: string
|
||||
currentStatus: 'processing' | 'retrying'
|
||||
currentStatus: RequestStatus
|
||||
/** Seconds remaining in a rate-limit pause (only set when status === 'rate_limited') */
|
||||
rateLimitWaitSecs?: number
|
||||
}
|
||||
|
||||
/** Max retries for generic (non-429) errors before skipping a single request */
|
||||
const MAX_RETRIES = 5
|
||||
/**
|
||||
* Max times the entire queue can be globally paused for rate limiting before
|
||||
* giving up and stopping the queue entirely.
|
||||
*/
|
||||
const MAX_GLOBAL_PAUSES = 5
|
||||
/**
|
||||
* Default pause length (ms) applied to the whole queue on a 429.
|
||||
* Used when the API does not return a Retry-After header.
|
||||
*/
|
||||
const DEFAULT_429_PAUSE_MS = 60_000
|
||||
|
||||
export class RequestQueue<T> {
|
||||
private eventEmitter = EventEmitter<{
|
||||
done: T[]
|
||||
progress: ProgressEvent
|
||||
} & Record<string, any[]>>()
|
||||
|
||||
private queue: Array<RequestObject<T>> = []
|
||||
private queue: Array<InternalRequestObject<T>> = []
|
||||
private results: T[] = []
|
||||
|
||||
private status: 'IDLE' | 'IN_PROGRESS' | 'STOPPED' | 'COMPLETED' = 'IDLE'
|
||||
@@ -30,12 +57,21 @@ export class RequestQueue<T> {
|
||||
private total = 0
|
||||
private completed = 0
|
||||
|
||||
/**
|
||||
* Timestamp (ms since epoch) until which the whole queue is frozen after
|
||||
* receiving a 429. While Date.now() < pauseUntil every process() iteration
|
||||
* waits out the remainder before making the next request.
|
||||
*/
|
||||
private pauseUntil = 0
|
||||
/** How many global rate-limit pauses have been applied so far */
|
||||
private globalPauses = 0
|
||||
|
||||
constructor(private minBackoff: number, private maxBackoff: number) {
|
||||
this.backoff = minBackoff
|
||||
}
|
||||
|
||||
add(requestObject: RequestObject<T>) {
|
||||
this.queue.push(requestObject)
|
||||
this.queue.push({ ...requestObject, retries: 0, rateRetries: 0 })
|
||||
}
|
||||
|
||||
start() {
|
||||
@@ -55,6 +91,8 @@ export class RequestQueue<T> {
|
||||
this.results = []
|
||||
this.status = 'IDLE'
|
||||
this.backoff = this.minBackoff
|
||||
this.pauseUntil = 0
|
||||
this.globalPauses = 0
|
||||
this.total = 0
|
||||
this.completed = 0
|
||||
}
|
||||
@@ -76,35 +114,83 @@ export class RequestQueue<T> {
|
||||
return
|
||||
}
|
||||
|
||||
// ── Global rate-limit pause ──────────────────────────────────────────
|
||||
// If a previous request set pauseUntil, wait for the remainder before
|
||||
// making any new request. This freezes the whole queue at once instead
|
||||
// of per-item retries, so 100 queued items don't each wait 30s in turn.
|
||||
const remaining = this.pauseUntil - Date.now()
|
||||
if (remaining > 0) {
|
||||
const waitSecs = Math.ceil(remaining / 1000)
|
||||
// Broadcast the pause status for every item currently at the front
|
||||
this.progress(this.queue[0].name, 'rate_limited', waitSecs)
|
||||
await sleep(remaining)
|
||||
this.pauseUntil = 0
|
||||
}
|
||||
|
||||
this.status = 'IN_PROGRESS'
|
||||
const requestObject = this.queue.shift()!
|
||||
const { name, request } = requestObject
|
||||
|
||||
let waitMs = this.backoff
|
||||
|
||||
try {
|
||||
this.progress(name, 'processing')
|
||||
const result = await request()
|
||||
this.results.push(result)
|
||||
this.completed++
|
||||
this.progress(name, 'processing')
|
||||
this.backoff = this.minBackoff // reset backoff on success
|
||||
this.backoff = this.minBackoff // reset on success
|
||||
requestObject.retries = 0
|
||||
}
|
||||
catch (error) {
|
||||
console.error(`Request ${name} failed:`, error)
|
||||
this.progress(name, 'retrying')
|
||||
if (error instanceof RateLimitError) {
|
||||
this.globalPauses++
|
||||
if (this.globalPauses > MAX_GLOBAL_PAUSES) {
|
||||
// Rate limit persists even after several long pauses — abort.
|
||||
console.warn('[Exporter] Queue stopped: API rate limit did not clear after', MAX_GLOBAL_PAUSES, 'pauses')
|
||||
this.stop()
|
||||
return
|
||||
}
|
||||
// Freeze the whole queue. Exponentially increase the pause so
|
||||
// we back off harder if the first pause wasn't long enough.
|
||||
const pauseMs = Math.max(
|
||||
error.retryAfterMs,
|
||||
DEFAULT_429_PAUSE_MS * this.globalPauses,
|
||||
)
|
||||
this.pauseUntil = Date.now() + pauseMs
|
||||
this.progress(name, 'rate_limited', Math.round(pauseMs / 1000))
|
||||
console.warn(`[Exporter] Rate limited (429). Pausing queue for ${Math.round(pauseMs / 1000)}s (pause #${this.globalPauses})`)
|
||||
// Put this item back — it will be retried after the pause clears
|
||||
this.queue.unshift(requestObject)
|
||||
waitMs = 0 // the sleep is handled at the top of the next process() call
|
||||
}
|
||||
else {
|
||||
console.error(`[Exporter] "${name}" failed:`, error)
|
||||
requestObject.retries++
|
||||
if (requestObject.retries > MAX_RETRIES) {
|
||||
console.warn(`[Exporter] "${name}" skipped after ${MAX_RETRIES} retries`)
|
||||
waitMs = 0 // skip — don't re-queue
|
||||
}
|
||||
else {
|
||||
this.backoff = Math.min(this.backoff * this.backoffMultiplier, this.maxBackoff)
|
||||
this.queue.unshift(requestObject) // add request back to the front of the queue
|
||||
waitMs = this.backoff
|
||||
this.progress(name, 'retrying')
|
||||
this.queue.unshift(requestObject)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await sleep(this.backoff)
|
||||
await sleep(waitMs)
|
||||
this.process()
|
||||
}
|
||||
|
||||
private progress(name: string, status: 'processing' | 'retrying') {
|
||||
private progress(name: string, status: RequestStatus, rateLimitWaitSecs?: number) {
|
||||
this.eventEmitter.emit('progress', {
|
||||
total: this.total,
|
||||
completed: this.completed,
|
||||
currentName: name,
|
||||
currentStatus: status,
|
||||
rateLimitWaitSecs,
|
||||
})
|
||||
}
|
||||
|
||||
|
||||
@@ -13,8 +13,64 @@ export function onloadSafe(fn: () => void) {
|
||||
}
|
||||
}
|
||||
|
||||
export function sleep(ms: number) {
|
||||
return new Promise(resolve => setTimeout(resolve, ms))
|
||||
/**
|
||||
* Background-tab-safe sleep.
|
||||
*
|
||||
* Chrome throttles setTimeout in non-focused tabs to ≥1 s, which stalls
|
||||
* the export queue when the user switches away. Running the timer inside a
|
||||
* Web Worker bypasses that throttle. We create one persistent worker and
|
||||
* reuse it for all sleeps. If Worker creation fails (CSP, etc.) we fall
|
||||
* back to a regular setTimeout so nothing breaks.
|
||||
*/
|
||||
|
||||
// Inline worker source — keeps us dependency-free and avoids extra bundle chunks
|
||||
const _WORKER_SRC = 'onmessage=function(e){setTimeout(function(){postMessage(e.data)},e.data.ms)}'
|
||||
|
||||
let _sleepWorker: Worker | null = null
|
||||
let _sleepWorkerFailed = false
|
||||
const _pendingResolves = new Map<number, () => void>()
|
||||
let _sleepIdCounter = 0
|
||||
|
||||
function _getSleepWorker(): Worker | null {
|
||||
if (_sleepWorkerFailed) return null
|
||||
if (_sleepWorker) return _sleepWorker
|
||||
try {
|
||||
const blob = new Blob([_WORKER_SRC], { type: 'application/javascript' })
|
||||
const url = URL.createObjectURL(blob)
|
||||
const w = new Worker(url)
|
||||
URL.revokeObjectURL(url)
|
||||
w.onmessage = (e: MessageEvent<{ id: number }>) => {
|
||||
const resolve = _pendingResolves.get(e.data.id)
|
||||
if (resolve) {
|
||||
_pendingResolves.delete(e.data.id)
|
||||
resolve()
|
||||
}
|
||||
}
|
||||
w.onerror = () => {
|
||||
// Worker crashed — fall back to setTimeout for remaining sleeps
|
||||
_sleepWorkerFailed = true
|
||||
_sleepWorker = null
|
||||
for (const resolve of _pendingResolves.values()) resolve()
|
||||
_pendingResolves.clear()
|
||||
}
|
||||
_sleepWorker = w
|
||||
return w
|
||||
}
|
||||
catch {
|
||||
_sleepWorkerFailed = true
|
||||
return null
|
||||
}
|
||||
}
|
||||
|
||||
export function sleep(ms: number): Promise<void> {
|
||||
if (ms <= 0) return Promise.resolve()
|
||||
const worker = _getSleepWorker()
|
||||
if (!worker) return new Promise(resolve => setTimeout(resolve, ms))
|
||||
return new Promise((resolve) => {
|
||||
const id = _sleepIdCounter++
|
||||
_pendingResolves.set(id, resolve)
|
||||
worker.postMessage({ id, ms })
|
||||
})
|
||||
}
|
||||
|
||||
export function dateStr(date: Date = new Date()) {
|
||||
|
||||
Reference in New Issue
Block a user