Created
December 3, 2025 23:41
-
-
Save matthew-gerstman/08c657c49a72a6bb61c1f31757e7f93e to your computer and use it in GitHub Desktop.
Artifact Preview Generation Flakiness Analysis
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Artifact Preview Generation Flakiness Analysis | |
| ## Root Cause Analysis | |
| The preview generation system has multiple failure points causing previews to not display: | |
| ### 1. Race Condition (Primary Issue) | |
| The flow is: | |
| 1. Create artifact → Send `artifact:created` SSE (NO preview URL) | |
| 2. Generate preview async → Upload to S3 → Update artifact metadata | |
| 3. Send `artifact:updated` SSE (WITH preview URL) | |
| **Problem**: Frontend caches artifact from step 1. The step 3 SSE event either: | |
| - Doesn't include `previewUrl`/`previewExpiresAt` in the hydrated payload | |
| - Gets filtered by Mirror due to `origin: 'system:preview'` | |
| ### 2. Silent Early Returns | |
| In `apps/api/src/services/preview.service.ts`: | |
| - **Workbooks**: Returns early if `recordCount === 0` (lines 112-114) - common for new sheets | |
| - **Documents**: Returns early if content empty or >100KB (lines 177-186) | |
| - **No logging** on any early returns | |
| ### 3. External CDN Dependency | |
| In `functions/preview-generation/src/utils/emoji.ts`: | |
| - Fetches emojis from `twemoji.maxcdn.com` with no timeout | |
| - Network failures silently swallowed | |
| ### 4. No Retry Mechanism | |
| Lambda failures are logged but not retried. Preview generation is fire-and-forget. | |
| ### 5. Signed URL Expiration | |
| URLs expire in 1 hour. Cached artifacts in Mirror/IndexedDB may have expired `previewExpiresAt`. | |
| --- | |
| ## Key Files | |
| - `apps/api/src/services/preview.service.ts` - Main preview generation orchestration | |
| - `apps/api/src/services/artifact-preview.service.ts` - Preview URL hydration and SSE events | |
| - `apps/api/src/utils/lambda-client.ts` - Lambda invocation (invokePreviewGeneration) | |
| - `functions/preview-generation/src/index.ts` - Lambda handler | |
| - `functions/preview-generation/src/utils/emoji.ts` - External CDN dependency | |
| - `dashboard/src/hooks/use-artifact-preview.ts` - Frontend preview hook | |
| --- | |
| ## Proposed Fixes | |
| ### Phase 1: Add Observability (Quick Win) | |
| Add logging to understand failure patterns before making changes. | |
| - Add `logger.info()` for every early return in preview generation | |
| - Add timing metrics for Lambda invocation | |
| - Log when frontend receives artifact without preview data | |
| ### Phase 2: Fix Race Condition | |
| Ensure preview URL is included in SSE events after generation. | |
| **Option A (Simpler)**: In `artifact-preview.service.ts`, ensure `buildUpdatedEventPayload` always hydrates fresh preview URLs for `system:preview` origin updates. | |
| **Option B (More Robust)**: Move preview generation into the artifact creation flow (synchronous) for presentation/document types, so `artifact:created` event already has preview. | |
| ### Phase 3: Add Retry Mechanism | |
| Wrap preview Lambda invocation with retry logic (exponential backoff, max 3 attempts). | |
| ### Phase 4: Fix CDN Timeout | |
| Add 5-second timeout to emoji fetch in preview-generation Lambda. | |
| --- | |
| ## Code Flow | |
| ``` | |
| Artifact Creation | |
| │ | |
| ▼ | |
| ┌─────────────────────────────┐ | |
| │ artifact:created SSE event │ ← No preview URL yet | |
| │ (frontend caches artifact) │ | |
| └─────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────────────────┐ | |
| │ generatePreviewForArtifact │ | |
| │ (async, fire-and-forget) │ | |
| └─────────────────────────────┘ | |
| │ | |
| ▼ (may fail silently) | |
| ┌─────────────────────────────┐ | |
| │ Lambda: preview-generation │ | |
| │ - Generate SVG via satori │ | |
| │ - Upload to S3 │ | |
| └─────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────────────────┐ | |
| │ updateArtifact with │ | |
| │ metadata.preview.key/bucket │ | |
| │ origin: 'system:preview' │ | |
| └─────────────────────────────┘ | |
| │ | |
| ▼ | |
| ┌─────────────────────────────┐ | |
| │ artifact:updated SSE event │ ← Should have previewUrl | |
| │ (frontend should update) │ | |
| └─────────────────────────────┘ | |
| ``` |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment