Telemetry Implementation Plan: Wide Events with Convex Tracing

This document outlines the implementation plan for structured logging using "wide events" (canonical log lines), integrated directly into our existing Convex custom function builders.

Table of Contents

• Overview
• Architecture
• Core Concepts
• Implementation
• Usage Examples
• Example Output
• Future Enhancements
• References

---

Overview

What are Wide Events?

Instead of scattered log lines like:

INFO: User logged in
DEBUG: Fetching wishlist
INFO: Wishlist fetched
DEBUG: Creating wish
INFO: Wish created

We emit one comprehensive event per function call with all relevant context:

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "b7ad6b7169203331",
  "event_type": "convex.function",
  "source": "convex",
  "duration_ms": 45,
  "function_name": "wishlists.create",
  "user_id": "jx83mf92k4n5p6q7r8s9",
  "user_authenticated": true,
  "wishlist_id": "jh72kd8s9f3n4m5p6q7r",
  "input_title_length": 11,
  "outcome": "success"
}

Key Principles

1. Flat structure - No nested objects, use descriptive key prefixes (user_id, wishlist_title_length)
2. High cardinality - Include specific IDs and values, not just categories
3. One event per function - Each Convex function emits exactly one event
4. Trace ID correlation - Events can be correlated when client-side propagation is added
5. Simple API - Just mutate ctx.event in your handler, emission is automatic

Design Decisions

Decision	Choice	Rationale
Package structure	Inline in authHelpers.ts	No separate package needed; keeps it simple
ID generation	crypto.getRandomValues()	Works in Convex runtime, no external deps
OTel SDK	Not used	Convex is serverless; OTel SDK requires Node.js
Client trace propagation	Deferred	90% of debugging value comes from Convex traces
Error handling	withErrorCapture wrapper	Explicit, opt-in per handler

---

Architecture

Current App Architecture

┌─────────────────────────────────────────────────────────────────────────────────┐
│                              DATA FLOW                                          │
├─────────────────────────────────────────────────────────────────────────────────┤
│                                                                                 │
│  ┌──────────────────────────────────────────────────────────────────────────┐  │
│  │ CLIENT (Browser) - apps/web                                               │  │
│  │                                                                           │  │
│  │  User Action (e.g., "Create Wishlist")                                    │  │
│  │       │                                                                   │  │
│  │       ▼                                                                   │  │
│  │  useConvexMutation(api.wishlists.create)                                  │  │
│  │       │                                                                   │  │
│  │       │  Direct WebSocket to Convex                                       │  │
│  │       │  (No HTTP server in between for most operations)                  │  │
│  │       ▼                                                                   │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
│                                      │                                          │
│  ┌───────────────────────────────────▼──────────────────────────────────────┐  │
│  │ CONVEX BACKEND - packages/convex-backend/convex                           │  │
│  │                                                                           │  │
│  │  ┌─────────────────────────────┐                                          │  │
│  │  │ authenticatedMutationWithRLS │  (existing builder + telemetry)         │  │
│  │  │ • Authenticate user          │                                          │  │
│  │  │ • Wrap db with RLS           │                                          │  │
│  │  │ • Create ctx.event           │  ◄── NEW                                │  │
│  │  │ • Start timing               │  ◄── NEW                                │  │
│  │  └─────────────────────────────┘                                          │  │
│  │       │                                                                   │  │
│  │       │  Handler executes                                                 │  │
│  │       │  ctx.event.wishlist_id = "..."                                    │  │
│  │       ▼                                                                   │  │
│  │  ┌─────────────────────────────┐                                          │  │
│  │  │ onSuccess → emit(event)     │──────► console.log (Convex dashboard)   │  │
│  │  │ withErrorCapture → emit     │──────► (on errors too)                  │  │
│  │  └─────────────────────────────┘                                          │  │
│  └──────────────────────────────────────────────────────────────────────────┘  │
│                                                                                 │
└─────────────────────────────────────────────────────────────────────────────────┘

Why This Architecture?

1. Direct Convex calls - Your app uses useConvexMutation/convexQuery directly from the client via WebSockets, not HTTP server functions
2. Existing custom builders - You already have authenticatedQueryWithRLS, authenticatedMutationWithRLS using convex-helpers
3. RLS integration - Telemetry is added alongside existing auth and RLS logic
4. No separate package needed - Everything lives in authHelpers.ts

---

Core Concepts

Wide Event Structure

type WideEvent = {
  // Correlation
  trace_id: string;           // 32-char hex, unique per event (or from client)
  span_id: string;            // 16-char hex, unique per event
  parent_span_id?: string;    // For future client propagation
  timestamp: string;          // ISO 8601

  // Classification
  event_type: string;         // "convex.function"
  source: "convex";           // Always "convex" for now
  outcome?: "success" | "error";
  duration_ms?: number;

  // Context
  service: string;            // "mywishbud-convex"
  function_name?: string;     // "wishlists.create" - set by handler

  // User (auto-populated by builders)
  user_id?: string;
  user_authenticated: boolean;

  // Error fields (populated on failure)
  error_type?: string;        // "ConvexError", "Error", etc.
  error_code?: string;        // "UNAUTHORIZED", "NOT_FOUND", etc.
  error_message?: string;
  error_stack?: string;       // Truncated to 500 chars

  // Extensible - handlers add custom fields
  [key: string]: unknown;
};

Flat Key Naming Convention

Instead of nested objects:

// DON'T do this
ctx.event.wishlist = { id: "123", type: "birthday" };

Use flat keys with prefixes:

// DO this
ctx.event.wishlist_id = "123";
ctx.event.wishlist_type = "birthday";
ctx.event.input_title_length = 11;
ctx.event.db_insert_count = 1;

Trace Args (Future Client Propagation)

All instrumented functions accept optional trace args:

const TraceArgs = {
  _traceId: v.optional(v.string()),
  _parentSpanId: v.optional(v.string()),
};

These are consumed by the builder and NOT passed to handlers. Currently, if not provided, trace IDs are generated server-side. When client propagation is added, clients will pass these args.

---

Implementation

File: packages/convex-backend/convex/lib/authHelpers.ts

import { ConvexError, v } from "convex/values";
import {
  customMutation,
  customQuery,
  customAction,
} from "convex-helpers/server/customFunctions";
import {
  type Rules,
  wrapDatabaseReader,
  wrapDatabaseWriter,
} from "convex-helpers/server/rowLevelSecurity";
import type { DataModel } from "../_generated/dataModel";
import type { MutationCtx, QueryCtx, ActionCtx } from "../_generated/server";
import { mutation, query, action } from "../_generated/server";
import { authComponent } from "../auth";

// ============================================================================
// TYPES
// ============================================================================

export type AuthUser = NonNullable<
  Awaited<ReturnType<typeof authComponent.safeGetAuthUser>>
>;

type WideEvent = {
  // Correlation
  trace_id: string;
  span_id: string;
  parent_span_id?: string;
  timestamp: string;

  // Classification
  event_type: string;
  source: "convex";
  outcome?: "success" | "error";
  duration_ms?: number;

  // Context
  service: string;
  function_name?: string;

  // User
  user_id?: string;
  user_authenticated: boolean;

  // Error fields
  error_type?: string;
  error_code?: string;
  error_message?: string;
  error_stack?: string;

  // Extensible
  [key: string]: unknown;
};

export interface TelemetryCtx {
  event: WideEvent;
  startTime: number;
}

// ============================================================================
// TELEMETRY HELPERS
// ============================================================================

const SERVICE_NAME = "mywishbud-convex";

function generateId(bytes: number): string {
  const arr = new Uint8Array(bytes);
  crypto.getRandomValues(arr);
  return Array.from(arr, (b) => b.toString(16).padStart(2, "0")).join("");
}

function createEvent(traceId?: string, parentSpanId?: string): WideEvent {
  return {
    trace_id: traceId ?? generateId(16),
    span_id: generateId(8),
    parent_span_id: parentSpanId,
    timestamp: new Date().toISOString(),
    event_type: "convex.function",
    source: "convex",
    service: SERVICE_NAME,
    user_authenticated: false,
  };
}

function captureError(event: WideEvent, err: unknown): void {
  event.outcome = "error";

  if (err instanceof ConvexError) {
    event.error_type = "ConvexError";
    const data = err.data as Record<string, unknown>;
    event.error_code = String(data?.code ?? "UNKNOWN");
    event.error_message = String(data?.message ?? err.message);
  } else if (err instanceof Error) {
    event.error_type = err.name;
    event.error_message = err.message;
    event.error_stack = err.stack?.slice(0, 500);
  } else {
    event.error_type = "UnknownError";
    event.error_message = String(err);
  }
}

function emit(event: WideEvent): void {
  console.log(JSON.stringify(event));
}

/**
 * Wrap handler body to capture errors and emit event on failure.
 * Use this in handlers that may throw errors you want to capture.
 */
export async function withErrorCapture<T>(
  ctx: TelemetryCtx,
  fn: () => Promise<T>
): Promise<T> {
  try {
    return await fn();
  } catch (err) {
    captureError(ctx.event, err);
    ctx.event.duration_ms = Date.now() - ctx.startTime;
    emit(ctx.event);
    throw err;
  }
}

// ============================================================================
// TRACE ARGS (consumed by builders, not passed to handlers)
// ============================================================================

const TraceArgs = {
  _traceId: v.optional(v.string()),
  _parentSpanId: v.optional(v.string()),
};

// ============================================================================
// OWNERSHIP RULES (RLS)
// ============================================================================

function ownershipRules(user: AuthUser): Rules<QueryCtx, DataModel> {
  return {
    wishlists: {
      read: async (_, doc) => doc.userId === user._id,
      insert: async (_, doc) => doc.userId === user._id,
      modify: async (_, doc) => doc.userId === user._id,
    },
    wishes: {
      read: async (ctx, doc) => {
        const wishlist = await ctx.db.get(doc.wishlistId);
        return wishlist?.userId === user._id;
      },
      insert: async (ctx, doc) => {
        const wishlist = await ctx.db.get(doc.wishlistId);
        return wishlist?.userId === user._id;
      },
      modify: async (ctx, doc) => {
        const wishlist = await ctx.db.get(doc.wishlistId);
        return wishlist?.userId === user._id;
      },
    },
  } satisfies Rules<QueryCtx, DataModel>;
}

// ============================================================================
// AUTHENTICATED QUERY WITH RLS + TELEMETRY
// ============================================================================

export const authenticatedQueryWithRLS = customQuery(query, {
  args: TraceArgs,
  input: async (ctx: QueryCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);

    const user = await authComponent.safeGetAuthUser(ctx);
    if (!user) {
      event.user_authenticated = false;
      captureError(
        event,
        new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" })
      );
      event.duration_ms = Date.now() - startTime;
      emit(event);
      throw new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" });
    }

    event.user_id = user._id;
    event.user_authenticated = true;

    return {
      ctx: {
        user,
        db: wrapDatabaseReader(ctx, ctx.db, ownershipRules(user)),
        event,
        startTime,
      },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

// ============================================================================
// AUTHENTICATED MUTATION WITH RLS + TELEMETRY
// ============================================================================

export const authenticatedMutationWithRLS = customMutation(mutation, {
  args: TraceArgs,
  input: async (ctx: MutationCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);

    const user = await authComponent.safeGetAuthUser(ctx);
    if (!user) {
      event.user_authenticated = false;
      captureError(
        event,
        new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" })
      );
      event.duration_ms = Date.now() - startTime;
      emit(event);
      throw new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" });
    }

    event.user_id = user._id;
    event.user_authenticated = true;

    return {
      ctx: {
        user,
        db: wrapDatabaseWriter(ctx, ctx.db, ownershipRules(user)),
        event,
        startTime,
      },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

// ============================================================================
// AUTHENTICATED QUERY (no RLS) + TELEMETRY
// ============================================================================

export const authenticatedQuery = customQuery(query, {
  args: TraceArgs,
  input: async (ctx: QueryCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);

    const user = await authComponent.safeGetAuthUser(ctx);
    if (!user) {
      event.user_authenticated = false;
      captureError(
        event,
        new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" })
      );
      event.duration_ms = Date.now() - startTime;
      emit(event);
      throw new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" });
    }

    event.user_id = user._id;
    event.user_authenticated = true;

    return {
      ctx: { user, event, startTime },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

// ============================================================================
// AUTHENTICATED MUTATION (no RLS) + TELEMETRY
// ============================================================================

export const authenticatedMutation = customMutation(mutation, {
  args: TraceArgs,
  input: async (ctx: MutationCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);

    const user = await authComponent.safeGetAuthUser(ctx);
    if (!user) {
      event.user_authenticated = false;
      captureError(
        event,
        new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" })
      );
      event.duration_ms = Date.now() - startTime;
      emit(event);
      throw new ConvexError({ code: "UNAUTHORIZED", message: "User not authenticated" });
    }

    event.user_id = user._id;
    event.user_authenticated = true;

    return {
      ctx: { user, event, startTime },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

// ============================================================================
// PUBLIC QUERY + TELEMETRY
// ============================================================================

export const publicQuery = customQuery(query, {
  args: TraceArgs,
  input: async (_ctx: QueryCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);
    event.user_authenticated = false;

    return {
      ctx: { event, startTime },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

// ============================================================================
// PUBLIC MUTATION + TELEMETRY
// ============================================================================

export const publicMutation = customMutation(mutation, {
  args: TraceArgs,
  input: async (_ctx: MutationCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);
    event.user_authenticated = false;

    return {
      ctx: { event, startTime },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

// ============================================================================
// AUTHENTICATED ACTION + TELEMETRY
// ============================================================================

export const authenticatedAction = customAction(action, {
  args: TraceArgs,
  input: async (ctx: ActionCtx, args) => {
    const startTime = Date.now();
    const event = createEvent(args._traceId, args._parentSpanId);
    
    // Note: Actions have different auth patterns
    // Adjust based on your action auth implementation
    event.user_authenticated = false;

    return {
      ctx: { event, startTime },
      args: {},
      onSuccess: () => {
        event.outcome = "success";
        event.duration_ms = Date.now() - startTime;
        emit(event);
      },
    };
  },
});

---

Usage Examples

Basic Query

// wishlists.ts
import { v } from "convex/values";
import { authenticatedQueryWithRLS } from "./lib/authHelpers";

export const getById = authenticatedQueryWithRLS({
  args: { id: v.id("wishlists") },
  handler: async (ctx, args) => {
    // Set function name for tracing
    ctx.event.function_name = "wishlists.getById";
    ctx.event.wishlist_id = args.id;

    const wishlist = await ctx.db.get(args.id);
    if (!wishlist) {
      ctx.event.wishlist_found = false;
      return null;
    }

    ctx.event.wishlist_found = true;
    
    const wishes = await ctx.db
      .query("wishes")
      .withIndex("by_wishlist", (q) => q.eq("wishlistId", args.id))
      .collect();

    ctx.event.wish_count = wishes.length;

    return { ...wishlist, wishes, wishCount: wishes.length };
  },
});

Mutation with Error Capture

// wishlists.ts
import { ConvexError, v } from "convex/values";
import {
  authenticatedMutationWithRLS,
  withErrorCapture,
} from "./lib/authHelpers";
import { ErrorCodes } from "./lib/errors";

export const create = authenticatedMutationWithRLS({
  args: {
    title: v.string(),
    description: v.optional(v.string()),
    type: wishlistType,
    isPublic: v.boolean(),
  },
  handler: async (ctx, args) => {
    ctx.event.function_name = "wishlists.create";
    ctx.event.input_title_length = args.title.length;
    ctx.event.input_is_public = args.isPublic;

    // Wrap in withErrorCapture to emit event on error
    return withErrorCapture(ctx, async () => {
      const now = Date.now();
      const id = await ctx.db.insert("wishlists", {
        userId: ctx.user._id,
        title: args.title,
        description: args.description,
        type: args.type,
        isPublic: args.isPublic,
        shareToken: args.isPublic ? crypto.randomUUID() : undefined,
        createdAt: now,
        updatedAt: now,
      });

      ctx.event.wishlist_id = id;
      ctx.event.db_insert_count = 1;

      return id;
    });
  },
});

export const remove = authenticatedMutationWithRLS({
  args: { id: v.id("wishlists") },
  handler: async (ctx, args) => {
    ctx.event.function_name = "wishlists.remove";
    ctx.event.wishlist_id = args.id;

    return withErrorCapture(ctx, async () => {
      const wishlist = await ctx.db.get(args.id);
      if (!wishlist) {
        throw new ConvexError({
          code: ErrorCodes.NOT_FOUND,
          message: "Wishlist not found",
        });
      }

      const wishes = await ctx.db
        .query("wishes")
        .withIndex("by_wishlist", (q) => q.eq("wishlistId", args.id))
        .collect();

      if (wishes.length > 0) {
        ctx.event.wish_count = wishes.length;
        throw new ConvexError({ code: ErrorCodes.WISHLIST_HAS_WISHES });
      }

      await ctx.db.delete(args.id);
      ctx.event.db_delete_count = 1;

      return { success: true };
    });
  },
});

Public Query (No Auth Required)

// wishlists.ts
import { query } from "./_generated/server";
import { publicQuery } from "./lib/authHelpers";
import { authComponent } from "./auth";

export const getByShareToken = publicQuery({
  args: { shareToken: v.string() },
  handler: async (ctx, args) => {
    ctx.event.function_name = "wishlists.getByShareToken";
    ctx.event.share_token_prefix = args.shareToken.slice(0, 8);

    const wishlist = await ctx.db
      .query("wishlists")
      .withIndex("by_share_token", (q) => q.eq("shareToken", args.shareToken))
      .first();

    if (!wishlist?.isPublic) {
      ctx.event.wishlist_found = false;
      return null;
    }

    ctx.event.wishlist_found = true;
    ctx.event.wishlist_id = wishlist._id;

    // ... rest of handler
  },
});

---

Example Output

Success Event

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a2fb4a1d1a96d312",
  "timestamp": "2026-01-11T17:35:00.000Z",
  "event_type": "convex.function",
  "source": "convex",
  "outcome": "success",
  "duration_ms": 45,
  "service": "mywishbud-convex",
  "function_name": "wishlists.create",
  "user_id": "jx83mf92k4n5p6q7r8s9",
  "user_authenticated": true,
  "input_title_length": 11,
  "input_is_public": true,
  "wishlist_id": "jh72kd8s9f3n4m5p6q7r",
  "db_insert_count": 1
}

Error Event (Business Logic)

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a2fb4a1d1a96d312",
  "timestamp": "2026-01-11T17:35:00.000Z",
  "event_type": "convex.function",
  "source": "convex",
  "outcome": "error",
  "duration_ms": 23,
  "service": "mywishbud-convex",
  "function_name": "wishlists.remove",
  "user_id": "jx83mf92k4n5p6q7r8s9",
  "user_authenticated": true,
  "wishlist_id": "jh72kd8s9f3n4m5p6q7r",
  "wish_count": 5,
  "error_type": "ConvexError",
  "error_code": "WISHLIST_HAS_WISHES",
  "error_message": "Cannot delete wishlist with wishes"
}

Error Event (Auth Failure)

{
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "a2fb4a1d1a96d312",
  "timestamp": "2026-01-11T17:35:00.000Z",
  "event_type": "convex.function",
  "source": "convex",
  "outcome": "error",
  "duration_ms": 5,
  "service": "mywishbud-convex",
  "user_authenticated": false,
  "error_type": "ConvexError",
  "error_code": "UNAUTHORIZED",
  "error_message": "User not authenticated"
}

---

Querying Events (Convex Dashboard)

In the Convex dashboard logs, filter by JSON fields:

# All errors
outcome:error

# Specific function
function_name:wishlists.create

# Specific user's actions
user_id:jx83mf92k4n5p6q7r8s9

# Slow functions (if you have log search)
duration_ms:>100

# Specific error code
error_code:WISHLIST_HAS_WISHES

# Trace correlation (when client propagation is added)
trace_id:4bf92f3577b34da6a3ce929d0e0e4736

---

Future Enhancements

Phase 2: Client-Side Trace Propagation

When you need to correlate multiple Convex calls from one user action:

// apps/web/src/lib/telemetry.ts
function generateId(bytes: number): string {
  const arr = new Uint8Array(bytes);
  crypto.getRandomValues(arr);
  return Array.from(arr, (b) => b.toString(16).padStart(2, "0")).join("");
}

export function startTrace() {
  return {
    traceId: generateId(16),
    spanId: generateId(8),
  };
}

export function getConvexTraceArgs(trace: { traceId: string; spanId: string }) {
  return {
    _traceId: trace.traceId,
    _parentSpanId: trace.spanId,
  };
}

// Usage in component
const trace = startTrace();
await createWishlist({ title: "Birthday", ...getConvexTraceArgs(trace) });
await addWish({ wishlistId, title: "Gift", ...getConvexTraceArgs(trace) });
// Both events will share the same trace_id

Phase 3: External Log Shipping

When you need logs in PostHog, Axiom, or similar:

1. Set up a Convex HTTP action that receives log events
2. Forward to external service via fetch
3. Or use Convex's built-in log streaming (if available)

Phase 4: Server Function Tracing

If you add TanStack server functions that call Convex:

// apps/web/src/lib/server-functions.ts
export const createWishlistServer = createServerFn({ method: "POST" })
  .validator((data: { title: string }) => data)
  .handler(async ({ data, request }) => {
    const startTime = Date.now();
    const traceId = generateId(16);
    const spanId = generateId(8);
    
    // ... call Convex with trace args
    // ... emit server-side event
  });

---

Dependencies

No new dependencies required. The implementation uses:

• convex-helpers (already installed: ^0.1.109)
• crypto.getRandomValues() (built into Convex runtime)

---

Summary

Component	What it Does
createEvent()	Creates wide event with trace IDs
captureError()	Populates error fields on event
emit()	Logs event as JSON to console
withErrorCapture()	Wrapper to catch and emit errors
authenticatedQueryWithRLS	Query builder with auth + RLS + telemetry
authenticatedMutationWithRLS	Mutation builder with auth + RLS + telemetry
publicQuery	Query builder with telemetry only
ctx.event	Wide event object - mutate in handlers
ctx.startTime	Start timestamp for duration calculation

---

References

• Logging Sucks - Wide Events Article
• convex-helpers Custom Functions
• W3C Trace Context
• Convex Runtimes