LibreChat Monitoring & Telemetry Documentation

Version: v0.8.1-rc1 Repository: github.com/danny-avila/LibreChat Last Updated: 2025-11-24

Overview
Current Monitoring Implementations
Data Being Captured
Configuration
Extension Points
Monitoring Gaps & Opportunities

Overview

LibreChat implements a comprehensive Winston-based logging and monitoring system that tracks user activity, API interactions, token usage, security violations, and system errors. This document provides a complete reference of what is currently monitored and how the system can be extended.

Key Monitoring Components

graph TB
    subgraph "Logging System"
        A[Winston Logger]
        B[Daily Rotating Files]
        C[Console Output]
        D[Sensitive Data Redaction]
    end

    subgraph "Tracking Systems"
        E[Token & Balance]
        F[Violation Tracking]
        G[User Activity]
        H[Error Tracking]
    end

    subgraph "Storage"
        I[File Logs]
        J[MongoDB]
        K[Redis Cache]
    end

    A --> B
    A --> C
    A --> D
    E --> J
    F --> K
    G --> J
    H --> I
    H --> J

Current Monitoring Implementations

1. Logging System (Winston-Based)

1.1 Main Logger Configuration

File: packages/data-schemas/src/config/winston.ts

Logging Levels:

{
  error: 0,    // Critical errors
  warn: 1,     // Warning messages
  info: 2,     // Informational messages
  http: 3,     // HTTP requests
  verbose: 4,  // Verbose debug info
  debug: 5,    // Debug messages
  activity: 6, // Custom activity tracking
  silly: 7     // Most verbose level
}

Log Transports:

Error Logs - winston.ts#L70-L76
- File: error-%DATE%.log
- Max Size: 20MB
- Max Files: 14 days
- Level: error only
Debug Logs - winston.ts#L77-L84
- File: debug-%DATE%.log
- Enabled when: DEBUG_LOGGING=true
- Max Size: 20MB
- Max Files: 14 days
Console Output - winston.ts#L85-L102
- Conditional based on DEBUG_CONSOLE
- Supports JSON format (CONSOLE_JSON=true)
- Color-coded for development

1.2 Sensitive Data Redaction

File: api/config/parsers.js

Redaction Patterns: parsers.js#L8-L13

const redactionPatterns = [
  /sk-[^\s]+/g,           // OpenAI API keys
  /Bearer\s+[^\s]+/gi,    // Bearer tokens
  /apikey[=:]\s*[^\s&]+/gi, // API key parameters
  /key=[^&\s]+/gi         // Query parameter keys
];

String Truncation: parsers.js#L67-L71

Console JSON: 255 characters (configurable via CONSOLE_JSON_STRING_LENGTH)
Debug messages: 150 characters (configurable via DEBUG_MESSAGE_LENGTH)

1.3 Specialized Loggers

MeiliSearch Logger: packages/data-schemas/src/config/meiliLogger.ts

Dedicated logger for search indexing operations
File: meiliSync-%DATE%.log
Tracks document indexing and sync operations

Login Logger: api/utils/logger.js

Simple file-based login tracking
File: login-logs.log

2. Token & Balance Tracking

2.1 Transaction System

Core Implementation: api/models/Transaction.js

Transaction Creation: Transaction.js#L19-L137

// Tracks:
// - User ID
// - Token type (prompt/completion/credits)
// - Raw amount and multiplier
// - Balance before/after
// - Conversation context

Balance Update Logic: Transaction.js#L43-L137

Optimistic concurrency control
10 retry attempts on conflicts
Atomic balance updates with version checking
Detailed logging at each step

2.2 Token Spending

File: api/models/spendTokens.js

Main Function: spendTokens.js#L15-L59

// Logs at line 17-24:
logger.debug('[spendTokens] conversationId...', {
  promptTokens,
  completionTokens,
  generations: tokenUsage?.generations
});

What's Tracked:

Prompt tokens (input)
Completion tokens (output)
Structured tokens (input_read, input_write for Claude)
Cache tokens (for Anthropic's prompt caching)
Image generation costs
Custom token types per endpoint

2.3 Balance Checking

File: api/models/balanceMethods.js

Balance Check Logic: balanceMethods.js#L40-L50

logger.debug('[Balance.check] Initial state', {
  user, model, endpoint, valueKey, tokenType,
  amount, balance, multiplier, endpointTokenConfig
});

Auto-Refill System: balanceMethods.js#L52-L73

Automatic balance top-up when depleted
Configurable refill amount and interval
Transaction creation for each refill
Logging of refill events and failures

3. Violation & Security Tracking

3.1 Violation Types

File: api/cache/getLogStores.js

All Violation Types: getLogStores.js#L5-L19

export enum ViolationTypes {
  LOGINS = 'logins',                    // Failed login attempts
  CONCURRENT = 'concurrent',            // Concurrent request violations
  NON_BROWSER = 'non_browser',          // Non-browser requests
  MESSAGE_LIMIT = 'message_limit',      // Message rate limits
  REGISTRATIONS = 'registrations',      // Registration rate limits
  TOKEN_BALANCE = 'token_balance',      // Insufficient balance
  TTS_LIMIT = 'tts_limit',             // Text-to-speech limits
  STT_LIMIT = 'stt_limit',             // Speech-to-text limits
  CONVO_ACCESS = 'convo_access',       // Unauthorized access
  TOOL_CALL_LIMIT = 'tool_call_limit', // Tool usage limits
  FILE_UPLOAD_LIMIT = 'file_upload_limit',     // File upload limits
  VERIFY_EMAIL_LIMIT = 'verify_email_limit',   // Email verification
  RESET_PASSWORD_LIMIT = 'reset_password_limit', // Password reset
  ILLEGAL_MODEL_REQUEST = 'illegal_model_request' // Unauthorized model access
}

3.2 Violation Logging

File: api/cache/logViolation.js

Main Function: logViolation.js#L17-L46

// Tracks:
// - User ID
// - Violation type
// - Current and previous violation count
// - Timestamp
// - Severity score (default: 1)

Persistent Logging: logViolation.js#L48-L59

All violations logged to file storage
Includes user ID, type, count, timestamp

3.3 Ban System

File: api/cache/banViolation.js

Ban Logic: banViolation.js#L21-L79

// Ban triggered when:
// violationCount % BAN_INTERVAL === 0

// Actions taken:
// 1. Store ban in MongoDB with TTL
// 2. Delete all user sessions
// 3. Clear refresh tokens
// 4. Log ban event with duration
// 5. Support both user ID and IP bans

Configuration:

BAN_VIOLATIONS - Enable/disable auto-banning
BAN_INTERVAL - Violations before ban (default: 20)
BAN_DURATION - Ban duration in ms (default: 600000 = 10 min)

4. Database Operation Logging

4.1 Message Operations

File: api/models/Message.js

Save Operations: Message.js#L141-L148

logger.debug('[Message save] Saving message', {
  messageId,
  conversationId,
  context: 'api/server/utils/streamResponse.js - sendError'
});

Update Operations: Message.js#L165-L172

Logs message updates with full context
Includes caller identification

Delete Operations: Message.js#L189-L196

Logs message deletions
Includes conversation context

4.2 User Operations

File: api/server/controllers/UserController.js

Account Deletion: UserController.js#L45-L52

logger.info(`User deletion request`, {
  email: user.email,
  id: user._id.toString()
});

S3 URL Refresh: UserController.js#L85

Logs avatar URL updates

4.3 Authentication Events

Login Controller: api/server/controllers/auth/LoginController.js

Login Errors: LoginController.js#L42-L48

Logs failed login attempts
Includes error details

Two-Factor Auth: api/server/controllers/TwoFactorController.js

2FA setup tracking
2FA validation attempts

5. Error Tracking

5.1 Error Middleware

File: packages/api/src/middleware/error.ts

Error Types Handled: error.ts#L15-L45

// 1. Validation errors (400)
// 2. Duplicate key errors (409) - MongoDB
// 3. Custom errors with statusCode
// 4. General 500 errors

Error Logging: error.ts#L30-L38

Logs error type, status code, message
Includes stack trace for 500 errors
Sanitizes sensitive data

5.2 Specialized Error Handlers

Assistants Errors: api/server/controllers/assistants/errors.js

Agents Errors: api/server/controllers/agents/errors.js

Streaming Errors: api/server/middleware/error.js

Handles SSE stream errors
Saves error messages to database

5.3 Process-Level Error Handling

File: api/server/index.js

Uncaught Exceptions: index.js#L185-L191

process.on('uncaughtException', (err) => {
  if (err.code === 'ABORT_ERR') return;
  logger.error('[uncaughtException]', err);
});

Signal Handlers: index.js#L193-L205

SIGTERM, SIGINT, SIGQUIT, SIGHUP
Graceful shutdown with cleanup

6. AI Provider Interaction Tracking

6.1 OpenAI Client

File: api/app/clients/OpenAIClient.js

Token Tracking: OpenAIClient.js#L672

await spendTokens(
  {
    user: this.user,
    model: this.modelOptions.model,
    context: 'message',
    conversationId,
  },
  { promptTokens, completionTokens },
);

6.2 Anthropic Client

File: api/app/clients/AnthropicClient.js

Advanced Token Tracking: AnthropicClient.js#L347-L365

// Tracks:
// - Standard prompt/completion tokens
// - Cache creation tokens
// - Cache read tokens (input_read)
// - Input write tokens

Stream Event Logging: AnthropicClient.js#L32

Logs message start/delta events
Tracks streaming metadata

6.3 Google Client

File: api/app/clients/GoogleClient.js

Token Tracking: GoogleClient.js#L847

Debug Logging: GoogleClient.js#L26

6.4 Tool & Image Generation

Flux Image API: api/app/clients/tools/structured/FluxAPI.js

Cost Tracking: FluxAPI.js#L68-L75

logger.debug('[FluxAPI] Generating image with parameters', {
  model,
  endpoint,
  cost: calculatedCost
});

7. API Request/Response Logging

7.1 Middleware Logging

Header Logging: api/server/middleware/logHeaders.js

X-Forwarded Headers: logHeaders.js#L5-L12

// Logs for OAuth requests:
// - X-Forwarded-Proto
// - X-Forwarded-Host
// - X-Forwarded-For

Data Being Captured

User Activity

Tracked Events:
  - User ID and authentication status
  - Login timestamps and failures
  - Account creation and deletion
  - Email verification attempts
  - Password reset requests
  - Session creation/destruction
  - Settings and preference changes
  - Avatar updates
  - Two-factor auth setup/usage

Storage:
  - MongoDB: User documents
  - File logs: Login events
  - Redis: Session data

API Interactions

Request Data:
  - Endpoint and HTTP method
  - User ID and conversation context
  - Model selection
  - Request headers (X-Forwarded-*)
  - Request parameters (size-limited)
  - Response status codes
  - Error responses with details

Storage:
  - File logs: Debug and error logs
  - MongoDB: Error messages in conversations

Token Usage & Costs

Token Metrics:
  - Prompt tokens (input)
  - Completion tokens (output)
  - Cache tokens (Anthropic)
  - Structured tokens (input_read, input_write)
  - Token multipliers per endpoint/model
  - Total cost in credits
  - Balance before/after

Transaction Data:
  - User ID and conversation ID
  - Timestamp of usage
  - Model and endpoint used
  - Raw and calculated amounts
  - Auto-refill events

Storage:
  - MongoDB: Transaction collection
  - File logs: Debug token calculations

Security Events

Violation Tracking:
  - Violation type and severity
  - User ID and IP address
  - Violation count (cumulative)
  - Timestamp
  - Ban status and duration

Violation Types:
  - Failed logins
  - Concurrent requests
  - Rate limit violations
  - Unauthorized access attempts
  - Model access violations
  - File upload violations

Storage:
  - Redis: Violation counters
  - MongoDB: Ban records
  - File logs: Violation history

Database Operations

Logged Operations:
  - Message CRUD with context
  - Conversation operations
  - User profile changes
  - File uploads and metadata
  - Session management
  - Permission updates

Metadata Logged:
  - Operation type
  - Document IDs
  - User context
  - Caller identification
  - Timestamps

Storage:
  - File logs: Debug logs
  - MongoDB: Operational data

Configuration

Environment Variables

Logging Configuration

# Enable JSON-formatted logs (for cloud deployments)
CONSOLE_JSON=false

# Enable debug file logging
DEBUG_LOGGING=true

# Enable debug console output
DEBUG_CONSOLE=false

# Provider-specific debug
DEBUG_OPENAI=false
DEBUG_PLUGINS=true

# String truncation for JSON logs
CONSOLE_JSON_STRING_LENGTH=255
DEBUG_MESSAGE_LENGTH=150

Reference: .env.example

Violation & Rate Limiting

# Login violations
LOGIN_VIOLATION_SCORE=1
LOGIN_MAX=7
LOGIN_WINDOW=5  # minutes

# Auto-ban configuration
BAN_VIOLATIONS=true
BAN_INTERVAL=20      # violations before ban
BAN_DURATION=600000  # milliseconds (10 min)

# Message rate limits
MESSAGE_VIOLATION_SCORE=1
MESSAGE_MAX=40
MESSAGE_WINDOW=5  # minutes

# Registration limits
REGISTRATION_MAX=5
REGISTRATION_WINDOW=1440  # minutes (24 hours)

Client-Side Analytics

# Google Tag Manager (optional)
ANALYTICS_GTM_ID=GTM-XXXXXXX

GTM Integration: client/src/hooks/Config/useAppStartup.ts

Log File Structure

api/logs/
├── error-2025-11-24.log       # Error-level only
├── debug-2025-11-24.log       # All levels (if DEBUG_LOGGING=true)
├── meiliSync-2025-11-24.log   # Search indexing
└── login-logs.log             # Login attempts

Retention: 14 days (configurable in Winston config) Max Size: 20MB per file Rotation: Daily with date suffix

Extension Points

1. Middleware Hooks

Server Initialization: api/server/index.js#L80-L148

// Pre-authentication (lines 80-96)
app.use(cors());
app.use(compression());
// → Add custom request tracking here

// Authentication (lines 103-114)
passport.use(jwtLogin());
// → Add auth event tracking here

// Route middleware (lines 116-146)
app.use('/api/messages', routes.messages);
// → Add per-route monitoring here

// Error handling (line 148)
app.use(ErrorController);
// → Extend error tracking here

2. Database Hooks

Mongoose Plugins: packages/data-schemas/src/models/plugins/mongoMeili.ts

// Available hooks:
schema.post('save', async (doc) => {
  // Add custom metrics collection
});

schema.post('updateOne', async (result) => {
  // Track update operations
});

schema.post('deleteOne', async (doc) => {
  // Log deletion events
});

schema.post('findOneAndUpdate', async (doc) => {
  // Monitor updates
});

Example Usage: Add monitoring alongside MeiliSearch sync

3. Token Tracking Hooks

Spend Tokens Function: api/models/spendTokens.js#L15-L59

// Current: Logs to Winston
logger.debug('[spendTokens]', { promptTokens, completionTokens });

// Extend: Add metrics collection
// → Send to Prometheus
// → Send to DataDog
// → Aggregate in time-series DB

Transaction Creation: api/models/Transaction.js#L19-L137

// Hook before transaction save
// → Track spending patterns
// → Trigger alerts on thresholds
// → Export to analytics

4. Violation Tracking Extensions

Add Custom Violations: api/cache/getLogStores.js#L5-L19

// Extend ViolationTypes enum:
export enum ViolationTypes {
  // ... existing types
  CUSTOM_VIOLATION = 'custom_violation',
  API_ABUSE = 'api_abuse',
  SUSPICIOUS_ACTIVITY = 'suspicious_activity'
}

// Add to namespaces object
const namespaces = {
  // ... existing namespaces
  [ViolationTypes.CUSTOM_VIOLATION]: 'custom_violation',
};

Use Existing Infrastructure: api/cache/logViolation.js

import { logViolation } from '~/cache';

// Log custom violation
await logViolation(req, {
  user_id: userId,
  type: 'custom_violation',
  errorMessage: 'Custom rule violated',
  score: 2 // Higher severity
});

5. Client-Side Event Tracking

GTM Integration Point: client/src/hooks/Config/useAppStartup.ts

// Current: Loads GTM if ANALYTICS_GTM_ID provided
// Extend: Add custom events

window.dataLayer = window.dataLayer || [];
window.dataLayer.push({
  event: 'custom_event',
  category: 'user_interaction',
  action: 'message_sent',
  label: modelName
});

6. Stream Response Monitoring

SSE Stream Events: api/server/utils/handleText.js

// Monitor streaming:
res.on('close', () => {
  // Track stream completion
  // → Duration metrics
  // → Bytes transferred
  // → Client disconnections
});

res.on('error', (err) => {
  // Track streaming errors
});

7. Performance Monitoring Points

Response Time Tracking:

// Add to Express middleware
app.use((req, res, next) => {
  const start = Date.now();

  res.on('finish', () => {
    const duration = Date.now() - start;
    // Log/export metrics:
    // → Endpoint
    // → Method
    // → Duration
    // → Status code
  });

  next();
});

Database Query Monitoring:

// Mongoose slow query logging
mongoose.set('debug', (collectionName, method, query, doc) => {
  const start = Date.now();
  // Track query execution time
});

Monitoring Gaps & Opportunities

✅ Currently Monitored

Error logging with stack traces
Token usage and costs
User activity (login, registration, etc.)
Security violations with auto-banning
Database operations (CRUD)
AI provider interactions
Transaction tracking with balance
File logging with rotation
Sensitive data redaction

❌ Not Currently Monitored

High Priority

API Response Times - Latency per endpoint
Database Query Performance - Slow query detection
Cache Hit/Miss Rates - Redis effectiveness
Request Throughput - Requests per second
Active Users - Concurrent user count
Queue Depths - Background job monitoring
Memory Usage - Heap and process memory
CPU Usage - Process CPU utilization

Medium Priority

Error Rates - Percentage of failed requests
Model Performance - Response quality metrics
Feature Usage - Which features are used most
Cost Analytics - Spending trends and forecasts
User Engagement - Session duration, messages per user
Geographic Distribution - User locations
Browser/Client Stats - Client environment data

Low Priority

A/B Test Tracking - Feature experiment results
Funnel Analytics - User journey tracking
Cohort Analysis - User retention metrics
Custom Business Metrics - Domain-specific KPIs

Recommended Monitoring Extensions

1. Add Prometheus Metrics

Implementation:

// Install: npm install prom-client
const promClient = require('prom-client');

// Define metrics
const httpRequestDuration = new promClient.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code']
});

const tokenUsage = new promClient.Counter({
  name: 'token_usage_total',
  help: 'Total tokens used',
  labelNames: ['user', 'model', 'type']
});

// Expose /metrics endpoint
app.get('/metrics', async (req, res) => {
  res.set('Content-Type', promClient.register.contentType);
  res.end(await promClient.register.metrics());
});

Integration Points:

Middleware for request duration
spendTokens() for token metrics
Error middleware for error rates

2. Add Structured Logging (JSON)

Already Supported: Set CONSOLE_JSON=true

Extend to External Services:

// Winston transport for DataDog/Logstash/etc.
import { transports } from 'winston';

logger.add(new transports.Http({
  host: 'logs.datadoghq.com',
  port: 443,
  path: `/v1/input/${DD_API_KEY}`,
  ssl: true
}));

3. Add APM (Application Performance Monitoring)

DataDog Example:

// Install: npm install dd-trace --save
const tracer = require('dd-trace').init({
  service: 'librechat',
  env: process.env.NODE_ENV
});

// Auto-instruments Express, MongoDB, Redis

New Relic Example:

// Install: npm install newrelic --save
require('newrelic');
// Must be first line in server/index.js

4. Add Real-Time Dashboards

Grafana + Prometheus:

Export Prometheus metrics (see #1)
Configure Grafana data source
Import LibreChat dashboard template

Custom Dashboard Metrics:

Active users (from Redis sessions)
Token usage per hour
Error rate by endpoint
Response time percentiles
Model usage distribution

5. Add User Analytics

Extend GTM Integration:

// Track custom events
const trackEvent = (category: string, action: string, label?: string) => {
  window.dataLayer?.push({
    event: 'custom_event',
    category,
    action,
    label,
    timestamp: Date.now()
  });
};

// Usage examples:
trackEvent('chat', 'message_sent', modelName);
trackEvent('agent', 'created', agentType);
trackEvent('tool', 'executed', toolName);

6. Add Health Checks

Enhanced Health Endpoint:

// api/server/index.js
app.get('/health', async (req, res) => {
  const health = {
    status: 'ok',
    timestamp: new Date().toISOString(),
    uptime: process.uptime(),
    checks: {
      mongodb: await checkMongoDB(),
      redis: await checkRedis(),
      meilisearch: await checkMeiliSearch()
    }
  };

  const allHealthy = Object.values(health.checks)
    .every(check => check.status === 'ok');

  res.status(allHealthy ? 200 : 503).json(health);
});

Integration Examples

Example 1: Add Request Duration Metrics

File: Create api/server/middleware/metrics.js

const startTimes = new Map();

export const metricsMiddleware = (req, res, next) => {
  const requestId = `${req.method}-${req.path}-${Date.now()}`;
  startTimes.set(requestId, Date.now());

  // Capture response
  const originalSend = res.send;
  res.send = function(data) {
    const duration = Date.now() - startTimes.get(requestId);
    startTimes.delete(requestId);

    // Log metrics
    logger.info('[metrics]', {
      method: req.method,
      path: req.path,
      statusCode: res.statusCode,
      duration,
      user: req.user?.id
    });

    return originalSend.call(this, data);
  };

  next();
};

Usage:

// api/server/index.js
import { metricsMiddleware } from './middleware/metrics';

app.use(metricsMiddleware);

Example 2: Track Model Usage

File: Extend api/models/spendTokens.js

import { logger } from '@librechat/data-schemas';

// After existing spendTokens logic, add:
const modelUsageMetrics = {
  model: options.model,
  endpoint: options.endpoint,
  promptTokens,
  completionTokens,
  totalCost: amount,
  timestamp: new Date(),
  userId: options.user
};

// Export to metrics system
logger.info('[model_usage]', modelUsageMetrics);

// Optional: Store aggregated metrics in MongoDB
await ModelUsageMetric.create(modelUsageMetrics);

Example 3: Monitor Violation Trends

File: Create api/analytics/violationAnalytics.js

import { ViolationTypes } from '~/cache/getLogStores';
import { logger } from '@librechat/data-schemas';

export const getViolationStats = async (timeWindow = '24h') => {
  // Aggregate from Redis/MongoDB
  const stats = {};

  for (const type of Object.values(ViolationTypes)) {
    const count = await getViolationCount(type, timeWindow);
    stats[type] = count;
  }

  logger.info('[violation_stats]', stats);
  return stats;
};

// Schedule periodic reporting
setInterval(() => {
  getViolationStats().then(stats => {
    // Export to monitoring system
  });
}, 60 * 60 * 1000); // Every hour

Summary

Current State

LibreChat has comprehensive logging and tracking for:

✅ User activity and security events
✅ Token usage and costs
✅ Error tracking with context
✅ Database operations
✅ AI provider interactions
✅ Violation tracking with auto-banning

Strengths

Winston-based logging with multiple transports
Sensitive data redaction built-in
Transaction system for financial tracking
Violation system with configurable thresholds
Well-structured code with clear extension points
File rotation and retention policies

Gaps

No external metrics system (Prometheus, DataDog, etc.)
No real-time dashboards (Grafana, custom)
No APM integration (New Relic, DataDog APM)
No distributed tracing
Limited performance metrics

Next Steps

Add Prometheus metrics for real-time monitoring
Implement APM for performance insights
Create Grafana dashboards for visualization
Extend GTM integration for user analytics
Add health check endpoints for uptime monitoring
Implement alerting based on thresholds

Last Updated: 2025-11-24 Maintainer: LibreChat Development Team Contributing: See CONTRIBUTING.md

ChakshuGautam/librechat-monitoring-telemetry.md