Skip to content

Instantly share code, notes, and snippets.

View jmealo's full-sized avatar

Jeff Mealo jmealo

51 followers · 166 following
View GitHub Profile
@jmealo
jmealo / search-retry-feeder-crashloop-rca.md
Last active March 11, 2026 16:37
RCA: search-retry-feeder CrashLoopBackOff (Production, 2026-03-11)

RCA: search-retry-feeder CrashLoopBackOff (Production)

Date: 2026-03-11 Reported by: Diego via Slack/Datadog Investigated by: Jeff Mealo (with Claude Code) Severity: Low (self-recovering, no data loss) Status: Root cause identified, fix committed and pending deploy

Summary

@jmealo
jmealo / notification-sender-42-red-herrings.md
Created March 6, 2026 16:47
42 red herrings across 2 days of notification-sender queue backup investigation (March 5-6, 2026)

42 Red Herrings: Notification-Sender Queue Backup (March 5-6, 2026)

Two incidents, one root cause chain, 42 documented false signals across 2 days of investigation.

Actual root cause: AAA API at 8 replicas (some crash-looping) couldn't serve permissions queries from intel-requests-api fast enough. This caused cascading request queuing through intel-requests-api and incidents-api, starving notification-sender of API capacity. SQL queries were sub-millisecond throughout.

March 5: 29 Red Herrings (5-hour investigation)

@jmealo
jmealo / notification-sender-explain-queries-2026-03-06.md
Last active March 6, 2026 08:53
EXPLAIN ANALYZE queries for notification-sender bottleneck — real prod IDs from 2026-03-06

EXPLAIN ANALYZE Queries for Notification-Sender Hot Path

Source: Production logs, 2026-03-06 08:44 UTC Charter org_id: 9de5d801-235a-4451-8c89-d2c3974c71e8

All queries use real IDs extracted from prod notification-sender logs.

WARNING: Run these inside a BEGIN; ... ROLLBACK; transaction on a read replica if possible. EXPLAIN ANALYZE actually executes the query.

@jmealo
jmealo / notification-sender-bottleneck-analysis-2026-03-06.md
Last active March 6, 2026 15:31
notification-sender bottleneck deep dive — recursive CTE, API call chain, optimization targets (2026-03-06)
@jmealo
jmealo / notification-sender-triage-2026-03-06.md
Created March 6, 2026 08:36
notification-sender prod triage 2026-03-06 — SLA breach analysis and optimization targets

Notification-Sender Production Triage — 2026-03-06

Date: March 6, 2026 (~04:00–08:30 UTC) Environment: Production Service: notification-sender (v0.5.45) Severity: SLA breach — notifications delayed beyond 15-minute target

Executive Summary

@jmealo
jmealo / Cargo.toml
Created February 20, 2026 23:49
Gisual H3 Utility Lookup Optimization Results & Implementation
[package]
name = "wasm-h3-map"
version = "0.1.0"
edition = "2021"
[lib]
crate-type = ["cdylib"]
[dependencies]
wasm-bindgen = "0.2"
@jmealo
jmealo / schema-evolution-handling.md
Created February 5, 2026 01:35
pgsyncd: Schema Evolution Handling - Design Proposal

Schema Evolution Handling for pgsyncd

Author: Engineering Team Date: 2026-02-04 Status: Proposal Type: Feature Enhancement

Executive Summary

@jmealo
jmealo / Dockerfile
Last active March 18, 2025 18:01
CNPG Volume Snapshot Cleanup Kubernetes Cron Job
FROM python:3.9-slim
# Set environment variables
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1
# Install the Kubernetes Python client
RUN pip install --no-cache-dir kubernetes==32.0.1
@jmealo
jmealo / cnpg_dashboard.json
Created February 11, 2025 20:48
{
"__inputs": [
{
"name": "DS_PROMETHEUS",
"label": "Prometheus",
"description": "",
"type": "datasource",
"pluginId": "prometheus",
"pluginName": "Prometheus"
},
@jmealo
jmealo / install-steps
Last active November 6, 2024 14:02
Workaround for JetBrains and Electron-based Apps not rendering (black, pink or white) screen after waking from sleep on MacOS
brew install sleepwatcher
# write restart_apps.sh to ~/.wakeup
chmod +x ~/.wakeup
brew services start sleepwatcher
# your mileage may vary getting sleepwatcher to run the script, but, the script should work if executed manually