Skip to content

Instantly share code, notes, and snippets.

@davidfirst

davidfirst/watchman-exhaust.mjs

Last active March 13, 2026 20:00
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save davidfirst/713e33812617bcbd0b01e3d1b592fbd7 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save davidfirst/713e33812617bcbd0b01e3d1b592fbd7 to your computer and use it in GitHub Desktop.
Download ZIP
macOS FSEventStream limits (undocumented by Apple): 512 per-process, 1024 system-wide, NOT multiplicative. Diagnose watcher exhaustion in VS Code, Watchman, Webpack, etc.
Raw
watchman-exhaust.mjs
/**
* fsevents-exhaust — macOS FSEventStream limit tester
*
* Measures the maximum number of FSEventStream instances that macOS allows,
* both per-process and system-wide. This is useful for diagnosing
* FSEventStreamStart failures in tools like Watchman, VS Code, Webpack, etc.
*
* Background:
* macOS's fseventsd daemon enforces a hard limit on FSEventStream clients.
* This limit is NOT documented by Apple. The per-process limit is 512
* (confirmed via `log show --predicate 'process == "fseventsd"'` which
* reports: "too many clients for pid <PID> (limit 512)").
* There may also be a system-wide limit (some sources report 1024).
* When the limit is reached, FSEventStreamStart() silently returns false,
* causing watchers to fail with no clear error from the OS.
*
* Modes:
*
* Default (no flags) — Per-process limit test
* Creates watchman watches via the watchman unix socket until
* FSEventStreamStart fails inside the watchman server process.
* All streams are in a single PID (watchman), so this measures
* the per-process limit (typically 512).
* Requires: watchman installed and running.
* Cleanup: shuts down watchman server to release all streams.
*
* --system — System-wide limit test (1 stream per process)
* Compiles a tiny C helper on the fly (~50KB binary), then spawns
* separate processes each holding exactly 1 FSEventStream via the
* CoreServices API. Each process has its own PID, so this bypasses
* the per-process limit and measures the system-wide cap.
* Requires: Xcode Command Line Tools (for `cc`).
* Cleanup: kills all child processes and removes temp directory.
*
* --system-packed — System-wide limit test (512 streams per process)
* Like --system, but each process fills up its full per-PID quota (512)
* before spawning the next process. This reveals how the per-process
* and system-wide limits interact.
* Reports: process count, streams per process, and total streams.
* Requires: Xcode Command Line Tools (for `cc`).
* Cleanup: kills all child processes and removes temp directory.
*
* Usage:
* node fsevents-exhaust.mjs # test per-process limit (via watchman)
* node fsevents-exhaust.mjs --system # test system-wide limit (1 per process)
* node fsevents-exhaust.mjs --system-packed # test system-wide limit (512 per process)
*
* Notes:
* - Creates temp directories under $TMPDIR (auto-cleaned on exit)
* - The watchman CLI exits 0 even on failure (error is in JSON response)
* - Uses fs.realpathSync() to resolve /var → /private/var (macOS symlink)
* - The C helper uses dispatch queues (not CFRunLoop) to avoid deprecation warnings
* - The --system tests use a C helper instead of Node.js child processes
* because each Node.js process costs ~30MB RAM vs ~50KB for the C binary,
* allowing us to test thousands of streams without exhausting memory
*/
import fs from 'fs';
import os from 'os';
import path from 'path';
import net from 'net';
import { execSync, spawn } from 'child_process';
const mode = process.argv[2];
if (mode === '--system') {
testSystemWide();
} else if (mode === '--system-packed') {
testSystemWidePacked();
} else {
testPerProcess();
}
// ── Per-process test (default) ──────────────────────────────────────────
//
// Connects to the watchman server's unix socket and sends watch commands
// in series. Each watch creates one FSEventStream in the watchman server
// process. Continues until watchman reports FSEventStreamStart failure.
// Uses the socket directly (not the CLI) for speed (~1ms per watch vs
// ~150ms per CLI invocation).
function testPerProcess() {
const tmpBase = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'wm-exhaust-')));
const sockPath = execSync('watchman get-sockname 2>&1').toString();
const sock = JSON.parse(sockPath).sockname;
const dirs = [];
let count = 0;
console.log(`Testing PER-PROCESS FSEventStream limit (via watchman)...`);
console.log(`(temp dir: ${tmpBase})\n`);
const client = net.createConnection(sock);
let buffer = '';
client.on('data', (data) => {
buffer += data.toString();
// Watchman uses newline-delimited JSON over the socket
let newlineIdx;
while ((newlineIdx = buffer.indexOf('\n')) !== -1) {
const line = buffer.slice(0, newlineIdx);
buffer = buffer.slice(newlineIdx + 1);
let json;
try { json = JSON.parse(line); } catch { continue; }
// Watchman returns {"error": "..."} on failure (exit code is still 0)
if (json.error) {
console.log(`\n=== FAILED at stream #${count + 1} ===`);
console.log(json.error.split('\n')[0]);
cleanup();
return;
}
count++;
if (count % 50 === 0) process.stdout.write(` ${count} streams OK...\n`);
sendNext();
}
});
client.on('connect', () => sendNext());
client.on('error', (err) => { console.error('Socket error:', err.message); cleanup(); });
function sendNext() {
const dir = path.join(tmpBase, `d${count}`);
fs.mkdirSync(dir, { recursive: true });
dirs.push(dir);
client.write(JSON.stringify(['watch', dir]) + '\n');
}
function cleanup() {
console.log(`\nPer-process limit: ${count} FSEventStreams (in watchman PID)`);
console.log(`Cleaning up — shutting down watchman server...`);
try { execSync('watchman shutdown-server 2>/dev/null', { timeout: 10000 }); } catch {}
fs.rmSync(tmpBase, { recursive: true, force: true });
console.log(`Done.`);
client.destroy();
}
}
// ── Compile the C helper used by both --system modes ────────────────────
function compileHelper(tmpBase, maxStreams) {
const helperSrc = path.join(tmpBase, 'fsevents-helper.c');
const helperBin = path.join(tmpBase, 'fsevents-helper');
// C program that creates up to `max` FSEventStreams in a single process.
// argv[1] = base directory (creates subdirs d0, d1, ... inside it)
// argv[2] = max streams to create (default: 1)
// Prints the count of successfully created streams, then sleeps.
// If any stream fails, prints the count so far and exits.
fs.writeFileSync(helperSrc, `
#include <CoreServices/CoreServices.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/stat.h>
static void cb(ConstFSEventStreamRef s, void *i, size_t n, void *p,
const FSEventStreamEventFlags *f, const FSEventStreamEventId *e) {}
int main(int argc, char **argv) {
if (argc < 2) { fprintf(stderr, "usage: fsevents-helper <basedir> [max]\\n"); return 1; }
int max = (argc >= 3) ? atoi(argv[2]) : 1;
dispatch_queue_t q = dispatch_queue_create("w", DISPATCH_QUEUE_SERIAL);
int count = 0;
for (int i = 0; i < max; i++) {
char dir[1024];
snprintf(dir, sizeof(dir), "%s/d%d", argv[1], i);
mkdir(dir, 0755);
CFStringRef p = CFStringCreateWithCString(NULL, dir, kCFStringEncodingUTF8);
CFArrayRef paths = CFArrayCreate(NULL, (const void **)&p, 1, NULL);
FSEventStreamRef stream = FSEventStreamCreate(NULL, cb, NULL, paths,
kFSEventStreamEventIdSinceNow, 1.0, kFSEventStreamCreateFlagNoDefer);
FSEventStreamSetDispatchQueue(stream, q);
if (!FSEventStreamStart(stream)) {
break;
}
count++;
CFRelease(p);
CFRelease(paths);
}
printf("%d\\n", count);
fflush(stdout);
if (count > 0) pause();
return (count == 0) ? 1 : 0;
}
`);
execSync(`cc -w -o '${helperBin}' '${helperSrc}' -framework CoreServices`);
return helperBin;
}
// ── System-wide test: 1 stream per process (--system) ───────────────────
//
// Tests the system-wide FSEventStream limit by spawning separate processes,
// each holding exactly 1 FSEventStream. This bypasses the per-PID limit
// because each stream lives in its own process.
async function testSystemWide() {
const tmpBase = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'wm-exhaust-')));
console.log(`Testing SYSTEM-WIDE FSEventStream limit (1 stream per process)...`);
console.log(`Compiling helper...`);
const helperBin = compileHelper(tmpBase);
const children = [];
let count = 0;
let failed = false;
const BATCH = 50;
console.log(`Spawning processes, each holding 1 FSEventStream...\n`);
try {
while (!failed) {
const batch = [];
for (let i = 0; i < BATCH; i++) {
const dir = path.join(tmpBase, `p${count + i}`);
fs.mkdirSync(dir);
batch.push(new Promise((resolve) => {
const child = spawn(helperBin, [dir, '1'], { stdio: ['ignore', 'pipe', 'ignore'] });
let output = '';
child.stdout.on('data', (data) => {
output += data.toString();
const num = parseInt(output.trim());
if (!isNaN(num)) {
resolve(num > 0 ? { ok: true, child, count: num } : { ok: false, child });
}
});
child.on('error', (err) => resolve({ ok: false, child: null, error: err.message }));
child.on('exit', (code) => {
if (!output.trim()) resolve({ ok: false, child: null });
});
setTimeout(() => resolve({ ok: false, child, timeout: true }), 5000);
}));
}
const results = await Promise.all(batch);
for (let i = 0; i < results.length; i++) {
const r = results[i];
if (!r.ok) {
console.log(`\n=== FAILED at process #${count + i + 1} ===`);
if (r.timeout) console.log('Timed out waiting for FSEventStreamStart');
if (r.error) console.log(r.error);
else console.log('FSEventStreamStart returned false');
failed = true;
for (let j = 0; j < i; j++) {
if (results[j].child) children.push(results[j].child);
}
break;
}
if (r.child) children.push(r.child);
count++;
}
if (!failed && count % 100 === 0) process.stdout.write(` ${count} processes OK...\n`);
}
} finally {
console.log(`\nSystem-wide limit: ${count} FSEventStreams across ${count} processes (1 per process)`);
console.log(`Cleaning up ${children.length} processes...`);
for (const c of children) { try { c.kill(); } catch {} }
await new Promise(r => setTimeout(r, 500));
fs.rmSync(tmpBase, { recursive: true, force: true });
console.log(`Done.`);
process.exit(0);
}
}
// ── System-wide packed test: 512 streams per process (--system-packed) ──
//
// Each process fills its full per-PID quota (up to 512 streams) before we
// spawn the next process. This reveals how per-process and system-wide
// limits interact. For example:
// - If process 1 gets 512, process 2 gets 512, process 3 gets 0
// → system-wide limit is 1024
// - If process 1 gets 512, process 2 gets 400, process 3 gets 0
// → system-wide limit is 912
// Reports the count per process so you can see exactly how the budget
// is distributed.
async function testSystemWidePacked() {
const tmpBase = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'wm-exhaust-')));
const PER_PROCESS = 512;
console.log(`Testing SYSTEM-WIDE FSEventStream limit (${PER_PROCESS} streams per process)...`);
console.log(`Compiling helper...`);
const helperBin = compileHelper(tmpBase);
const children = [];
let totalStreams = 0;
let processNum = 0;
console.log(`Each process will grab up to ${PER_PROCESS} streams, then we spawn the next.\n`);
try {
while (true) {
processNum++;
const dir = path.join(tmpBase, `proc${processNum}`);
fs.mkdirSync(dir);
const got = await new Promise((resolve) => {
const child = spawn(helperBin, [dir, String(PER_PROCESS)], {
stdio: ['ignore', 'pipe', 'ignore']
});
let output = '';
child.stdout.on('data', (data) => {
output += data.toString();
const num = parseInt(output.trim());
if (!isNaN(num)) {
if (num > 0) {
children.push(child);
}
resolve(num);
}
});
child.on('error', () => resolve(0));
child.on('exit', () => {
if (!output.trim()) resolve(0);
});
setTimeout(() => resolve(-1), 10000);
});
if (got <= 0) {
console.log(` Process #${processNum}: 0 streams — DONE`);
break;
}
totalStreams += got;
console.log(` Process #${processNum}: ${got} streams (total: ${totalStreams})`);
// If this process got fewer than PER_PROCESS, the system is full
if (got < PER_PROCESS) {
console.log(` (got fewer than ${PER_PROCESS} — system limit reached)`);
break;
}
}
} finally {
console.log(`\n=== RESULT ===`);
console.log(`${processNum} processes, ${totalStreams} total FSEventStreams`);
console.log(`Per-process limit: ${PER_PROCESS}`);
console.log(`System-wide limit: ~${totalStreams}`);
console.log(`\nCleaning up ${children.length} processes...`);
for (const c of children) { try { c.kill(); } catch {} }
await new Promise(r => setTimeout(r, 500));
fs.rmSync(tmpBase, { recursive: true, force: true });
console.log(`Done.`);
process.exit(0);
}
}
@davidfirst
Copy link
Author

davidfirst commented Mar 13, 2026

macOS FSEventStream Limits (Undocumented)

Apple doesn't document FSEventStream limits, so we had to figure them out empirically.

Per-process limit: 512 streams
Confirmed by creating temp directories and running watchman watch on each until failure. The system log revealed:

fseventsd: [com.apple.fsevents:daemon] too many clients for pid 75904 (limit 512)

(via log show --predicate 'process == "fseventsd"' --last 2h | grep -i "too many\|limit\|client")

System-wide limit: 1024 streams
Confirmed by spawning many processes, each watching a folder, until failure. The system log showed:

fseventsd: [com.apple.fsevents:daemon] too many clients in system (limit 1024)

The limits are not multiplicative. The total is 1024, not 1024 × 512. If one process uses 512 streams and a second uses 512, the third process can't create any. Confirmed with the --system-packed flag below, which fills each process to its 512-stream quota before spawning the next.

Test it yourself

curl -sL https://gist.githubusercontent.com/davidfirst/713e33812617bcbd0b01e3d1b592fbd7/raw/watchman-exhaust.mjs -o /tmp/fsevents-exhaust.mjs
node /tmp/fsevents-exhaust.mjs                # per-process limit (via watchman)
node /tmp/fsevents-exhaust.mjs --system       # system-wide limit (1 stream per process)
node /tmp/fsevents-exhaust.mjs --system-packed # system-wide limit (512 per process)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment