Gmail Message-ID Deduplication Problem

The Problem

Gmail deduplicates messages by Message-ID within a single mailbox. When forwarding mail from multiple recipients through a single Google Workspace account, CC'd messages only reach one person.

Scenario

1. External sender sends email to [email protected] and [email protected] (CC'd)
2. Both copies have identical Message-ID: <[email protected]>
3. Postfix forwards both to [email protected] (Workspace account with filters)
4. Gmail sees same Message-ID twice → keeps only ONE copy
5. Only one Delivered-To header survives
6. Workspace filters forward to personal Gmail based on Delivered-To
7. Result: only Alice OR Bob gets the email, not both

Why This Happens

Gmail's dedup is per-mailbox. When using a single Workspace account as a forwarding hub, all incoming mail hits the same mailbox before filters run. By then, the duplicate is already discarded.

The Solution: gws-forward

Distribute mail across multiple Workspace accounts (bbfilter, bbfilter2, bbfilter3), each with identical filters. A custom Postfix pipe transport tracks which Message-IDs have gone to which accounts.

How It Works

                                    ┌─► [email protected]  ─► filters ─► [email protected]
                                    │                                    ─► [email protected]
sender ─► camissa MX ─► gws-forward ┼─► [email protected] ─► filters ─► [email protected]
                           binary   │                                    ─► [email protected]
                                    └─► [email protected] ─► filters ─► [email protected]
                                                                         ─► [email protected]

1. Postfix invokes gws-forward as a pipe transport for each recipient
2. gws-forward reads the message, extracts Message-ID
3. SQLite tracks which accounts have seen this Message-ID
4. First recipient → preferred account (hash-based)
5. Second recipient (same Message-ID) → different account
6. Third recipient → third account
7. Each Workspace account has identical filters that forward based on Delivered-To

Concurrency Handling

Postfix may deliver the same message to multiple recipients simultaneously. The binary uses SQLite's BEGIN IMMEDIATE transaction to atomically select and record account assignments, preventing two processes from picking the same account.

Overflow Handling (4+ recipients)

With only 3 accounts, the 4th+ recipient would cause Gmail dedup again. Solution:

1. Modify the Message-ID: <[email protected]> → <[email protected]>
2. Send warning email to affected recipients
3. Risk: breaks DKIM if Message-ID is signed (~8% of messages)

Cost

3 × ~R120/month = ~R360/month (vs R2565/month for 11 individual Workspace accounts)

Implementation

Go binary: gws-forward (if published) or local at ~/work/gws-forward/

Postfix master.cf

gwsforward unix - n n - - pipe
  flags=DRhu user=virtual:virtual
  argv=/usr/local/bin/gws-forward ${recipient} ${sender} ${queue_id}

Key Files

File	Purpose
/usr/local/bin/gws-forward	Go binary
/etc/gws-forward.conf	Config (accounts, DB path, logging)
/var/lib/gws-forward/messages.db	SQLite tracking DB
/var/log/gws-forward/gws-forward.log	Logs with Postfix queue ID

Cleanup

# /etc/cron.d/gws-forward-cleanup
0 3 * * * root /usr/local/bin/gws-forward --cleanup

Uses retention_hours from config (default 48h). Message-IDs don't repeat after that window.

Manual Testing

Test gws-forward before going live using --dry-run-full (full routing with DB but skip sendmail):

# Create test config
cat > /tmp/test-gws.conf << 'CONF'
[database]
path = /tmp/test-gws.db
retention_hours = 48

[workspace]
accounts = [email protected],[email protected],[email protected]

[warning]
enabled = false

[logging]
level = debug
file = /tmp/test-gws.log
CONF

# Clean slate
rm -f /tmp/test-gws.db /tmp/test-gws.log

# Test first recipient
echo -e "Message-ID: <[email protected]>\nFrom: [email protected]\nSubject: Test\n\nBody" | \
  ./gws-forward --config /tmp/test-gws.conf --dry-run-full [email protected] [email protected] Q001

# Test second recipient (same Message-ID) - should get different account
echo -e "Message-ID: <[email protected]>\nFrom: [email protected]\nSubject: Test\n\nBody" | \
  ./gws-forward --config /tmp/test-gws.conf --dry-run-full [email protected] [email protected] Q002

# Inspect DB
sqlite3 /tmp/test-gws.db "SELECT * FROM message_routing"

# Check logs
cat /tmp/test-gws.log

Testing overflow (4+ recipients):

rm -f /tmp/test-gws.db

# Fill all 3 accounts
for r in alice bob charlie; do
  echo -e "Message-ID: <[email protected]>\nFrom: [email protected]\nSubject: Test\n\nBody" | \
    ./gws-forward --config /tmp/test-gws.conf --dry-run-full ${r}@bluebird.co.za [email protected] Q$r
done

# 4th recipient triggers overflow
echo -e "Message-ID: <[email protected]>\nFrom: [email protected]\nSubject: Test\n\nBody" | \
  ./gws-forward --config /tmp/test-gws.conf --dry-run-full [email protected] [email protected] Qdave

# Check logs for overflow warning
grep -i overflow /tmp/test-gws.log

Modes:

Flag	DB Access	Sends Mail	Use Case
--dry-run	No	No	Quick header parsing check
--dry-run-full	Yes	No	Full routing test, records to DB
(none)	Yes	Yes	Production

Why "Hare-Brained"

• Relies on Gmail's undocumented dedup behavior
• Overflow breaks DKIM for some senders
• Adds complexity and a new failure mode to mail delivery

But it works, and it's cheaper than 11 Workspace accounts.