Mar 12, 5:00-6:00 AM ET — SL Failure Forensics

Summary

• ETH DOWN (SL 887): Entry 88¢, SL triggered at 50.1¢. GTC sell placed → cancelled after 14s. FAK fallback floor (48¢) was above market (~35¢) → all attempts missed. 29.35 shares never sold. Total loss ~$25.83.
• BTC DOWN (SL 888): Entry 82¢, SL triggered at 52¢. GTC sell placed → filled on Polymarket at 60¢ ($18.26) but trader saw gtc_terminal_status:canceled. FAK fallback: "not enough balance/allowance" (shares already sold). Loss ~$6.95.

Key Findings

1. BTC GTC sell DID fill on-chain but trader status check returned "canceled" → infinite retry loop on already-sold position
2. ETH FAK floor too high — market crashed from 84.5¢ to ~35¢ by the time FAKs ran, but the 48¢ floor prevented all sell attempts from matching. NOT an empty orderbook — the floor was the problem.
3. Both SLs stuck in retry loops (50+ attempts each) burning cycles on dead/sold positions
4. entry_price = 0.00 bug also present on ETH SL 887 (avg_fill_price not recorded)
5. Account snapshot evidence confirms: ETH DOWN shares stayed at 29.35 throughout (never sold), BTC DOWN went from 30.58 → 0.14 shares at ~09:48 UTC (GTC fill reflected)

Account Snapshot Timeline (Polymarket position data)

Time (UTC)  | BTC DOWN shares | ETH DOWN shares | Cash     | ETH DOWN price
09:42:02    | 30.58           | 29.35           | $213.92  | 84.5¢
09:43:32    | 30.58           | 29.35           | $213.92  | 34.5¢  ← crash, both SLs triggered
09:45:02    | 30.58           | 29.35           | $213.92  | 3.5¢   ← still no sell
09:48:04    | 0.14 (SOLD!)    | 29.35 (stuck)   | $231.92  | 6.5¢   ← BTC GTC fill reflects (+$18)
09:49:34    | 0.14            | 29.35           | $231.92  | 37.5¢  ← price bounced, too late

Key observations:

• BTC GTC sell filled sometime between 09:42:10 (placement) and 09:42:25 (code cancelled it), but account snapshot didn't reflect until ~09:48 (CLOB settlement delay)
• ETH DOWN shares NEVER decreased — the sell genuinely never executed
• Cash increased by exactly ~$18 = 30.44 shares × 60¢ = BTC GTC sell at 60¢

ETH DOWN: Why the Sell Failed (Corrected Analysis)

Original theory: Empty orderbook, zero liquidity. Actual cause: FAK floor too high relative to crashed market.

Price data around trigger (ETH DOWN):

Minute | TWAP DOWN bid | Current Price
40     | 81.2¢         | —
41     | 78.7¢         | —
42     | 69.2¢         | 84.5¢
43     | 60.8¢         | 34.5¢  ← SL triggers, GTC sell placed
44     | 50.1¢         | 34.5¢  ← FAK running, all fail
45     | 37.7¢         | 3.5¢   ← market crashes past floor
46     | 30.3¢         | 3.5¢
47     | 14.9¢         | 6.5¢

The FAK ladder: [60¢, 57¢, 54¢, 51¢, 48¢] with effective_floor=0.48

The problem: By minute 43-44, current_price=34.5¢. Every FAK price (60¢ down to 48¢) was well above the actual market. The floor of 48¢ prevented any lower attempts. By minute 45 the price was 3.5¢ — way beyond recovery.

BTC vs ETH difference: BTC's GTC sell (at 99¢ sweep price) happened to fill at 60¢ within the 15-second window before the code cancelled it. ETH's GTC sell didn't fill in its 14-second window — thinner ETH book, and by the time FAKs kicked in, the market had already crashed past the floor.

Fix: The FAK floor (B5) needs to be much lower in volatile conditions, or the "protected" floor mode should have a panic override when the market has moved significantly past the stop price.

Root Cause Analysis — B1: fill_price=0 on resting GTC fills

The Complete Bug Chain

Polymarket WS sends TWO event types: order and trade.

• order events: Carry size_matched, status, side, asset_id, market (condition_id), id (order_id). No fill price field — parser hardcodes fill_price=None.
• trade events: Carry price (fill price), size, taker_order_id, maker_orders[], asset_id, market. Has the actual fill price.

What happens when a resting GTC buy fills:

1. WS sends order event with size_matched=N, status=filled → parser yields UserOrderEvent(fill_price=None)
2. _handle_user_event() passes fill_price=None → _apply_order_snapshot() coerces to fill_price=0.0
3. _persist_fill_progress() runs:
   • UPDATE orders SET avg_fill_price = COALESCE(NULL, avg_fill_price) → stays NULL (because fill_price if fill_price > 0 else None evaluates to None)
   • fill_price > 0 is False → no fill event recorded in order_fill_events
   • But _upsert_stop_loss_for_fill(fill_price=0.0) still runs → creates stop_loss with entry_price=0.0
4. If a trade event arrives later with the real price:
   • _apply_order_snapshot() checks new_filled <= (prev_filled + _EPSILON) → True (no new fill progress)
   • Returns early — never updates the stop_loss or order price!

Result: Stop loss has entry_price=0.0, order has avg_fill_price=NULL, no fill event recorded. PnL cascades to garbage everywhere.

Evidence from DB

Fill event sources (no user_ws_order events exist at all — only user_ws_trade):

source            | side | cnt | has_price | no_price
user_ws_trade     | BUY  |   5 |         5 |        0
user_ws_trade     | SELL |  23 |        23 |        0

Affected orders (all BUY, resting GTC, got order event but never got trade event):

Order  | Asset | Direction | avg_fill_price | sl_entry_price | requested_price
1112   | ETH   | DOWN      | NULL           | 0              | 0.88
1106   | ETH   | UP        | NULL           | 0              | 0.87
1076   | BTC   | DOWN      | NULL           | 0              | 0.90
1075   | ETH   | DOWN      | NULL           | 0              | 0.92
1048   | BTC   | UP        | NULL           | 0              | 0.90
1044   | BTC   | UP        | NULL           | 0              | 0.90

Key Insight: order events are NOT sufficient for price

Polymarket order events NEVER carry a fill price — they only report status changes and cumulative fill size. The fill price comes exclusively from trade events. But trade events don't always arrive (or may arrive before the order event, causing the "already filled" early-return).

Proposed Fix — B1

Option A (recommended): Fetch from CLOB API when fill_price is 0/None

• In _persist_fill_progress(), when fill_price <= 0 and source is user_ws_order:
  1. Call get_order(order_id) to get authoritative fill_price from CLOB API
  2. Use that price for both the order update and stop_loss creation
  3. If API also returns 0, use requested_price as last-resort fallback (fill can't be worse than limit price for BUY)
• Cost: One extra API call per resting GTC fill (~1-3 per hour)
• Benefit: Always-correct prices, no cascading PnL bugs

Option B: Allow trade events to update even when order already "filled"

• In _apply_order_snapshot(), don't early-return when new_filled <= prev_filled if fill_price > 0 and existing avg_fill_price is NULL
• Risk: More complex logic, race conditions between order/trade event processing

Option C: Don't create stop_loss until fill_price > 0

• Delay _upsert_stop_loss_for_fill() when fill_price <= 0
• Risk: Position unprotected during delay, trade event may never arrive

SELL Order Tracking Architecture

Two separate sell paths — important for understanding B2:

1. Take-Profit (TP) sells: Tracked in orders table with side='SELL', linked to stop_losses via tp_order_id. These get full fill monitoring via the same _apply_order_snapshot() path.

2. Stop-Loss (SL) sells: Placed by the SL engine directly. NOT in the orders table. Tracked only via stop_losses.sell_order_id. Fill detection relies on:

   • WS trade events → matched to stop_loss via load_sell_order_linkage()
   • data_api_external_sell backfill (settlement poller)
   • Settlement residual events

Key finding for B2: The GTC sell order 0x1f07... (SL 888) is NOT in the orders table at all. This means:

• The WS order event for this sell (with status:canceled) bypasses fill_monitor's _apply_order_snapshot() (which requires a matching DB row)
• The SL engine's own status-check logic sees "canceled" and enters FAK fallback
• Even if a WS trade event arrived showing the fill, the sell_fill_runtime would need to match it via sell_order_id on stop_losses — which WAS populated for SL 888

SELL fill events by source (DB evidence):

source                       | side | count | all_have_price
user_ws_trade                | SELL |    23 | yes (100%)
data_api_external_sell       | SELL |   314 | yes (100%)
settlement_residual          | SELL |   894 | 78% (701/894)
sell_order_snapshot_backfill  | SELL |    78 | yes (100%)

All SELL orders in the orders table (TP sells) have both condition_id and token_id populated. ✅

Frontend Bugs

F1: ETH 5am DOWN PnL shows $0.00 instead of -$25.83

• Trade: ETH DOWN, Mar 12 05:34, cost $25.83
• UI shows: 0.0% | $0.00 LOST
• Actual: -$25.83 total loss (position expired worthless, SL sell never executed)
• Root cause: entry_price = 0.00 in stop_losses (SL 887) due to avg_fill_price NULL on order 1112. PnL formula: (0 - 0) × shares = $0

F2: BTC 5am DOWN PnL shows -$25.22 instead of -$6.95

• Trade: BTC DOWN, Mar 12 05:37, cost $25.22
• UI shows: -100.0% | -$25.22 LOST
• Actual: -$6.95 loss (GTC sell filled at 60¢ on Polymarket for $18.26, but DB never recorded the fill)
• Root cause: GTC sell status misread as "canceled" → DB has no sell record → UI assumes 100% loss

F3: ETH 10pm DOWN PnL shows +$20.84 instead of ~+$1.55

• Trade: ETH DOWN, Mar 11 22:34, cost $25.00
• UI shows: +83.4% | +$20.84 WON
• Actual: ~+$1.55 profit (entry was 92¢, not 0¢)
• Root cause: Same entry_price = 0.00 bug (SL 872, order 1075). Inflates revenue calculation.

F4: ETH 2am UP SC outcome mismatch

• Trade: ETH UP SC, Mar 12 02:50, cost $16.86
• UI shows: +2.2% | +$0.38 WON
• DB shows: LOST | -$16.67
• Root cause: Unknown — DB outcome and PnL both wrong. UI may be correct here.

F5: ETH 11pm UP PnL $0.00 in DB

• Trade: ETH UP FC, Mar 11 23:34, cost $25.00
• UI shows: -21.7% | -$5.42 LOST
• DB shows: pnl = $0.00, outcome = lost
• Root cause: entry_price = 0.00 in SL 875. DB PnL wrong; UI may compute from different source.

Backend Bugs

B1: avg_fill_price NULL on some GTC order fills

• Impact: Causes entry_price = 0.00 in stop_losses → cascading PnL errors everywhere
• Affected orders (last 12h): 1075, 1076, 1081 (partial), 1106, 1112
• Pattern: Only affects SOME GTC fills — others (1063, 1072, 1091, 1102, 1103) record correctly
• Severity: HIGH — corrupts PnL for both UI and DB, affects modeler training data

B2: GTC sell order status misread as "canceled" when filled on-chain

• Impact: Trader enters infinite FAK retry loop on already-sold positions
• Evidence: BTC SL 888 GTC sell 0x1f07... placed at 60¢. Polymarket shows "Sold 30 Down at 60¢". Trader logs show gtc_terminal_status:canceled → fallback loop.
• Severity: HIGH — causes phantom retry loops, incorrect loss reporting, and potentially double-sells if balance were available

B3: No post-sell balance/position check before retrying

• Impact: FAK retries on already-sold positions get "not enough balance/allowance" errors indefinitely (50+ attempts observed)
• Fix: After any sell failure, check if shares are actually still held before retrying
• Severity: MEDIUM — wastes API calls, spams error logs, but doesn't cause additional monetary loss

B4: Settlement poller doesn't reconcile missed on-chain sells

• Impact: DB never learns about sells that happened outside trader's tracked orders
• Evidence: BTC SL 888 still shows remaining_shares = 30.75 and status = triggered hours after Polymarket sold the shares
• Severity: MEDIUM — affects portfolio tracking and aggregate PnL reporting

B5: FAK floor too high in crash scenarios (NEW)

• Impact: FAK sell ladder can't reach actual market price during fast crashes → position expires worthless instead of salvaging partial value
• Evidence: ETH SL 887 — market crashed to 34.5¢ but FAK floor was 48¢ (effective_floor=0.48, pct_floor=0.48). All FAK attempts at 60¢→48¢ got "no orders found to match" because bids were at ~35¢. Price then crashed to 3.5¢ by minute 45.
• Fix: Lower the floor in volatile conditions, or add a "panic mode" that drops the floor to near-zero when the market has moved significantly past stop_price. Alternatively, extend the GTC sell timeout (currently 14-15s) to give more time for the sweep order to fill before falling back to FAK.
• Severity: HIGH — directly caused ~$25.83 total loss on ETH. Without this bug, could have recovered ~$10 at 35¢.

Timeline — Key Events (UTC → ET-4)

ETH DOWN (SL 887) — Entry & Setup

09:34:33 (5:34) bot_settings_loaded              settings_version_id=4863
09:35:09 (5:35) tp_link_attempt_failed           attempt=1 reason=tp_order_placement_failed stop_loss_id=887 target_shares=29.35
09:35:17 (5:35) tp_order_placed                  stop_loss_id=887 tp_price=0.975 shares=29.0565 tp_order_id=0x1cdd...
09:35:17 (5:35) fill_stop_loss_upserted          asset=ETH entry_price=0.0 ← 🐛 BUG! shares=29.35 sl_id=887 stop_price=0.6
09:35:17 (5:35) fc_gtc_order_filled              asset=ETH fill_price=0.0 ← 🐛 BUG! order_id=0x19b5... source=user_ws_order status=filled

BTC DOWN (SL 888) — Entry & Setup

09:37:06 (5:37) stop_loss_upserted               asset=BTC entry_price=0.82 ✅ correct  sl_id=888 stop_price=0.6
09:37:11 (5:37) tp_order_placed                  stop_loss_id=888 tp_price=0.975 shares=30.4425 tp_order_id=0x9f9d...

BTC SL Trigger & Sell Attempt

09:42:10 (5:42) sl_raw_breach_triggered          asset=BTC raw_ask=0.53 raw_bid=0.52 sl_id=888 stop_price=0.6
09:42:10 (5:42) stop_loss_triggered_from_twap    asset=BTC
09:42:10 (5:42) order_cancelled                  order_id=0x9f9d... (TP cancelled)
09:42:10 (5:42) tp_cancelled_for_sl              sl_id=888
09:42:10 (5:42) stop_loss_triggered              asset=BTC entry_price=0.82 active_sell_shares=30.75 twap_bid=0.52
09:42:10 (5:42) sl_gtc_sell_placed               order_id=0x1f07... sl_id=888 ← GTC SELL AT 60¢ (stop_price)
                                                  ⚡ Polymarket: FILLED here (Sold 30 Down at 60¢ = $18.26)
                                                  ⚡ But trader doesn't know...
09:42:25 (5:42) sl_triggered_fallback_started    reason=gtc_timeout_status:live ← GTC still "live" after 15s
09:42:25 (5:42) order_cancelled                  order_id=0x1f07... ← CANCELLED THE GTC (but already filled on-chain!)
09:42:26 (5:42) market_sell_failed               "no orders found to match with FAK order" (book empty, BTC)
09:42:35 (5:42) sl_triggered_fallback_started    reason=gtc_terminal_status:canceled retry_count=1 ← sees "canceled"
09:42:36 (5:42) market_sell_failed               "no orders found to match" (FAK retry #1)
  ... (5 FAK attempts per burst, all fail — "no orders found" then "not enough balance/allowance")
  ... (retry loop continues every 30s: attempt counts 2→3→4→...→53+)

ETH SL Trigger & Sell Attempt (1 min after BTC)

09:43:10 (5:43) sl_raw_breach_triggered          asset=ETH raw_ask=0.51 raw_bid=0.501 sl_id=887 stop_price=0.6
09:43:10 (5:43) stop_loss_triggered_from_twap    asset=ETH
09:43:10 (5:43) order_cancelled                  order_id=0x1cdd... (TP cancelled)
09:43:11 (5:43) tp_cancelled_for_sl              sl_id=887
09:43:11 (5:43) stop_loss_triggered              asset=ETH entry_price=0.0 ← 🐛 active_sell_shares=29.35 twap_bid=0.501
09:43:11 (5:43) sl_gtc_sell_placed               order_id=0xd9d5... sl_id=887 ← GTC SELL AT ~50¢
09:43:25 (5:43) sl_triggered_fallback_started    reason=gtc_timeout_status:live ← GTC still "live" after 14s
09:43:25 (5:43) order_cancelled                  order_id=0xd9d5... ← CANCELLED GTC
09:43:26 (5:43) market_sell_failed               "no orders found to match" (ETH orderbook completely empty)
  ... (FAK ladder [0.6, 0.57, 0.54, 0.51, 0.48, 0.01] — ALL fail, no buyers at any price)
  ... (retry loop: 39+ attempts, all "no orders found to match")

Both Stuck in Infinite Retry Until Expiry

09:48:25→09:59:59  Both SLs retrying every ~10-30s with 5-6 FAK attempts per burst
                    BTC: "not enough balance/allowance" (shares already sold on-chain)
                    ETH: "no orders found to match" (genuinely no buyers)
09:59:58 (5:59)    ETH still trying with 1.254 seconds_to_expiry
09:59:59 (5:59)    Last ETH FAK attempt fails — market expires at 6:00

Portfolio Snapshots (showing the damage)

09:30:44 (5:30) portfolio_value=268.60  cash=264.96  positions=4  orders=0  (pre-entry)
09:35:15 (5:35) portfolio_value=268.46  cash=239.14  positions=5  orders=0  (ETH entered)
09:37:31 (5:37) portfolio_value=268.17  cash=213.92  positions=6  orders=2  (BTC entered, both TPs placed)
09:42:47 (5:42) portfolio_value=266.07  cash=213.92  positions=6  orders=1  (SL triggered, TP cancelled)
09:43:32 (5:43) portfolio_value=237.32  cash=213.92  positions=6  orders=0  (both SLs triggered, ~$30 unrealized loss)

---

---

Raw logs omitted (~2000 lines). See full file for detailed log output.