Skip to content

Instantly share code, notes, and snippets.

@jfarcand
Created February 27, 2026 19:22
Show Gist options
  • Select an option

  • Save jfarcand/877aa80eeb1d1111e06389b648b7db85 to your computer and use it in GitHub Desktop.

Select an option

Save jfarcand/877aa80eeb1d1111e06389b648b7db85 to your computer and use it in GitHub Desktop.
Plan: Improve Component Calibration & Chevron Resilience

Plan: Improve Component Calibration & Chevron Resilience

Context

Testing component detection on the Santé (Health) app reveals three problems:

  1. Calibration is misleading — it shows per-row matches but not the post-absorption result. ChefFamille sees value rows ("Bouger", "893 cal") classified separately when in reality BFS absorbs them into the summary-card above. The calibration needs to show what BFS actually sees.
  2. Calibration is viewport-only — only analyzes visible elements. ChefFamille wants full-page coverage via scrolling.
  3. Distance card misses chevron — OCR inconsistently detects ">" next to "11:33" for one card while 3/4 similar cards get it. The hard row_has_chevron: true constraint causes hard failure.

SAM (Segment Anything Model) was considered for bounding-box detection but rejected — too heavy (50-500MB). The existing absorption mechanism (absorbs_below_within_pt: 80) already handles bounding-box suppression correctly; the calibration tool just doesn't show it.


Changes

Phase 1: Extract ComponentScoring (prerequisite)

ComponentDetector.swift is at 538 lines (over 500 limit). Extract scoring into its own file.

File Action
Sources/mirroir-mcp/ComponentScoring.swift New — enum namespace with bestMatch() and scoreMatch() extracted from ComponentDetector
Sources/mirroir-mcp/ComponentDetector.swift Remove bestMatch() and scoreMatch(), call ComponentScoring.bestMatch() instead (~460 lines after)
Sources/mirroir-mcp/ComponentTester.swift Update calls: ComponentDetector.bestMatch()ComponentScoring.bestMatch()
Tests Update any test references to the moved methods

Phase 2a: Detection Pipeline View in Calibration

Add a === Detection Result (after absorption) === section to calibration output showing the actual component grouping BFS would see.

File Action
Sources/mirroir-mcp/ComponentTester.swift Add formatDetectionView() static method. Call ComponentDetector.detect() + applyAbsorption(), format each resulting ScreenComponent (kind, elements, tap target, Y range). Append to diagnose() output before the return.

Output format per component:

  summary-card [y=278-356] → 5 elements (3 absorbed)
    "O Activité", ">", "12:07", "Bouger", "893 cal"
    tapTarget: "O Activité" (clickable, navigates)

Phase 2b: Chevron Soft Constraint (parallel with 2a)

Add chevron_mode field to component definitions: required (hard fail), forbidden (hard fail), preferred (score bonus, no hard fail).

File Action
Sources/mirroir-mcp/ComponentSkillParser.swift Add ChevronMode enum (required/forbidden/preferred). Add let chevronMode: ChevronMode? to ComponentMatchRules. Parse chevron_mode: key from .md files. Backward compat: row_has_chevron: true.required, false.forbidden, absent → nil.
Sources/mirroir-mcp/ComponentScoring.swift In scoreMatch(): when chevronMode is .preferred, don't hard-fail on missing chevron — give +3.0 bonus when present, +0.0 when absent. Legacy rowHasChevron path unchanged when chevronMode is nil.
Sources/mirroir-mcp/ComponentTester.swift In explainMismatch(): for .preferred, don't report chevron absence as a mismatch — note it as "chevron: preferred but absent (score penalty only)".
mirroir-skills/components/ios/summary-card.md Change row_has_chevron: truechevron_mode: preferred
Tests ComponentScoringTests for all 3 modes. ComponentSkillParserTests for parsing chevron_mode.

Phase 3: Scroll-Through Calibration

Add scroll: true option to calibrate_component tool for full-page coverage.

File Action
Sources/mirroir-mcp/CalibrationScroller.swift New — enum namespace. collectFullPage(describer:input:bridge:maxScrolls:) -> ScrollResult. Swipes up, OCR each viewport, deduplicates elements by text (keeps latest coordinates), detects scroll exhaustion.
Sources/mirroir-mcp/ComponentTools.swift Add scroll boolean parameter to tool schema. When true, call CalibrationScroller.collectFullPage() instead of single describer.describe(). Pass aggregated elements to ComponentTester.diagnose().
Tests CalibrationScrollerTests — test element deduplication logic with synthetic data.

Phase 4: Real-Device Validation (Tier 4)

  1. swift build && swift test
  2. Launch Santé on iPhone
  3. calibrate_component(component_path: "summary-card.md") — verify detection view shows absorption of value rows
  4. calibrate_component(component_path: "summary-card.md", scroll: true) — verify full-page elements appear
  5. Verify Distance card matches as summary-card with chevron_mode: preferred

Files Summary

File Lines (after) Change
ComponentScoring.swift ~100 New: extracted scoring
ComponentDetector.swift ~460 Reduced: scoring extracted
ComponentTester.swift ~370 Added: detection view + chevron mode mismatch
ComponentSkillParser.swift ~350 Added: ChevronMode enum + parsing
CalibrationScroller.swift ~90 New: scroll-based element collection
ComponentTools.swift ~140 Added: scroll parameter
summary-card.md ~42 Changed: chevron_mode: preferred

All files under 500-line limit.


Answers to ChefFamille's Questions

  1. Distance card chevron miss — OCR inconsistency (Vision framework sometimes misses small ">" glyphs). Fixed via chevron_mode: preferred soft constraint.
  2. Bounding box exclusion — Existing absorption already does this (80pt, condition: any). Calibration just wasn't showing it. Phase 2a fixes visibility.
  3. Profile icon is tappable — Correct, no change needed.
  4. "Modifier" is tappable — Correct, no change needed.
  5. Status bar in nav_bar zone — OCR returns text ("91", "13:47") which falls in the nav_bar zone (top 12%). This is OCR-based. Correct behavior — decoration elements have no tap target.
  6. Calibration should scroll full page — Phase 3 adds scroll: true parameter.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment