Goal: collect a large list of public pages (no login, no paywalls) on a daily schedule (and near-real-time when it makes sense) without tripping bot protections. Stealth first; I’ll also add a short path to become a recognized/allow-listed crawler if the business later wants it.
I would keep three constraints in front of me at all times: