Headless Browser Detection: How Sites Know You're a Bot
The Detection Arms Race You launch your Puppeteer script, it works perfectly in testing, then fails in production. The site knows you are a bot. But how? Modern bot detection goes far beyond checki...
Source: DEV Community
The Detection Arms Race You launch your Puppeteer script, it works perfectly in testing, then fails in production. The site knows you are a bot. But how? Modern bot detection goes far beyond checking user agents. Let's dive into exactly how sites detect headless browsers and how to defend against each technique. Detection Method 1: The WebDriver Flag The simplest check. Every automated browser sets navigator.webdriver = true: // What sites check if (navigator.webdriver) { // Block this visitor } Defense in Python with Playwright: from playwright.sync_api import sync_playwright def create_stealth_browser(): p = sync_playwright().start() browser = p.chromium.launch( headless=True, args=["--disable-blink-features=AutomationControlled"] ) context = browser.new_context() # Remove webdriver flag context.add_init_script(""" Object.defineProperty(navigator, 'webdriver', { get: () => undefined }); """) return browser, context Detection Method 2: Chrome DevTools Protocol Sites detect if CDP (
