The Headless Browser Benchmark

For years, Selenium WebDriver was the unchallenged king of browser automation. However, when orchestrating 50 concurrent automated sessions on a Linux server to monitor dynamic financial quotes or extract Javascript-rendered catalog data, Selenium's architecture introduces severe bottlenecks:

  • WebDriver Protocol Overhead: Selenium sends JSON commands over an HTTP REST wrapper to a local driver binary (chromedriver), which in turn translates commands to the browser. This double-hop introduces latency.
  • Process Bloat: In Selenium, spinning up 20 isolated sessions requires launching 20 separate OS browser processes, rapidly consuming 12+ GB of system memory.
  • Playwright Contexts: Playwright communicates natively with Chromium via the Chrome DevTools Protocol (CDP) through persistent bi-directional WebSockets. More crucially, Playwright supports Browser Contexts—lightweight isolated incognito sessions within a single parent browser process, reducing memory overhead by up to 80%.

Playwright Context Pool Implementation in Python

import asyncio
from playwright.async_api import async_playwright

async def run_concurrent_scraping_pool(urls: list[str]):
    async with async_playwright() as p:
        # Launch single browser instance
        browser = await p.chromium.launch(headless=True, args=["--disable-gpu", "--no-sandbox"])
        
        # Concurrently process pages using lightweight isolated contexts
        async def scrape_url(url: str):
            context = await browser.new_context(
                user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36",
                viewport={"width": 1280, "height": 720}
            )
            page = await context.new_page()
            try:
                await page.goto(url, wait_until="networkidle", timeout=20000)
                title = await page.title()
                return {"url": url, "title": title, "status": "success"}
            except Exception as e:
                return {"url": url, "error": str(e), "status": "failed"}
            finally:
                await context.close()

        tasks = [scrape_url(u) for u in urls]
        results = await asyncio.gather(*tasks)
        await browser.close()
        return results