Skip to content

Polite Web Scraping: Rate-Limit Design Patterns, Whether You Need Proxies, and the AI-Agent-Friendly Open Source Stack

A research article on how to scrape a website that explicitly allows robots: the politeness/rate-limit design pattern, when (if ever) a proxy is needed, and which open-source GitHub frameworks are worth using — including the new AI-agent-friendly generation. Research date: Aug 4, 2026. All URLs verified by direct fetch.

TL;DR

Scraping a robots-allowed site politely is a well-specified design pattern, mostly built on IETF RFCs, not folklore:

  1. Read and honor robots.txt — it's now RFC 9309 (Sept 2022). Identify yourself in the User-Agent with a product token + contact URL. Note Google does not support crawl-delay; sites throttle you by serving 429/500/503 instead.
  2. Handle 429/Retry-After (RFC 6585 / RFC 9110): on a 429 or 5xx, back off — jittered exponential backoff is the standard (AWS's own simulation cut call count by >half). Retry only idempotent requests.
  3. Cache aggressively — conditional requests (If-None-Match/ETag, If-Modified-Since) avoid re-downloading unchanged pages (RFC 9110). Wikimedia's own etiquette literally says "take steps to cache it."
  4. Cap concurrency and rate — concrete published numbers: Wikimedia allows website crawling at <10 concurrent / <20 req/s average, and unauthenticated API at ≤3 concurrent / <5 req/s. "Serial = safe" is the default politeness baseline.

Do you need a proxy? For polite scraping of a robots-allowed site: no. Wikimedia and framework best practices show a descriptive User-Agent + rate limits + delays suffice, and some sites explicitly forbid rotating identities to hide load. Proxies only become necessary when: per-IP limits bind at high volume, content is geo-restricted, or the site sits behind anti-bot systems (Cloudflare, DataDome). If you might need one, use an escalating proxy tier — Crawlee's tieredProxyUrls starts at [null] (no proxy) and only moves up when blocked.

Open-source, agent-friendly stack on GitHub (stars Aug 4, 2026):

Project Stars License Agent angle
Firecrawl 160.8k AGPL-3.0 (SDKs MIT) self-host scrape API, LLM markdown/JSON, official MCP, "Agent ready"
Crawl4AI 76k Apache-2.0 LLM-ready markdown, self-host Docker + MCP, natural-language extraction
ScrapeGraphAI 29k MIT "extract X" via LLM, MCP
Crawlee (JS) 25.2k Apache-2.0 polite autoscaling, tiered proxies, "data for LLMs"
Colly (Go) 25.4k Apache-2.0 lightweight, per-domain delays
Scrapy (Python) 63.6k BSD-3 AutoThrottle + robots.txt middleware
Firecrawl MCP 7.1k MIT hosted keyless MCP server

For a project that "provides the service": Firecrawl (open source + hosted cloud) and Crawl4AI (Docker API server with JWT auth) are the two that run a real scrape API you can self-host and point an agent at. Scrapy/Crawlee are libraries, not services; Apify and Bright Data platforms are closed source (their SDKs/MCPs are OSS).

Part 1 — The design pattern: polite, rate-limited scraping

The whole discipline is "be a good HTTP citizen." Every piece maps to a standard:

1.1 robots.txt is a standard now (RFC 9309)

robots.txt became an IETF standard in Sept 2022 as RFC 9309, formalizing Koster's 1994 original (whose motivation was robots that "swamped servers with rapid-fire requests" — https://www.robotstxt.org/orig.html).

The RFC makes the etiquette explicit:

  • "These rules are not a form of access authorization" — crawlers are requested to honor them; the site enforces if it wants (https://www.rfc-editor.org/rfc/rfc9309.html).
  • Error handling: if robots.txt returns a 4xx (except 429), the crawler MAY access everything; if it's unreachable (5xx), the crawler MUST assume complete disallow (https://www.rfc-editor.org/rfc/rfc9309.html).
  • Cache robots.txt, but not forever: crawlers MAY cache it but SHOULD NOT reuse it beyond 24 hours unless unreachable (https://www.rfc-editor.org/rfc/rfc9309.html).
  • Identify yourself: the product token SHOULD be a substring of the User-Agent, and the identification string SHOULD describe the crawler's purpose, e.g., a contact URL (https://www.rfc-editor.org/rfc/rfc9309.html §2.2.1).

Google's framing matches: robots.txt "is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google" (https://developers.google.com/search/docs/crawling-indexing/robots/intro). Two implementation gotchas from Google's spec:

  • Google supports only user-agent, allow, disallow, sitemap — crawl-delay is NOT supported (https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec).
  • Google's 4xx/5xx handling mirrors the RFC: all 4xx except 429 → treat as no robots.txt; 5xx → stop and reuse last good copy (https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec).

So don't expect crawl-delay to save you — sites throttle crawlers by HTTP status codes (500/503/429), which Google documents explicitly: "your site's crawling rate" drops when the server returns "a significant number of URLs with 500, 503, or 429" (https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate).

1.2 The HTTP rate-limit pattern: 429 → Retry-After → jittered backoff

  • 429 Too Many Requests is defined in RFC 6585 §4: "the user has sent too many requests in a given amount of time ('rate limiting')"; it MAY carry Retry-After and MUST NOT be cached (https://www.rfc-editor.org/rfc/rfc6585.html).
  • Retry-After (RFC 9110 §10.2.3) tells the client how long to wait before the next attempt — either a delay in seconds or an HTTP-date. 503 can carry it too (https://www.rfc-editor.org/rfc/rfc9110.html). MDN documents 429+Retry-After as the canonical "slow down" handshake (https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429).
  • Back off with jitter. AWS's engineering blog shows jittered exponential backoff is the standard approach for remote clients and that un-jittered backoff is measurably worse — in their simulation jitter "reduced our call count by more than half" (https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/). Google Cloud's retry guidance agrees: retry 408/429/5xx and transient socket errors, use exponential backoff with jitter, and only retry idempotent requests (defaults ≈1s initial, ×2 multiplier, capped ~30–64s) (https://cloud.google.com/storage/docs/retry-strategy).

1.3 Cache to cut load in half

The cheapest "rate limit" is not making the request twice.

  • Conditional requests: If-None-Match (ETag validation) exists "to enable efficient updates of cached information with a minimum amount of transaction overhead" — the server replies 304 Not Modified when the stored tag matches (RFC 9110 §13.1.2, §15.4.5). If-Modified-Since avoids transferring data when nothing changed (RFC 9110 §13.1.3) (https://www.rfc-editor.org/rfc/rfc9110.html).
  • Wikimedia's etiquette for API clients is blunt: "If your requests obtain data that can be cached for a while, you should take steps to cache it, so you don't request the same data over and over again" (https://www.mediawiki.org/wiki/API:Etiquette).

1.4 Concrete numbers: what "polite" actually means

The best published, site-verified numbers come from Wikimedia's Robot policy (https://wikitech.wikimedia.org/wiki/Robot_policy):

Target Concurrency Rate
Website (/wiki/Article, no query params) < 10 concurrent avg < 20 req/s
REST API (unauthenticated) ≤ 3 < 5 req/s
Action API (unauthenticated) 1 < 5 req/s
Media (upload.wikimedia.org) ≤ 2 ≤ 25 Mbps
Other services (Gerrit, Phabricator, etc.) ≤ 1 ≥ 1 s delay between requests

Plus: honor every robots.txt directive, use gzip, respect Retry-After on 429, and pause ≥15 minutes on a 5xx. Wikimedia also prefers you use dumps or CDN-cached endpoints over live requests — read https://wikitech.wikimedia.org/wiki/Robot_policy.

Framework-level politeness defaults:

  • Scrapy AutoThrottle: "spiders always start with a download delay of AUTOTHROTTLE_START_DELAY" (default 5s, max 60s), target concurrency ~1.0 in-flight per domain, and latencies of non-200 responses "are not allowed to decrease the delay." Lowering target concurrency (e.g., 0.5) makes the crawler "more conservative and polite" (https://docs.scrapy.org/en/latest/topics/autothrottle.html).
  • MediaWiki's baseline: making requests "in series rather than in parallel... should result in a safe request rate" (https://www.mediawiki.org/wiki/API:Etiquette).
  • Reddit's (archived) API was a documented numeric example: 60 requests/min via OAuth, with a descriptive User-Agent and a rule to "NEVER lie about your user-agent" (https://github.com/reddit-archive/reddit/wiki/API).

The resulting pattern (in pseudocode):

identify via User-Agent (product + contact URL)
parse robots.txt (RFC 9309 semantics) before each host
schedule with a per-host politeness delay (start ~1s, tune up)
bound concurrency (start 1)
on 200: cache; store ETag/Last-Modified; revalidate with If-None-Match next time
on 429: sleep Retry-After, else jittered exp backoff
on 5xx: backoff; pause long on repeated failures
never retry non-idempotent requests

Part 2 — Is a proxy needed?

2.1 No — for polite, low-rate scraping of robots-allowed sites

The proxy is a tool for defeating IP-based blocking — Crawlee's docs call IP blocking "one of the oldest and most effective ways of preventing access to a website" and the proxy "the most powerful weapon in our anti IP blocking arsenal" (https://crawlee.dev/docs/guides/proxy-management). If you're not being blocked, there is nothing to defeat.

The evidence that compliant scrapers don't need proxies:

  • Wikimedia enforces limits by identity, not by IP. Its User-Agent policy requires an informative UA with contact info; compliance is judged on the header + rate, so a single IP is fine (https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy). Its robot policy asks high-volume operators to publish their own CIDRs or authenticate — not to rotate exit IPs (https://wikitech.wikimedia.org/wiki/Robot_policy).
  • Rotation to hide load is against policy, not a "polite technique." Wikimedia's API usage guidelines prohibit "spread[ing] Wikimedia API requests over multiple user agents to hide excessive use by a single operator" (https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_API_Usage_Guidelines).
  • Framework best practice puts politeness before proxies. Scrapy's "Avoiding getting banned" section leads with rotating User-Agents, disabling cookies, and download delays (≥2s), suggests Common Crawl to avoid hitting sites directly, and lists "a pool of rotating IPs" only among later options (https://docs.scrapy.org/en/latest/topics/practices.html).
  • Crawlee's default posture is "no proxy until blocked." Its tiered-proxy feature starts with [null] and only switches to a proxy "if Crawlee recognizes we're getting blocked by the target website," then downgrades back when unblocked (https://crawlee.dev/docs/guides/proxy-management).

2.2 When a proxy DOES become necessary

Three concrete triggers (proxy providers' docs are the clearest sources):

  1. Anti-bot systems. Cloudflare's bot docs state bots "can scrape content," and Cloudflare products "detect this automated traffic and let you decide how to respond" — challenge or block (https://developers.cloudflare.com/bots/). DataDome installs its detection at the CDN/edge and tracks bot traffic in real time (https://docs.datadome.co/docs/getting-started). Once a site runs these, IP diversity matters.
  2. Geo-restricted content. "Residential proxies are, in fact, the way most scrapers bypass geo-restrictions," while datacenter proxies "have a limited range of locations and are more easily detected" (https://www.scraperapi.com/blog/how-to-scrape-geo-restricted-data/). Residential networks route through real end-user IPs so "target sites see your requests as genuine local users" (https://docs.brightdata.com/proxy-networks/residential/introduction).
  3. Per-IP rate limits at scale. "As your requests increase, your target site will block your machine's IP address" (https://www.scraperapi.com/blog/curl-with-proxy/).

2.3 If you need one: how the frameworks do it

  • Scrapy ships HttpProxyMiddleware — proxies from http_proxy/https_proxy env vars or a per-request proxy meta key, including user:pass@host:port and SOCKS (https://docs.scrapy.org/en/latest/topics/downloader-middleware.html). Scrapy docs also name Tor, ProxyMesh (paid), and scrapoxy (open source) for rotation (https://docs.scrapy.org/en/latest/topics/practices.html).
  • Crawlee is the strongest: ProxyConfiguration rotates a static list round-robin, supports a per-request custom proxy function, session-sticky proxies, and tiered escalation ([null] → cheap proxy → expensive proxy) with automatic downgrade (https://crawlee.dev/docs/guides/proxy-management).
  • Avoid free public proxies — "easy for websites to detect, throttle, or block," fine only "for testing and low-volume scraping" (https://www.scraperapi.com/blog/best-10-free-proxies-and-free-proxy-lists-for-web-scraping/).

Part 3 — The open-source landscape on GitHub

3.1 Traditional frameworks (libraries, not services)

Project Stars License Why it matters
Scrapy 63.6k — https://github.com/scrapy/scrapy BSD-3-Clause AutoThrottle (dynamic politeness), built-in RobotsTxtMiddleware (https://docs.scrapy.org/en/latest/topics/autothrottle.html, https://docs.scrapy.org/en/latest/topics/downloader-middleware.html), the canonical Python framework
Crawlee 25.2k — https://github.com/apify/crawlee Apache-2.0 Ex-Apify SDK. Autoscaling, integrated proxy rotation, Cheerio/JSDOM/Playwright/Puppeteer crawlers, README markets "extract data for AI, LLMs, RAG, or GPTs" (https://github.com/apify/crawlee). Apify's hosted platform is closed; Crawlee is the OSS core. Also has a Python port https://github.com/apify/crawlee-python
Colly 25.4k — https://github.com/gocolly/colly Apache-2.0 Lightweight Go framework: "manages request delays and maximum concurrency per domain," robots.txt support, caching — a great low-overhead polite crawler
ScrapyRT 883 — https://github.com/scrapinghub/scrapyrt BSD-3-Clause HTTP API wrapper: "You send a request... with spider name and URL... you get items collected by a spider" — turns Scrapy into a service

3.2 AI-agent-friendly scrapers (the new generation)

Project Stars License Agent angle
Firecrawl 160.8k — https://github.com/firecrawl/firecrawl AGPL-3.0 (SDKs/UI MIT) Self-hostable scrape API (SELF_HOST.md, docker-compose.yaml) that returns clean Markdown or LLM-structured JSON; "By default, Firecrawl respects robots.txt directives"; "Agent ready: connect to any AI agent or MCP client with a single command"
Crawl4AI 76k — https://github.com/unclecode/crawl4ai Apache-2.0 "Crawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines." Clean/Fit markdown (BM25/pruning filters), LLM + CSS/XPath extraction, self-host Docker API server on port 11235 with JWT auth and MCP integration "for direct connection to AI tools like Claude Code" (https://github.com/unclecode/crawl4ai)
ScrapeGraphAI 29k — https://github.com/ScrapeGraphAI/Scrapegraph-ai MIT "web scraping python library that uses LLM and direct graph logic... Just say which information you want to extract." MIT-licensed self-host with your own LLM; managed cloud API separately; ships an MCP server via Smithery (https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp)

Firecrawl and Crawl4AI are the two that "provide the service": both run a real HTTP scrape API you can self-host. (Firecrawl's core is AGPL-3.0 — the copyleft outlier in this list; its SDKs and MCP server are MIT. Crawl4AI and ScrapeGraphAI are permissively licensed.)

3.3 MCP servers — the agent plumbing

  • Firecrawl MCP — 7.1k★, MIT, hosted keyless endpoint https://mcp.firecrawl.dev/v2/mcp plus self-host via FIRECRAWL_API_URL; "automatic retries and rate limiting" (https://github.com/firecrawl/firecrawl-mcp-server).
  • Bright Data MCP — 2.6k★, MIT, "an all-in-one solution for public web access" (https://github.com/brightdata/brightdata-mcp) — OSS wrapper over a paid proxy/data platform.
  • ScrapeGraphAI MCP — via Smithery (https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp).
  • DataForSEO — closed service, but official OSS clients (MIT) and an Apache-2.0 MCP server (https://github.com/orgs/dataforseo/repositories, https://github.com/dataforseo/mcp-server-typescript).

3.4 HTML → LLM-ready Markdown helpers (pair with any crawler)

Project Stars License Notes
Jina Reader 11.8k — https://github.com/jina-ai/reader Apache-2.0 Open-source core of r.jina.ai (URL→Markdown) and s.jina.ai (search→Markdown); self-host via Docker ghcr.io/jina-ai/reader:oss
Mozilla Readability 11.4k — https://github.com/mozilla/readability Apache-2.0 Main-content extraction used by Firefox Reader View; basis of many scrapers
Trafilatura 6.4k — https://github.com/adbar/trafilatura Apache-2.0 HTML→TXT/MD/JSON; "efficient and polite processing of download queues"

Part 4 — Which stack, when? (decision guide)

Polite scraping of one robots-allowed site, low volume (the case in the question): No proxy. A single-threaded client with a descriptive User-Agent, robots.txt honoring, a 1s+ per-host delay, caching with ETag revalidation, and 429/Retry-After + jittered backoff is the entire design. In practice: requests + a robots parser, or Scrapy with AutoThrottle + RobotsTxtMiddleware, or Crawlee with defaults. This is ~20 lines of logic you can also get for free from any framework's politeness defaults.

Whole-site crawl at moderate volume: Scrapy or Crawlee, both of which give you rate/scheduling for free. Crawlee's autoscaling + session pool is the most "reactive" — it scales up politely and (via tiered proxies) only escalates to a proxy when the site actually blocks you.

An agent / LLM needs to extract data (structured JSON, clean markdown) from many sites: The agent-friendly layer wins. Self-host Firecrawl (LLM extraction + official MCP + skill for Claude Code/OpenCode, AGPL caveat) or Crawl4AI (Apache-2.0, Docker API + MCP). For "just tell me what to extract," ScrapeGraphAI. All three self-host, so you keep control of robots.txt respect and rate limits on your own infrastructure rather than a hosted scraper's.

The site is protected (Cloudflare/DataDome), geo-restricted, or needs millions of pages: Now proxies matter. Use Crawlee's tiered proxy configuration (escalate only when blocked), and only then consider residential proxies from Bright Data/Oxylabs/ScraperAPI — or pay a hosted service (Apify, Firecrawl Cloud) that manages rotation for you. If the site doesn't want robots, this is no longer "polite scraping," it's anti-bot evasion — a different (riskier) game entirely.

Sources

  • RFC 9309 (robots.txt): https://www.rfc-editor.org/rfc/rfc9309.html
  • robots.txt origin: https://www.robotstxt.org/orig.html
  • Google robots.txt intro: https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • Google robots.txt spec (no crawl-delay): https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec
  • Google reduce crawl rate: https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate
  • RFC 6585 (429): https://www.rfc-editor.org/rfc/rfc6585.html
  • RFC 9110 (Retry-After, ETag, 304): https://www.rfc-editor.org/rfc/rfc9110.html
  • MDN 429: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
  • AWS jittered backoff: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
  • Google Cloud retry strategy: https://cloud.google.com/storage/docs/retry-strategy
  • Wikimedia Robot policy (limits): https://wikitech.wikimedia.org/wiki/Robot_policy
  • Wikimedia API etiquette (cache): https://www.mediawiki.org/wiki/API:Etiquette
  • Wikimedia User-Agent policy: https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_User-Agent_Policy
  • Wikimedia API usage guidelines: https://foundation.wikimedia.org/wiki/Policy:Wikimedia_Foundation_API_Usage_Guidelines
  • Scrapy AutoThrottle: https://docs.scrapy.org/en/latest/topics/autothrottle.html
  • Scrapy avoiding being banned: https://docs.scrapy.org/en/latest/topics/practices.html
  • Scrapy downloader middleware (RobotsTxtMiddleware, HttpProxyMiddleware): https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
  • Reddit archived API: https://github.com/reddit-archive/reddit/wiki/API
  • Crawlee proxy management (tiered): https://crawlee.dev/docs/guides/proxy-management
  • Cloudflare bots: https://developers.cloudflare.com/bots/
  • DataDome getting started: https://docs.datadome.co/docs/getting-started
  • ScraperAPI proxies & geo: https://www.scraperapi.com/blog/curl-with-proxy/ | https://www.scraperapi.com/blog/how-to-scrape-geo-restricted-data/ | https://www.scraperapi.com/blog/best-10-free-proxies-and-free-proxy-lists-for-web-scraping/
  • Bright Data proxy networks: https://docs.brightdata.com/proxy-networks | https://docs.brightdata.com/proxy-networks/residential/introduction
  • Oxylabs proxies quick start: https://developers.oxylabs.io/get-started/quick-start-proxies.md
  • Scrapy: https://github.com/scrapy/scrapy | Crawlee: https://github.com/apify/crawlee | Crawlee Python: https://github.com/apify/crawlee-python
  • Firecrawl: https://github.com/firecrawl/firecrawl | Firecrawl MCP: https://github.com/firecrawl/firecrawl-mcp-server
  • Crawl4AI: https://github.com/unclecode/crawl4ai
  • ScrapeGraphAI: https://github.com/ScrapeGraphAI/Scrapegraph-ai | MCP: https://smithery.ai/server/@ScrapeGraphAI/scrapegraph-mcp
  • Colly: https://github.com/gocolly/colly
  • ScrapyRT: https://github.com/scrapinghub/scrapyrt
  • Jina Reader: https://github.com/jina-ai/reader | Trafilatura: https://github.com/adbar/trafilatura | Mozilla Readability: https://github.com/mozilla/readability
  • Bright Data MCP: https://github.com/brightdata/brightdata-mcp
  • DataForSEO repos: https://github.com/orgs/dataforseo/repositories