1. Look for an API First
Many sites offer an API, a data feed, a sitemap or a bulk export. It is faster, it does not break when the page layout changes, and it is the clearest sign you are collecting the data the way the site wants. Check the footer, a /developers page, or /sitemap.xml.
2. Read robots.txt and the Terms
curl -s https://example.com/robots.txt User-agent: * Crawl-delay: 10 Disallow: /cart/ Disallow: /account/
Disallow lines are paths crawlers should not fetch; Crawl-delay, where present, is the number of seconds to wait between requests. Then read the site's terms of use for anything about automated access, scraping or reuse of content. If the terms forbid collecting what you want, the site can take action against you for breaking them, so take advice before going ahead. Getting past a login or paywall to collect it is a step further, and our Acceptable Use Policy prohibits it.
3. Identify Your Crawler
Send a User-Agent that names your crawler and gives a way to reach you, such as ExampleCoPriceBot/1.0 (+https://example.co/bot). A site owner who can see who you are can ask you to slow down instead of blocking you outright.
4. Set a Pace and Back Off
One request at a time per site, with a pause between them: the Crawl-delay if the site sets one, otherwise several seconds. When the site answers 429 Too Many Requests or 503, wait for the time in Retry-After, or longer each time if there is none, and if the errors continue, stop and look at why.
import time from urllib.parse import urljoin from urllib.robotparser import RobotFileParser import requests BASE = "https://example.com" AGENT = "ExampleCoPriceBot/1.0 (+https://example.co/bot)" PROXY = "http://USER:[email protected]:PORT" session = requests.Session() session.headers["User-Agent"] = AGENT session.proxies = {"http": PROXY, "https": PROXY} robots = RobotFileParser() robots.parse(session.get(urljoin(BASE, "/robots.txt"), timeout=30).text.splitlines()) delay = robots.crawl_delay(AGENT) or 5 # seconds between requests def fetch(path): url = urljoin(BASE, path) if not robots.can_fetch(AGENT, url): return None # disallowed: skip it for attempt in range(5): r = session.get(url, timeout=30) if r.status_code in (429, 503): wait = r.headers.get("Retry-After", "") time.sleep(int(wait) if wait.isdigit() else 60 * (attempt + 1)) continue time.sleep(delay) return r return None # still refused: stop and look
5. Cache and Ask Only for Changes
Keep what you have fetched and do not fetch it again until it might have changed. Many sites support conditional requests: send back the ETag they gave you, and an unchanged page costs them a short 304 Not Modified instead of the whole page.
r = session.get(url, headers={"If-None-Match": saved_etag}, timeout=30)
if r.status_code == 304:
pass # unchanged: use your copy
else:
saved_etag = r.headers.get("ETag")6. Collect Only What You Need
Fetch the pages that hold your data and skip images, scripts and everything else. Avoid personal data: names, emails and profiles bring privacy-law duties with them, and collecting them unlawfully is prohibited on our services. Never collect pages behind a login, paywall or other access control you are not authorized to pass.
Where Proxies Fit
Proxies let you spread a fixed amount of work across a few addresses, so each address stays at a gentle pace. How many proxies do I need? shows the arithmetic. They are a poor way to keep going after a site has refused you: adding addresses to push past 429s draws complaints that can get a service restricted, and carrying on after a site has told you in writing to stop is prohibited by our Acceptable Use Policy. For the code to use a proxy in other libraries, see using a proxy in Python, Playwright, Puppeteer and Selenium.