The Calculation
addresses = requests per hour to one site ÷ polite requests per hour per address
+ 1 or 2 sparesWork it out for the busiest site you visit, not for all sites together: an address that sends one request a minute to each of ten different sites is quiet on every one of them. Two figures go in:
- Requests per hour to one site. Pages to fetch, multiplied by how often, divided by the hours you allow for the run. Fetching 2,000 pages once a day over a six-hour window is about 340 an hour.
- A polite rate per address. There is no universal number. A site's robots.txt and terms may state one; otherwise begin with one request at a time and a pause of several seconds between them, and slower still for search engines.
Starting Points by Use
| Use | What decides it | Typical start |
|---|---|---|
| Rank tracking | Keywords × checks per day, at a slow pace per search engine | 3–10 |
| Price monitoring | Products × checks per day on the largest retailer | 2–10 |
| Ad verification | Placements and links checked per hour | 2–5 |
| Social media management | One per account you manage | 1 per account |
| QA and geo testing | Test workers running at the same time | 1–5 |
| Allowlisting | One address to put on the list | 1 |
These are starting points to measure from, not limits. Your own error rates are the real guide.
Worked Example: Rank Tracking
You check 500 keywords once a day. Each check is one search results page, so 500 requests a day to one search engine. You allow eight hours for the run: about 63 an hour.
At a cautious pace of one search a minute per address, each address handles 60 an hour. 63 ÷ 60 rounds up to 2 addresses, plus one spare: 3. Checking twice a day, or doing it in four hours instead of eight, doubles that to about 5.
Worked Example: Price Monitoring
You track 2,000 products on one retailer, once a day. If the run can take six hours, that is about 340 requests an hour. One address sending one request every ten seconds manages 360 an hour, so one address and a spare is enough.
If you need the whole run finished in one hour, it is 2,000 an hour: 2,000 ÷ 360 rounds up to 6, plus a spare, 7. Most people find that a longer window is cheaper than more addresses, and kinder to the site.
When the Numbers Say Slow Down
A rise in 429 Too Many Requests, 503 or sudden 403 responses means you are going faster than the site will serve. The fix is to slow down and honor any Retry-After header, not to add addresses so the same volume keeps arriving. Using more addresses to push past a site's limits is the quickest way to lose access to the data altogether, and the complaints it draws can get a service restricted under our Acceptable Use Policy.
How to fix proxy errors explains which errors come from the proxy and which from the site.