<\/span><\/h2>\n\n\n\nDoes the target filter by network type?Do you need a location finer than a country?Is the workload stateful?What is your latency budget?How heavy is each page?<\/span>Measure it: a small benchmark<\/span><\/h2>\n\n\n\nMost providers sell a small starter package for each type. That is enough to run a real comparison against your real targets. The script below sends the same list of URLs through each proxy type and reports four numbers per type.
import statistics\nimport time\nfrom concurrent.futures import ThreadPoolExecutor\n\nimport requests\n\nTIERS = {\n \"datacenter\": \"http:\/\/USER:PASS@dc-gateway.example.com:8000\",\n \"residential\": \"http:\/\/USER:PASS@resi-gateway.example.com:9000\",\n}\nMUST_CONTAIN = \"product-price\" # a string every good page includes\nWORKERS = 10\n\nwith open(\"urls.txt\") as f:\n URLS = [line.strip() for line in f if line.strip()]\n\ndef fetch(url, proxy):\n start = time.perf_counter()\n try:\n r = requests.get(url, proxies={\"http\": proxy, \"https\": proxy}, timeout=30)\n ok = r.status_code == 200 and MUST_CONTAIN in r.text\n return ok, time.perf_counter() - start, len(r.content)\n except requests.RequestException:\n return False, time.perf_counter() - start, 0\n\nfor tier, proxy in TIERS.items():\n with ThreadPoolExecutor(WORKERS) as pool:\n results = list(pool.map(lambda url: fetch(url, proxy), URLS))\n good = [r for r in results if r[0]]\n if not good:\n print(f\"{tier:12} no valid pages out of {len(results)}\")\n continue\n times = sorted(r[1] for r in good)\n p95 = times[min(len(times) - 1, int(len(times) * 0.95))]\n mb = sum(r[2] for r in good) \/ 1e6\n print(\n f\"{tier:12} valid {len(good) \/ len(results):6.1%} \"\n f\"p50 {statistics.median(times):.2f}s p95 {p95:.2f}s \"\n f\"MB per valid page {mb \/ len(good):.3f}\"\n )<\/code><\/pre>\n\n\n\nA few choices in there are deliberate.<\/strong><\/p>\n\n\n\n\n“Valid” is not “200”. The check looks for a string that every good page contains, such as the CSS class of the price element. Some sites answer unwanted traffic with a normal status code and a page that has the important parts removed. Counting status codes alone would score that as a success.<\/li>\n\n\n\n Megabytes per valid page is the cost figure. Multiply it by your price per gigabyte and you have the cost of a thousand usable pages for each type, which is the number to compare. The script counts traffic from valid pages only, which matches providers that do not charge for failed or blocked attempts. Check how yours bills. If every attempt is metered, sum over results instead of good.<\/li>\n\n\n\n The byte count is an estimate. The script measures decoded response bodies. Your provider bills what crossed the wire, including headers and TLS overhead and before decompression. Use the script to compare the two types and the provider’s dashboard for the true total.<\/li>\n<\/ul>\n\n\n\n<\/span>How to run it so the result means something<\/span><\/h2>\n\n\n\nA benchmark is easy to get wrong in ways that flatter one side.<\/p>\n\n\n\n
\nUse your real URLs. A few hundred pages per target, mixed across the page types you will collect. A homepage is often less protected than a product or search page.<\/li>\n\n\n\n Run at your real concurrency. A target that accepts ten parallel datacenter connections may refuse a hundred. Set WORKERS to what production will use.<\/li>\n\n\n\n Repeat it at different hours and on different days. Residential pools change through the day as devices come and go, and target sites tighten or relax their rules over time. One run is an anecdote.<\/li>\n\n\n\n Read p95 before p50. Median latency on residential IPs often looks fine. The slow tail is what fills your worker pool and sets your timeout.<\/li>\n\n\n\n Keep the client identical. Same headers, same library, same delays for both types. Otherwise you are benchmarking your HTTP client, not the IPs.<\/li>\n<\/ul>\n\n\n\n<\/span>Datacenter vs residential proxies by workload<\/span><\/h2>\n\n\n\nWorkload<\/th> Usual pick<\/th> Reason<\/th><\/tr><\/thead> Polling public or partner APIs<\/td> Datacenter<\/td> Limits are per IP or per key, not per network type<\/td><\/tr> Uptime and performance checks from several countries<\/td> Datacenter<\/td> Needs steady latency, country is enough<\/td><\/tr> Crawling documentation, news, open data<\/td> Datacenter<\/td> Rarely filtered, high volume<\/td><\/tr> Prices on large marketplaces and travel sites<\/td> Residential<\/td> Server ranges are commonly challenged<\/td><\/tr> Search results for a given city<\/td> Residential<\/td> Needs city level location<\/td><\/tr> Ad verification and localisation QA<\/td> Residential<\/td> The point is to see what a real local user sees<\/td><\/tr> Long catalogue with a few strict targets<\/td> Both<\/td> Route each target to the cheapest type that passes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<\/p>\n\n\n\n
The last row is more common than either pure case. Nothing forces a single choice for the whole system. A per-target setting in your config, filled in from the benchmark, is often all the routing logic you need.
<\/span>Common mistakes<\/span><\/h2>\n\n\n\n\nTesting against a speed test page. It tells you how fast the proxy is to a site that welcomes everyone. Your target is not that site.<\/li>\n\n\n\n Deciding once and never checking again. Sites change their defences, sometimes to stricter and sometimes to looser. Rerun the benchmark every quarter, or you may keep paying residential rates for a target that no longer filters servers.<\/li>\n\n\n\n Committing to a large plan before the first test. Buy the smallest package of each, measure, then size the order from the megabytes-per-page figure.<\/li>\n\n\n\n Switching type mid-session. If a logged-in flow starts on one network and continues on another, the site sees an account that jumped from a home to a server rack. Pick one type per session.<\/li>\n\n\n\n Skipping the sourcing question. Residential addresses belong to real people. Ask the provider how those people agreed to share their connection, and walk away if the answer is vague.<\/li>\n<\/ul>\n\n\n\n<\/span>Wrapping up<\/span><\/h2>\n\n\n\n
In most teams the proxy question shows up as a one-line ticket: “Which proxies do we buy for the collector?” Then it gets answered the way database choices used to be answered, by whoever has the strongest opinion in the room. One person swears residential is the only thing that works. Another says it is […]<\/p>\n","protected":false},"author":1,"featured_media":2816,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_sitemap_exclude":false,"_sitemap_priority":"","_sitemap_frequency":"","footnotes":""},"categories":[2],"tags":[],"class_list":["post-2815","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-development"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/posts\/2815","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/comments?post=2815"}],"version-history":[{"count":2,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/posts\/2815\/revisions"}],"predecessor-version":[{"id":2818,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/posts\/2815\/revisions\/2818"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/media\/2816"}],"wp:attachment":[{"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/media?parent=2815"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/categories?post=2815"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/extendsclass.com\/blog\/wp-json\/wp\/v2\/tags?post=2815"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}