Scrapy proxy middleware for rotating residential IPs

Test a Ludax proxy with one spider

If the log shows a scraped item with a residential IP, your Scrapy proxy works. This is the quickest test; for a real project use the middleware.

1. Find your proxy URL parts

   On the dashboard’s Generate page: the username lx_k3m9q2p7xa, written resi-lx_k3m9q2p7xa for Residential, premi- for Residential Premium, and the plan password. Options such as the country go after the password (all options).

2. Install Scrapy

   install.shCOPY

   pip install scrapy

3. Run this spider

   whoami_spider.pyCOPY

   import scrapy
   
   PROXY = "http://resi-lx_k3m9q2p7xa:[email protected]:7777"
   
   
   class WhoamiSpider(scrapy.Spider):
       name = "whoami"
   
       async def start(self):
           # the proxy goes in meta; Scrapy sends the login for you
           yield scrapy.Request("https://api.ludaxproxy.com/whoami", meta={"proxy": PROXY})
   
       def parse(self, response):
           yield response.json()

   run.shCOPY

   scrapy runspider whoami_spider.py

   Among the log lines you should see the scraped item, with your own exit IP:

   output

   DEBUG: Scraped from <200 https://api.ludaxproxy.com/whoami>
   {'ip': '203.0.113.24', 'country': 'United States', 'type': 'residential', 'tls_profile': None, 'latency_ms': 412}

   That is a real home connection in the United States, picked by -country-us in the password. tls_profile is None because the site sees Scrapy’s own handshake (why that matters). Codes for every country are in the locations directory.

   Official docs: Scrapy: HttpProxyMiddleware

A Scrapy proxy middleware for Ludax

One file and five settings, and every request in the project goes through Ludax. Your spiders stay as they are.

1. A new IP on every request (LUDAX_STICKY_MINUTES = 0), or one IP for the whole spider (30), or one per account with meta["ludax_session"].
2. The country for every request in one setting.
3. Stops at the first refused login. Without it Scrapy retries a 407 and every queued request tries again, which is how a typo turns into a 10-minute IP block.
4. A proxy in meta still wins, for the odd request that needs another country.

import secrets

from scrapy.core.downloader.handlers.http11 import TunnelError
from scrapy.exceptions import IgnoreRequest
from scrapy.utils.defer import deferred_from_coro


class LudaxProxyMiddleware:
    """Sends every request through Ludax and stops the crawl on a refused login."""

    def __init__(self, crawler):
        s = crawler.settings
        self.crawler = crawler
        self.user = s["LUDAX_USER"]
        self.password = s["LUDAX_PASSWORD"]
        self.gateway = s.get("LUDAX_GATEWAY", "gw.ludaxproxy.com:7777")
        self.country = s.get("LUDAX_COUNTRY")
        self.sticky_minutes = s.getint("LUDAX_STICKY_MINUTES")  # 0: a new IP per request
        self.tls = "-AE-" in self.user or "-NAE-" in self.user  # a TLS port
        self.spider_token = secrets.token_hex(4)

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler)

    def process_request(self, request):
        if "proxy" in request.meta:
            return None  # the spider chose a proxy itself
        if self.sticky_minutes:
            # one IP for the whole spider; meta["ludax_session"] gives a flow its own IP
            token = request.meta.get("ludax_session", self.spider_token)
            options = f"-session-{token}-time-{self.sticky_minutes}"
            if self.tls:
                options += "-sameip"  # on the TLS ports, hold the IP as well as the browser
        else:
            # a token of its own: a new connection, so a new IP, for every request
            options = f"-session-{secrets.token_hex(4)}-time-1"
        if self.country:
            options += f"-country-{self.country}"
        request.meta["proxy"] = f"http://{self.user}:{self.password}{options}@{self.gateway}"
        return None

    def refused(self, detail):
        # every other request would be refused the same way, and retries only
        # count towards the block (30 failed logins in 5 minutes)
        deferred_from_coro(self.crawler.engine.close_spider_async(reason="proxy_login_refused"))
        raise IgnoreRequest(f"Ludax refused the login (407): {detail}")

    def process_response(self, request, response):
        if response.status == 407:  # http:// URLs get the gateway's answer as a response
            self.refused(request.url)
        return response

    def process_exception(self, request, exception):
        if isinstance(exception, TunnelError) and "'status': 407" in str(exception):
            self.refused(exception)  # https:// URLs get it as a tunnel error
        return None

import os

DOWNLOADER_MIDDLEWARES = {
    # before HttpProxyMiddleware (750), which turns meta["proxy"] into the login header
    "myproject.middlewares.LudaxProxyMiddleware": 610,
}

LUDAX_USER = "resi-lx_k3m9q2p7xa"
LUDAX_PASSWORD = os.environ["LUDAX_PASSWORD"]  # the plan password, kept out of the code
LUDAX_GATEWAY = "gw.ludaxproxy.com:7777"
LUDAX_COUNTRY = "us"  # None: any country
LUDAX_STICKY_MINUTES = 0  # 0: a new IP per request; 30: one IP for 30 minutes

export LUDAX_PASSWORD="YOUR_PASSWORD"
scrapy crawl myspider

Replace myproject with your project’s package name. Priority 610 places it before Scrapy’s HttpProxyMiddleware (750), which turns the proxy URL into the login header, and lets it see errors before the RetryMiddleware (550) does. Tested: with a wrong password and 20 queued requests the crawl closed with finish_reason: proxy_login_refused after 4 attempts, one per concurrent request.

Official docs: Scrapy: downloader middleware · Scrapy: settings

Scrapy proxy settings generator

Choose the product, the country and whether the IP should stay the same, and copy the settings for the middleware above, or a spider with the proxy in meta for a quick test. Type your own username and password to get code you can run as is: they stay in this browser tab and are never sent anywhere.

On the TLS gateways Scrapy needs no certificate setup: it does not check certificates by default. The middleware adds -sameip for sticky sessions. Why the TLS ports?

Spiders for sessions, geo and speed

Each spider runs with the middleware above, unless it says otherwise.

import scrapy


class AccountsSpider(scrapy.Spider):
    """Each account keeps its own IP: run with LUDAX_STICKY_MINUTES = 30."""

    name = "accounts"

    async def start(self):
        for account in ["shop1", "shop2", "shop3"]:
            # the same ludax_session = the same IP; letters and digits only
            for page in range(3):
                yield scrapy.Request(
                    f"https://api.ludaxproxy.com/whoami?account={account}&page={page}",
                    meta={"ludax_session": account},
                    cb_kwargs={"account": account},
                    dont_filter=True,
                )

    def parse(self, response, account):
        yield {"account": account, "ip": response.json()["ip"]}

{'account': 'shop1', 'ip': '203.0.113.24'}
{'account': 'shop1', 'ip': '203.0.113.24'}
{'account': 'shop2', 'ip': '198.51.100.7'}
...

import scrapy

USER = "resi-lx_k3m9q2p7xa"
PASSWORD = "YOUR_PASSWORD"
GATEWAY = "gw.ludaxproxy.com:7777"


class PricesSpider(scrapy.Spider):
    """The same page seen from three countries."""

    name = "prices"

    async def start(self):
        for country in ["us", "de", "jp"]:
            proxy = f"http://{USER}:{PASSWORD}-country-{country}@{GATEWAY}"
            yield scrapy.Request(
                "https://api.ludaxproxy.com/whoami",
                meta={"proxy": proxy},  # a proxy in meta wins over the middleware
                cb_kwargs={"country": country},
                dont_filter=True,
            )

    def parse(self, response, country):
        yield {"asked": country, "got": response.json()["country"]}

{'asked': 'us', 'got': 'United States'}
{'asked': 'de', 'got': 'Germany'}
{'asked': 'jp', 'got': 'Japan'}

# settings.py: fast, but gentle on the site and on your GB
CONCURRENT_REQUESTS = 16
CONCURRENT_REQUESTS_PER_DOMAIN = 8
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4
DOWNLOAD_TIMEOUT = 30

# a dropped exit (502) is retried by Scrapy's own RetryMiddleware: 502 is in its
# default RETRY_HTTP_CODES, and a failed tunnel is retried as well
RETRY_TIMES = 3

Scrapy log lines and what they mean

The exact line Scrapy logs, what it means and the one thing to do. All gateway answers: status codes.

Scrapy proxy questions

Short answers. The full option list is in proxy authentication.

Beyond Scrapy

The products behind the spiders, and guides for the jobs Scrapy runs.

• Residential proxiesReal home IPs in 137 countries, rotating or sticky.
• Residential TLS proxiesHome IPs plus 139 browser TLS fingerprints, made at the gateway.
• Python guiderequests, aiohttp and HTTPX with the same proxies.
• Web scraping with residential proxiesRotation, sessions and plans for scrapers at scale.
• Price monitoringLocal prices and stock, checked from the shopper's own country.
• AI data pipelinesPublic web data for training and retrieval, collected without blocks.