Introduction

HTTPX — An Overview

HTTPX is a next-generation HTTP client for Python with an API that is broadly compatible with the popular requests library, plus support for asynchronous requests. It allows for synchronous and asynchronous HTTP requests, making it versatile for both blocking and non-blocking operations. HTTPX supports HTTP/1.1 and HTTP/2, and includes features like connection pooling, cookie management, and proxy support. It's ideal for modern web scraping and API interaction tasks, offering advanced functionality and performance. For more information, visit HTTPX documentation.

Prerequisites

Install Python from the official website.

Check if Python is installed correctly, you can use the following command in your terminal or command prompt

python --version

The output should be in the following format:

Python 3.11.2

If you receive an error or see a version number starting with 2.x, you need to download Python 3.x and follow the installation instructions to set it up

Next, create a new Python project and install the HTTPX package

pip install -U httpx

The -U matters if HTTPX is already on your machine: the code on this page passes the proxy with the proxy argument, and HTTPX 0.28 removed the older proxies argument that many older tutorials use. To connect through the SOCKS5 port as well, add the SOCKS extra, which installs the socksio package:

pip install -U "httpx[socks]"

After installing the HTTPX library, let's make a request to retrieve our IP address. This can be achieved by sending a GET request to a service that returns your public IP. Here’s how you can do it:

import asyncio
import httpx

async def main():
    url = 'https://httpbin.org/ip'

    async with httpx.AsyncClient() as client:
        req = await client.get(url)
        if req.status_code == 200:
            try:
                res = req.json()
                print(res)
            except Exception as e:
                print(f"Failed to decode JSON: {e}")
        else:
            print(f"Request failed with status code: {req.status_code}")

asyncio.run(main())

After running the script, a GET request is sent to https://httpbin.org/ip, retrieving your public IP address from the response. The IP address is then displayed in JSON format.

After setting up HTTPX, let's configure it to integrate with ProxyPanel proxies for secure and anonymous web scraping.

Configuring the HTTPX Package with ProxyPanel

To integrate the request with our proxy, we need to first get the proxy configuration from the ProxyPanel dashboard. This includes obtaining the proxy server address, port, and any necessary authentication details

Open My Proxies in your dashboard. A proxy that shows Pending hasn't been started yet; press Start proxy before you use it.

In the proxy's Connection column, the "HTTP proxy" entry gives the host and the HTTP port, and the "SOCKS5 proxy" entry gives the same host with the SOCKS5 port. The Credentials column holds the username (Copy username) and the password, which stays masked until you press the eye button (Show / hide); Copy password copies it. Your account password isn't needed.

Save the values as environment variables in the terminal that will run your script, so they never end up in the code:

# macOS / Linux
export PROXY_HOST="HOST"
export PROXY_HTTP_PORT="HTTP_PORT"
export PROXY_SOCKS5_PORT="SOCKS5_PORT"   # only needed for SOCKS5
export PROXY_USERNAME="USERNAME"
export PROXY_PASSWORD="PASSWORD"

# Windows (PowerShell)
$env:PROXY_HOST = "HOST"
$env:PROXY_HTTP_PORT = "HTTP_PORT"
$env:PROXY_SOCKS5_PORT = "SOCKS5_PORT"
$env:PROXY_USERNAME = "USERNAME"
$env:PROXY_PASSWORD = "PASSWORD"

Once we have this information, we can pass a proxy to the AsyncClient with its proxy argument, as shown in the code.

import asyncio
import os

import httpx

async def main():
    url = 'https://httpbin.org/ip'
    # Host and HTTP port from the "HTTP proxy" entry, login from Credentials
    proxy = httpx.Proxy(
        url=f"http://{os.environ['PROXY_HOST']}:{os.environ['PROXY_HTTP_PORT']}",
        auth=(os.environ['PROXY_USERNAME'], os.environ['PROXY_PASSWORD']),
    )

    async with httpx.AsyncClient(proxy=proxy, timeout=30.0) as client:
        req = await client.get(url)
        if req.status_code == 200:
            try:
                res = req.json()
                print(res)
            except Exception as e:
                print(f"Failed to decode JSON: {e}")
        else:
            print(f"Request failed with status code: {req.status_code}")

asyncio.run(main())

Nothing in the script needs replacing; it reads the host, port, username and password from the environment. Because the login goes in auth rather than inside the URL, special characters in the password need no encoding.

You will notice that the IP address with proxy configuration differs from the one without a proxy. Now you have successfully integrated the HTTPX package with ProxyPanel.

Prefer the SOCKS5 port? With the SOCKS extra installed, swap the url line for the one below; auth stays as it is.

        url=f"socks5h://{os.environ['PROXY_HOST']}:{os.environ['PROXY_SOCKS5_PORT']}",