Selenium — An Overview

Selenium is a powerful tool for automating web browsers, widely used for web application testing and web scraping. It supports various browsers (Chrome, Firefox, Safari, etc.) and allows you to write test scripts in Python, among other languages. The core component, Selenium WebDriver, interacts with web elements to simulate user actions like clicking, typing, and navigating. Selenium also includes features like Selenium Grid for parallel test execution across different environments, making it a versatile and essential tool for developers and testers.

Prerequisites

Install Python from the official website.

python --version

If you receive an error or see a version number starting with 2.x, you need to download Python 3.x and follow the installation instructions to set it up

Next, create a new Python project and install the Selenium-Python package

pip install selenium

After installing Selenium, you also need to install a web driver. The web driver is a crucial component that allows Selenium to interact with a web browser. It acts as a bridge between Selenium and the browser, enabling automated control and interaction with web pages.

To simplify managing and installing web drivers, use the webdriver-manager package. Install it with the following command:

pip install webdriver-manager

Let's Start Scraping !

Now, we're ready to scrape http://httpbin.io/ip to grab the IP address from it using Selenium and ProxyPanel for routing traffic through proxies to maintain anonymity and bypass restrictions.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

# define custom options for the Selenium driver
options = Options()

driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    options=options
)
driver.get('http://httpbin.io/ip')

print(driver.find_element(By.TAG_NAME, "body").text)
										

The output should be in the following format:

However, we haven't used a ProxyPanel proxy in this script. Next, we'll configure the proxy to route our traffic through ProxyPanel for enhanced anonymity

Selenium Wire

First, we need to install selenium-wire, which allows us to configure our script to use proxies with authentication. Selenium Wire extends Selenium's functionality, making it easier to handle complex scenarios such as working with authenticated proxies. This is particularly useful when you need to ensure that your web scraping activities are routed through a secure and reliable proxy service like ProxyPanel.

To install selenium-wire, run the following command:

pip install selenium-wire

Retrieve Proxy Information from Your Dashboard

Go to your Dashboard panel and navigate to the "My Proxy" section to view your IP information.

Click on the "Show Password" button and enter your account password to display your proxy password.

Set Up Proxy Configuration in Your Script

Update your Selenium setup with the proxy information you retrieved from your dashboard. You'll need to integrate the proxy details, including the IP address, port, and password, into your script to ensure that all traffic is routed through the proxy server.

Refer to the following script as a guide:

											from seleniumwire import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By

# Proxy configuration for Selenium Wire
proxy_username = 'john.MyProxy-1'
proxy_password = 'proxypanel!123'
proxy_address = '154.128.31.247:8083'

seleniumwire_options = {
    'proxy': {
        'http': f'http://{proxy_username}:{proxy_password}@{proxy_address}',
        'https': f'https://{proxy_username}:{proxy_password}@{proxy_address}',
        'verify_ssl': False,
    },
}

# Initialize ChromeDriver with Selenium Wire and proxy options
driver = webdriver.Chrome(
    service=ChromeService(ChromeDriverManager().install()),
    seleniumwire_options=seleniumwire_options
)

# Open a test page to verify the proxy setup
driver.get('http://httpbin.io/ip')
print(driver.find_element(By.TAG_NAME, 'body').text)

# Close the browser
driver.quit()
										

Make sure to replace the placeholders with your actual proxy details to complete the setup. This will ensure that Selenium uses the specified proxy for all browser interactions.

You have successfully configured Selenium with the ProxyPanel proxy. Below is the output from the configuration: