How to Set Proxy in AIOHTTP

{ "@context": "https://schema.org", "@type": "HowTo", "name": "Setting Proxy in AIOHTTP", "description": "This guide will teach you how to set an HTTP, HTTPS, or SOCKS proxy in AIOHTTP, including handling proxy authentication, dealing with SSL certificate errors, implementing IP rotation, and more.", "step": [ { "@type": "HowToStep", "name": "Understand AIOHTTP", "text": "Learn about AIOHTTP, a popular Python async HTTP client/server framework that supports both client and server sides of the HTTP protocol, web sockets, and provides a web server with middleware and routing." }, { "@type": "HowToStep", "name": "Know Why to Use a Proxy", "text": "Using a proxy with AIOHTTP can protect your online identity, add confidentiality to your operations, avoid IP bans, and overcome geo-restrictions." }, { "@type": "HowToStep", "name": "Install AIOHTTP and Dependencies", "text": "Ensure Python is installed and set up a virtual environment for your project. Install AIOHTTP and its recommended dependencies using pip." }, { "@type": "HowToStep", "name": "Understand Proxy Prerequisites", "text": "Familiarize yourself with the structure of a proxy URL and the importance of choosing reliable premium proxies over free ones for production uses." }, { "@type": "HowToStep", "name": "Set an HTTP Proxy in AIOHTTP", "text": "Learn to integrate an HTTP proxy into your AIOHTTP requests using the 'proxy' argument in the session.get() method." }, { "@type": "HowToStep", "name": "Set an HTTPS Proxy in AIOHTTP", "text": "Understand the requirements for using HTTPS proxies in AIOHTTP, including necessary Python and AIOHTTP versions, and apply similar integration methods as for HTTP proxies." }, { "@type": "HowToStep", "name": "Integrate SOCKS Proxy with AIOHTTP", "text": "Install the aiohttp_socks extension to enable SOCKS proxy support in AIOHTTP and use it in your requests." }, { "@type": "HowToStep", "name": "Implement Advanced Proxy Use Cases", "text": "Explore advanced use cases such as setting global proxies through environment variables, handling proxy authentication, avoiding SSL certificate errors, and rotating proxies in AIOHTTP." }, { "@type": "HowToStep", "name": "Use demlon Proxy with AIOHTTP", "text": "Sign up for demlon, configure a residential proxy, and integrate it into your AIOHTTP project for enhanced web scraping capabilities." } ], "supply": [ { "@type": "HowToSupply", "name": "Python installation" }, { "@type": "HowToSupply", "name": "AIOHTTP package" }, { "@type": "HowToSupply", "name": "aiohttp_socks extension" }, { "@type": "HowToSupply", "name": "demlon account" } ], "tool": [ { "@type": "HowToTool", "name": "Visual Studio Code or similar text editor" }, { "@type": "HowToTool", "name": "Virtual environment for Python project" }, { "@type": "HowToTool", "name": "Terminal or Command Prompt" } ], "estimatedCost": { "@type": "MonetaryAmount", "currency": "USD", "value": "Varies" }, "totalTime": "P0DT12H0M" }

At the end of this AIOHTTP proxy integration guide, you will know:

What AIOHTTP is.

Why you should use a proxy.

How to set an HTTP, HTTPS, or SOCKS proxy in AIOHTTP.

ow to deal with proxy authentication, SSL certificate errors, IP rotation, and more.

What Is AIOHTTP?

AIOHTTP is a popular, open-source, Python async HTTP client/server framework. Its main features are:

Support for both client and server side of HTTP protocol.

Support for both client and server web sockets.

Provide a web server with middleware and pluggable routing.

What matters in this tutorial is its ability to act as an HTTP client to perform HTTP requests on top of asyncio. AIOHTTP is particularly well-suited for building scripts that need to handle multiple concurrent requests efficiently.

As of this writing, AIOHTTP has over 14k stars on GitHub .

Why Do You Need an AIOHTTP Proxy?

A good reason for using a proxy in AIOHTTP is to protect your online identity. By routing your request to a proxy server , you can hide your IP and location, making the destination site believe that your request comes from the proxy server and not from you. This means that it will be more difficult to identify and track you, adding confidentiality to your online operations while avoiding IP bans and geo-restrictions.

Note that you can achieve the same result with requests , as covered in our requests proxy integration tutorial .

Setting a Proxy in AIOHTTP Proxy

In this step-by-step section, you will learn how to set an HTTP, HTTPS, or SOCKS proxy in AIOHTTP.

AIOHTTP Prerequisites

Here, we will assume that you already have Python installed on your machine and a Python project with a virtual environment in place. Install AIOHTTP and its recommended dependencies with the following command:

pip install aiohttp[speedups]

This will add aiohttp to your project’s dependencies.

Open your Python script file and add the aiohttp import:

import aiohttp

To make aiohttp work with the async/await syntax, you will have to use asyncio . So import that library as well and set up a basic async main() function:

import aiohttp
import aiohttp

import asyncio

async def main():
  # aiohttp logic...

asyncio.run(main())

Perfect! You are ready to write the AIOHTTP proxy integration logic!

Proxy Prerequisites

Before seeing how to integrate a proxy in AIOHTTP, you need to understand what a proxy URL looks like.

This is the format of the URL required to connect to a proxy server in AIOHTTP:

<PROTOCOL>://[<USERNAME>:<PASSWORD>@]<HOST>[:<PORT>]

In detail, that consists of:

<PROTOCOL> : http for HTTP proxies, https for HTTPS proxies, and socks , socks4 , or socks5 for proxies in the different SOCKS protocols.

<HOST> : The IP of the proxy server or its entire domain (e.g. proxydomain.com ).

<PORT> : The port the proxy server listens to.

<USERNAME> : The optional username to authenticate the proxy connection.

<PASSWORD> : The optional password to authenticate the proxy connection.

Note that <PROTOCOL> and <HOST> are the only required parts of the URL. The <PORT> section is optional, but you typically have to specify it. Lastly, the credentials section <USERNAME>:<PASSWORD> is only necessary when dealing with authenticated proxies.

If you do not have the URL of a proxy server, you can get one online for free. At the same time, keep in mind that free proxies are unreliable, data-greedy, inconsistent, error-prone, and shady. You can use them only for learning purposes . In production, you will need a reliable premium proxy from a top-notch provider like demlon.

Later in this article, you will see how to get a demlon proxy and use it in AIOHTTP.

HTTP Proxy Integration

As mentioned in the documentation, AIOHTTP supports HTTP proxy integration. Assume this is the URL of your HTTP proxy:

http://190.6.23.219:999

You can specify it in your AIOHTTP request by using the proxy option as below:

async with aiohttp.ClientSession() as session:
    async with session.get(
        'https://example.com',
        proxy='http://190.6.23.219:999'
    ) as response:
        print(response.status)

AIOHTTP will now route the GET request through the proxy URL passed as a proxy argument to the get() function.

Keep in mind that you can use an HTTP proxy for both HTTP and HTTPS traffic.

HTTPS Proxy Integration

When it comes to HTTPS proxies, things become a bit more complex. As stated in the AIOHTTP documentation:

“In Python 3.10, support for TLS in TLS is disabled for the transports that asyncio uses . If the further release of Python (say v3.11) toggles one attribute, it’ll just work™. aiohttp v3.8 and higher is ready for this to happen and has code in place supports TLS-in-TLS, hence sending HTTPS requests over HTTPS proxy tunnels.”

“In Python 3.10, support for TLS in TLS is disabled for the transports that asyncio uses . If the further release of Python (say v3.11) toggles one attribute, it’ll just work™.

aiohttp v3.8 and higher is ready for this to happen and has code in place supports TLS-in-TLS, hence sending HTTPS requests over HTTPS proxy tunnels.”

In other words, if you want to use HTTPS proxies in AIOHTTP, you need to upgrade Python to version 3.11 and aiohttp to v3.8 or higher. The approach to HTTPS AIOHTTP proxy integration remains the same as seen above. The only aspect that changes is that the proxy protocol will be https and no longer http .

SOCKS Proxy Integration

AIOHTTP does not support SOCKS proxies out of the box. If you have to use a SOCKS proxy in AIOHTTP, you have to install aiohtpp-socks :

pip install aiohttp_socks

This library uses python-socks to extend AIOHTTP with SOCKS4(a) and SOCKS5(h) support.

Consider this as the URL of your SOCKS proxy:

socks5://98.175.31.195:4145

Use it in AIOHTTP as follows:

# initialize a SOCKS proxy connector
connector = ProxyConnector.from_url('socks5://98.175.31.195:4145')

# initialize an AIOHTTP client with the SOCKS proxy connector
async with aiohttp.ClientSession(connector=connector) as session:
    async with session.get('https://example.com') as response:
        print(response.status)

AIOHTTP will now route the GET request through the specified SOCKS proxy.

ProxyConnector comes from aiohttp-socks , so do not forget to import with:

from aiohttp_socks import ProxyConnector

Behind the scenes, aiohttp-socks defines an AIOHTTP Connecto r to tweak the requests and make them go through the given SOCKS proxy.

Complete Example

You just learned how to use a proxy with AIOHTTP. Let’s verify that the above approaches work in a complete example!

For the sake of simplicity, we will work with an HTTP proxy, but you can easily adapt the example below to HTTPS and SOCKS.

To target URL of the GET request will be the /ip endpoint from the HTTPBin project. This API returns the IP of the incoming request. Thus, if the proxy integration is successful, the endpoint should respond with the IP of the proxy server and not yours.

Again, assume this is the URL of your HTTP proxy server:

http://190.6.23.219:999

Integrate it into AIOHTTP and print the data returned by the https://httpbin.io/ip endpoint with:

import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        # perform a GET request through an HTTP proxy
        async with session.get(
          'https://httpbin.io/ip',
          proxy='http://190.6.23.219:999'
        ) as response:
            # extract the response data and print it
            json = await response.json()
            print(json)

asyncio.run(main())

Execute this script, and you should get:

{
  'origin': '190.6.23.219'
}

Et voilà! That is exactly the IP of the proxy server, which means that the AIOHTTP proxy integration works like a charm.

Note : Proxy servers are short-lived. By the time you read this tutorial, the selected proxy will no longer work and the above script will fail. To make it work, replace the HTTP proxy with a fresh and working one.

AIOHTPP Proxy: Advanced Use Cases

You know the basics of AIOTTP proxy integration, but what about more complex techniques? Find out more in this section!

Global Proxies Via Environment Variables

AIOHTTP supports two ways to determine which proxy to use:

It uses the proxy URL passed to the proxy argument.

If trust_env is set to True , it reads the HTTP_PROXY and HTTPS_PROXY environment variables.

You can therefore configure a proxy globally in AIOHTTP by setting the envs below:

HTTP_PROXY : The URL of the proxy server to use for HTTP requests.

HTTPS_PROXY : The URL of the proxy server to use for HTTPS requests.

For example, set them on Linux or macOS with the following commands:

export HTTP_PROXY="<PROTOCOL>://[<USERNAME>:<PASSWORD>@]<HOST>[:<PORT>]"
export HTTPS_PROXY="<PROTOCOL>://[<USERNAME>:<PASSWORD>@]<HOST>[:<PORT>]"

Next, pass trust_env=True to aiohttp.ClientSession() :

async with aiohttp.ClientSession(trust_env=True) as session:

You can now call session.get() with no proxy argument

async with aiohttp.ClientSession(trust_env=True) as session:
    async with session.get('https://example.com') as response:
        print(response.status)

Great! AIOHTTP will use the proxies specified in the environment variables for each HTTP and HTTPS request.

Follow the link to learn more, as the same mechanism also works with proxies in cURL .

Proxy Authentication Approaches

Trying to connect to an authenticated proxy without the right credentials will result in a 407 Proxy Authentication Required error.

Suppose the string below is the URL to connect to an authenticated HTTP proxy:

http://admin:[email protected]:8928

Here, the bare proxy URL is:

http://20.198.112.312:892

And the authentication credentials are:

Username: admin

Password: pK5ip98NWp56l9sjy

AIOHTTP supports two ways to deal with authenticated proxies:

Specify the credentials directly in the proxy URL:

async with session.get(
  'https://example.com/',
  proxy='http://admin:[email protected]:8928'
) as response:

2. Pass the username and password credentials to an aiohttp.BasicAuth() object and then use it in the proxy_auth option:

proxy_auth = aiohttp.BasicAuth('admin', 'pK5ip98NWp56l9sjy')
async with session.get(
    'https://example.com/',
    proxy="http://20.198.112.312:892",
    proxy_auth=proxy_auth
) as response:

Awesome! AIOHTTP proxy authentication is no longer a secret.

Avoid SSL Certificate Errors

When setting a proxy in AIOHTTP, your requests may fail because of the following error:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain 

To avoid these SSL certificate issues, turn off SSL verification with ssl=False :

async with aiohttp.ClientSession() as session:
    async with session.get(
      'https://example.com',
      proxy='http://190.6.23.219:999',
      ssl=False # turn of SSL certificate verification
    ) as response:
        print(response.status)

The AIOHTTP SSL certificate verification errors will immediately disappear.

Rotating Proxies in AIOHTTP

If you rely on the same proxy server over and over again, there is a chance that the target site will block its IP address. To avoid that, you must ensure that your request uses different proxy servers.

Here is how you can do it:

Populate a Python list with many proxy URLs.

Randomly pick a proxy URL before each request.

Set the chosen proxy URL in AIOHTTP.

The main problem with this approach is that it involves boilerplate code. Also, it requires access to a pool of reliable proxy servers, which is not free. Luckily, demlon has a more effective solution! Its rotating proxies automatically change the exit IP addresses for you, are available in 195 countries, boast exceptional network uptime, and guarantee a success rate of 99.9%.

In the next chapter, you will learn how to get started with demlon’s rotating proxies in AIOHTTP.

Integrating AIOHTTP with a demlon Proxy

demlon controls the best proxy servers in the world, serving Fortune 500 companies and over 20,000 customers. Its worldwide proxy network involves:

Datacenter proxies – Over 770,000 datacenter IPs.

Residential proxies – Over 72M residential IPs in more than 195 countries.

ISP proxies – Over 700,000 ISP IPs.

Mobile proxies – Over 7M mobile IPs.

This is one of the largest and most reliable proxy networks in the world!

Follow the steps below and learn how to use demlon’s proxies in AIOHTTP.

If you already have an account, log in to demlon. Otherwise, create an account for free. You will gain access to the following user dashboard:

Click the “View proxy products” button as below:

view proxy products

You will be redirected to the following “Proxies & Scraping Infrastructure” page:

Scroll down, find the “Residential Proxies” card, and click on the “Get started” button:

resi proxy

You will reach the residential proxy configuration dashboard. Follow the guided wizard and set up the proxy service based on your needs. If you have any doubts about how to configure the proxy, feel free to contact the 24/7 support.

Go to the “Access parameters” tab and retrieve your proxy’s host, port, username, and password as follows:

access parameter

Note that the “Host” field already includes the port.

That is all you need to build the proxy URL and use it in AIOHTTP. Put all the information together, and build a URL with the following syntax:

<Username>:<Password>@<Host>

For example, in this case it would be:

brd-customer-hl_4hgu8dwd-zone-residential:[email protected]:XXXXX

Toggle “Active proxy,” follow the last instructions, and you are ready to go!

active proxy

Your AIOHTTP proxy snippet for demlon integration will look like as follows:

import aiohttp
import asyncio

async def main():
    async with aiohttp.ClientSession() as session:
        # perform a GET request through an HTTP proxy
        async with session.get(
          'https://lumtest.com/myip.json',
          proxy='http://brd-customer-hl_4hgu8dwd-zone-residential:[email protected]:XXXXX'
        ) as response:
            # extract the response data and print it
            json = await response.json()
            print(json)

asyncio.run(main())

Conclusion

In this AIOHTTP proxy integration tutorial, you saw why you should use proxies and how to do it with aiohttp . You now know how to set up an HTTP, HTTPS, or SOCKS proxy in AIOHTTP, a popular Python HTTP client library. As proved here, that only takes a few lines of code!

Thanks to this guide, you also understood why you should never use free proxy services. Instead, you should adopt reliable proxy servers from a reputable proxy service provider. The best provider in the market? demlon! The proxy integration procedure in AIOHTTP remains the same, but the benefits of premium proxies are endless.

Start free trial

Start free trial