--- title: "Concurrent HTTP Requests in Python with aiohttp and asyncio" lang: "en" author: "Mohammad Abu Mattar" canonical: https://mkabumattar.com/post/python-async-http-aiohttp-concurrent-requests --- ![Blog post image for Python Async HTTP Requests with aiohttp: Fetch Multiple URLs Concurrently - A Python snippet demonstrating concurrent HTTP requests using aiohttp and asyncio. Covers creating sessions, fetching multiple URLs in parallel, handling errors gracefully, and controlling concurrency with semaphores.](/_astro/hero.DFXqyT8a_1dJYNF.webp) [Home](/)›[Codesnippets](/codesnippets)›[All Categories](/codesnippets/categories)›[Python](/codesnippets/categories/python) Codesnippets [Next in PythonList S3 Buckets](/codesnippets/post/python-list-s3-buckets) [Python](/codesnippets/categories/python)[Async](/codesnippets/categories/async)[Networking](/codesnippets/categories/networking)[Python](/codesnippets/python) # Python Async HTTP Requests with aiohttp: Fetch Multiple URLs Concurrently [Mohammad Abu Mattar](/authors/mohammad-abu-mattar)Published: 26 Jul 2026Updated: 26 Jul 202605 Mins read07 Mins listen [Markdown for AI(opens in a new tab)](/post/python-async-http-aiohttp-concurrent-requests/index.md "Open the plain-Markdown version of this page, for pasting into an AI tool") TL;DR A Python snippet demonstrating concurrent HTTP requests using aiohttp and asyncio. Covers creating sessions, fetching multiple URLs in parallel, handling errors gracefully, and controlling concurrency with semaphores. Series [Python Snippets](/series/python-snippets)1/1 All posts in this series (1) Code Snippets1 1. [Python Async HTTP Requests with aiohttp: Fetch Multiple URLs ConcurrentlyYou are here](/codesnippets/post/python-async-http-aiohttp-concurrent-requests) ### Python Async HTTP Requests with aiohttp: Fetch Multiple URLs Concurrently Contents [The Problem](#the-problem)[Why sequential requests waste time](#why-sequential-requests-waste-time)[The real-world impact](#the-real-world-impact)[The Solution](#the-solution)[Script Implementation](#script-implementation)[Setup and imports](#setup-and-imports)[Fetching one URL safely](#fetching-one-url-safely)[Fetching the whole batch](#fetching-the-whole-batch)[Entry point](#entry-point)[Usage and Benefits](#usage-and-benefits)[Real invocations](#real-invocations)[Tuning concurrency for the target](#tuning-concurrency-for-the-target)[Comparison](#comparison)[Frequently Asked Questions](#frequently-asked-questions)[References](#references) **Quick Tip** Reuse one `aiohttp` session, fan your requests out with `asyncio.gather`, and cap them with a semaphore to fetch hundreds of URLs in the time one loop would take. ## [The Problem](#the-problem) **The Problem** A script that fetches a list of URLs with `requests` in a plain `for` loop spends almost all of its time doing nothing. Each call opens a connection, sends the request, and then blocks until the response comes back before the next one even starts. The network is slow and your CPU is fast, so the program just sits and waits, one round trip at a time. ### [Why sequential requests waste time](#why-sequential-requests-waste-time) HTTP is I/O bound. The bottleneck isn’t your code, it’s the latency of each request. If a single call takes 200ms and you have 200 URLs, a sequential loop takes 40 seconds, and 39 of those seconds are pure waiting. Nothing about that work needs to happen in order, yet the loop forces it to. ### [The real-world impact](#the-real-world-impact) That waiting shows up as slow batch jobs, sluggish data-collection scripts, and API clients that time out because they can’t keep up. Scale the list up and it gets worse linearly: a report that pulls from a few hundred endpoints turns a quick task into a coffee break, and a service that fans out to several upstreams makes every user wait for the slowest chain of calls. ## [The Solution](#the-solution) **The Fix** Use `aiohttp` with `asyncio` so the requests overlap. While one call waits on the network, the event loop starts the next one. Reuse a single `ClientSession` so connections get pooled, launch everything with `asyncio.gather`, and put a `Semaphore` in front to keep the number of in-flight requests sane. **TL;DR** - Create one `aiohttp.ClientSession` and reuse it for every request. - Fire all the fetches concurrently with `asyncio.gather` instead of a blocking loop. - Cap concurrency with `asyncio.Semaphore` so you don’t overwhelm the server or get rate-limited. ## [Script Implementation](#script-implementation) ### [Setup and imports](#setup-and-imports) Start with the async pieces you need: `asyncio` for the event loop and `aiohttp` for the HTTP calls. A small result type keeps the return values easy to read. fetch\_urls.py ``` 1#!/usr/bin/env python32"""fetch_urls.py - fetch many URLs concurrently with aiohttp."""3import asyncio4import time5from dataclasses import dataclass6 7import aiohttp8 9# Cap how many requests run at once. Tune to the server and your rate limits.10MAX_CONCURRENCY = 1011REQUEST_TIMEOUT = aiohttp.ClientTimeout(total=15)12 13 14@dataclass15class Result:16 url: str17 status: int | None # HTTP status, or None if the request never completed18 body: str | None # response text on success19 error: str | None # error message on failure ``` ### [Fetching one URL safely](#fetching-one-url-safely) Each fetch takes the shared session and the semaphore. The semaphore blocks once `MAX_CONCURRENCY` requests are already in flight, so the rest queue up instead of stampeding the server. The `try/except` keeps a single bad URL from crashing the whole batch. ``` 1async def fetch_one(2 session: aiohttp.ClientSession,3 sem: asyncio.Semaphore,4 url: str,5) -> Result:6 # The semaphore is the throttle: only MAX_CONCURRENCY of these run at once.7 async with sem:8 try:9 async with session.get(url) as response:10 # Read the body while the connection is still open.11 text = await response.text()12 return Result(url, response.status, text, None)13 except aiohttp.ClientError as exc:14 # Network, DNS, or connection errors land here.15 return Result(url, None, None, f"client error: {exc}")16 except asyncio.TimeoutError:17 return Result(url, None, None, "timed out") ``` ### [Fetching the whole batch](#fetching-the-whole-batch) Create one session and one semaphore, then build a task per URL and hand them all to `asyncio.gather`. Because every `fetch_one` already catches its own errors, `gather` returns a clean list of `Result` objects with no partial failures to unpack. ``` 1async def fetch_all(urls: list[str]) -> list[Result]:2 sem = asyncio.Semaphore(MAX_CONCURRENCY)3 # One session for the whole run so connections get pooled and reused.4 async with aiohttp.ClientSession(timeout=REQUEST_TIMEOUT) as session:5 tasks = [fetch_one(session, sem, url) for url in urls]6 # gather runs them concurrently and preserves input order.7 return await asyncio.gather(*tasks) ``` ### [Entry point](#entry-point) Wrap the run in `asyncio.run`, then print a short summary. Timing the run makes the win obvious the first time you compare it to a sequential loop. ``` 1def main() -> None:2 urls = [f"https://httpbin.org/delay/1?id={i}" for i in range(20)]3 4 start = time.perf_counter()5 results = asyncio.run(fetch_all(urls))6 elapsed = time.perf_counter() - start7 8 ok = sum(1 for r in results if r.status == 200)9 failed = [r for r in results if r.error]10 print(f"fetched {ok}/{len(results)} in {elapsed:.2f}s")11 for r in failed:12 print(f" failed: {r.url} ({r.error})")13 14 15if __name__ == "__main__":16 main() ``` ## [Usage and Benefits](#usage-and-benefits) **Why This Helps** One session, one semaphore, and one `gather` call turn a linear wait into an overlapping one. The requests still take the same time individually, but they no longer take that time one after another, so a batch that scaled with the number of URLs now scales with your concurrency limit instead. ### [Real invocations](#real-invocations) Terminal window ``` 1# Install the dependency first.2pip install aiohttp3 4# Run the script. Twenty 1-second requests finish in about 2-3 seconds,5# not 20, because up to MAX_CONCURRENCY of them run at the same time.6python fetch_urls.py ``` ### [Tuning concurrency for the target](#tuning-concurrency-for-the-target) There’s no universal number for `MAX_CONCURRENCY`. A generous internal API might handle 50 in-flight requests; a rate-limited public one might block you above 5. Start low, watch for `429` responses, and raise it until you hit the server’s comfort zone. ``` 1# Lower the limit for a strict public API to avoid 429s.2MAX_CONCURRENCY = 53 4# Raise it for a fast internal service that can take the load.5MAX_CONCURRENCY = 50 ``` ## [Comparison](#comparison) How the concurrent approach stacks up against the other common ways to fetch a list of URLs. Approach Concurrent Connection reuse Concurrency control Best for `aiohttp` + `asyncio.gather` Yes Yes, pooled Yes, semaphore Large async I/O-bound fetches `requests` in a `for` loop No Per session only N/A A handful of simple calls `requests` + `ThreadPool` Yes Yes Pool size Mixing with blocking code `httpx.AsyncClient` Yes Yes, pooled Yes, limits/semaphore Async with a requests-like API The threaded pool is a fine middle ground when the rest of your code is synchronous, but an event loop scales to far more concurrent connections without spinning up a thread per request. ## [Frequently Asked Questions](#frequently-asked-questions) A `ClientSession` owns a connection pool. Reusing it lets aiohttp keep TCP and TLS connections alive across requests, which skips the handshake cost on every call. Creating a fresh session per request throws that away and leaks connections. Open one session for the batch, pass it to every fetch, and close it with `async with` when you’re done. Without a limit, `asyncio.gather` starts every request at once. A thousand URLs means a thousand simultaneous connections, which can exhaust file descriptors on your side and trip rate limits or overload the server on the other. The semaphore caps how many run concurrently: each `async with sem` acquires a slot and releases it when the request finishes, so the rest wait their turn. Catch the errors inside each task, which is what `fetch_one` does with its `try/except`. Because every coroutine returns a `Result` instead of raising, `asyncio.gather` never sees an exception and returns a full list. If you’d rather let exceptions propagate, use `asyncio.gather(*tasks, return_exceptions=True)` and inspect the returned values, some of which will be exception objects. Yes. `gather` returns results in the same order you passed the awaitables, regardless of which requests finish first. So `results[i]` always corresponds to `urls[i]`, even though the fetches complete out of order. If you don’t care about order and want to process results as they arrive, look at `asyncio.as_completed` instead. Reach for a thread pool with `requests` when the surrounding code is synchronous and you don’t want to convert it to async just for a few calls. Reach for `httpx.AsyncClient` when you want an async client with an API closer to `requests`, or you need HTTP/2. `aiohttp` shines when you’re already in an async application and need to fan out to many endpoints at high concurrency. Pass an `aiohttp.ClientTimeout` to the session, as the snippet does with `REQUEST_TIMEOUT`. A `total` timeout bounds the whole request, including connect and read. When it’s exceeded, aiohttp raises `asyncio.TimeoutError`, which the `except` in `fetch_one` turns into a clean failed `Result` instead of a hung coroutine. ## [References](#references) - [aiohttp documentation: client quickstart](https://docs.aiohttp.org/en/stable/client_quickstart.html) - [aiohttp documentation: client reference and ClientSession](https://docs.aiohttp.org/en/stable/client_reference.html) - [Python docs: asyncio.gather](https://docs.python.org/3/library/asyncio-task.html#asyncio.gather) - [Python docs: asyncio.Semaphore](https://docs.python.org/3/library/asyncio-sync.html#asyncio.Semaphore) - [Python docs: asyncio.as\_completed](https://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed) - [httpx documentation: async client](https://www.python-httpx.org/async/) Was this useful? ## Tags [#Python](/codesnippets/tags/python)[#Aiohttp](/codesnippets/tags/aiohttp)[#Asyncio](/codesnippets/tags/asyncio)[#Concurrency](/codesnippets/tags/concurrency)[#HTTP](/codesnippets/tags/http)[#Performance](/codesnippets/tags/performance) ## Share [Facebook](https://facebook.com/sharer/sharer.php?u=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests "Share on Facebook")[Twitter](https://twitter.com/intent/tweet/?text=Python%20Async%20HTTP%20Requests%20with%20aiohttp%3A%20Fetch%20Multiple%20URLs%20Concurrently&url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests "Share on Twitter")[LinkedIn](https://www.linkedin.com/shareArticle?mini=true&url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests&title=Python%20Async%20HTTP%20Requests%20with%20aiohttp%3A%20Fetch%20Multiple%20URLs%20Concurrently&summary=A%20Python%20snippet%20demonstrating%20concurrent%20HTTP%20requests%20using%20aiohttp%20and%20asyncio.%20Covers%20creating%20sessions%2C%20fetching%20multiple%20URLs%20in%20parallel%2C%20handling%20errors%20gracefully%2C%20and%20controlling%20concurrency%20with%20semaphores.&source=https://mkabumattar.com "Share on LinkedIn")[WhatsApp](https://wa.me/?text=Python%20Async%20HTTP%20Requests%20with%20aiohttp%3A%20Fetch%20Multiple%20URLs%20Concurrently%20https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests "Share on WhatsApp")[Telegram](https://t.me/share/url?url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests&text=Python%20Async%20HTTP%20Requests%20with%20aiohttp%3A%20Fetch%20Multiple%20URLs%20Concurrently "Share on Telegram")[Reddit](https://www.reddit.com/submit?url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests&title=Python%20Async%20HTTP%20Requests%20with%20aiohttp%3A%20Fetch%20Multiple%20URLs%20Concurrently "Share on Reddit")[Hacker News](http://news.ycombinator.com/submitlink?u=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests&t=Python%20Async%20HTTP%20Requests%20with%20aiohttp%3A%20Fetch%20Multiple%20URLs%20Concurrently "Share on Hacker News")[Pinterest](https://pinterest.com/pin/create/button/?url=https%3A%2F%2Fmkabumattar.com%2Fcodesnippets%2Fpost%2Fpython-async-http-aiohttp-concurrent-requests&media=&description=A%20Python%20snippet%20demonstrating%20concurrent%20HTTP%20requests%20using%20aiohttp%20and%20asyncio.%20Covers%20creating%20sessions%2C%20fetching%20multiple%20URLs%20in%20parallel%2C%20handling%20errors%20gracefully%2C%20and%20controlling%20concurrency%20with%20semaphores. "Share on Pinterest")[Email]() ## Comments ## You might also enjoy More posts on similar topics [![Optimizing your python code with \_\_slots\_\_?](/_astro/hero.DP_vYsHU_gYDXn.webp)](/codesnippets/post/python-slots-optimization) ## [Optimizing your python code with \_\_slots\_\_?](/codesnippets/post/python-slots-optimization) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Python](/codesnippets/categories/python) - [Productivity](/codesnippets/categories/productivity) Memory optimization with slots Understanding the problem Optimizing data models in big data workflows with slots In big data and MLOps workflows, you often work with massive [#Python](/codesnippets/tags/python)[#MemoryOptimization](/codesnippets/tags/memoryoptimization)[#DataScience](/codesnippets/tags/datascience)+4 tags [read more](/codesnippets/post/python-slots-optimization) [![List S3 Buckets](/_astro/hero.BsiJ6hry_1X9XLn.webp)](/codesnippets/post/python-list-s3-buckets) ## [List S3 Buckets](/codesnippets/post/python-list-s3-buckets) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Aws](/codesnippets/categories/aws) - [Python](/codesnippets/categories/python) - [Devops](/codesnippets/categories/devops) Overview Multi-profile S3 management Ever juggled multiple AWS accounts and needed a quick S3 bucket inventory across all of them? This Python script handles it. Use case Perfect for or [#Python](/codesnippets/tags/python)[#Boto3](/codesnippets/tags/boto3)[#AWS](/codesnippets/tags/aws)+5 tags [read more](/codesnippets/post/python-list-s3-buckets) [![Redis Caching Patterns: Cache-Aside, Write-Through & Cache Invalidation](/_astro/hero.Ck2oLF89_ZPiYlL.webp)](/codesnippets/post/redis-caching-patterns-architecture) ## [Redis Caching Patterns: Cache-Aside, Write-Through & Cache Invalidation](/codesnippets/post/redis-caching-patterns-architecture) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Backend](/codesnippets/categories/backend) - [Performance](/codesnippets/categories/performance) Need to scale your backend without throwing money at servers? Start with Redis caching patterns. Most databases can handle hundreds of queries per second, but thousands? Your app slows to a crawl [#Redis](/codesnippets/tags/redis)[#Caching](/codesnippets/tags/caching)[#Performance](/codesnippets/tags/performance)+6 tags [read more](/codesnippets/post/redis-caching-patterns-architecture) [![AWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances](/_astro/hero.PnHlvJay_ZkFqWG.webp)](/codesnippets/post/aws-ec2-instance-management-boto3-python) ## [AWS EC2 Instance Management with Boto3: Start, Stop, and Query Instances](/codesnippets/post/aws-ec2-instance-management-boto3-python) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Cloud](/codesnippets/categories/cloud) - [Aws](/codesnippets/categories/aws) - [Devops](/codesnippets/categories/devops) - [Automation](/codesnippets/categories/automation) If you've ever spent 20 minutes clicking through the AWS Console just to stop a handful of dev instances, you already know the pain. It's tedious, it doesn't scale, and one wrong click can ruin your a [#AWS](/codesnippets/tags/aws)[#EC2](/codesnippets/tags/ec2)[#Boto3](/codesnippets/tags/boto3)+6 tags [read more](/codesnippets/post/aws-ec2-instance-management-boto3-python) [![PostgreSQL Query Optimization: Indexes, EXPLAIN ANALYZE & Execution Plans](/_astro/hero.WD7Zwlat_2dqoYO.webp)](/codesnippets/post/postgresql-query-optimization-indexes-explain) ## [PostgreSQL Query Optimization: Indexes, EXPLAIN ANALYZE & Execution Plans](/codesnippets/post/postgresql-query-optimization-indexes-explain) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Database](/codesnippets/categories/database) - [Performance](/codesnippets/categories/performance) Need to optimize slow PostgreSQL queries? Use EXPLAIN ANALYZE and targeted indexing. Slow database queries kill application performance. Most developers don't know where the actual bottleneck is, [#PostgreSQL](/codesnippets/tags/postgresql)[#QueryOptimization](/codesnippets/tags/queryoptimization)[#Indexes](/codesnippets/tags/indexes)+4 tags [read more](/codesnippets/post/postgresql-query-optimization-indexes-explain) [![Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff](/_astro/hero.Dap62rVN_Z1q0nar.webp)](/codesnippets/post/bash-retry-function-exponential-backoff) ## [Bash Retry Function: Automatically Retry Failing Commands with Exponential Backoff](/codesnippets/post/bash-retry-function-exponential-backoff) - [Mohammad Abu Mattar](/authors/mohammad-abu-mattar) - [Devops](/codesnippets/categories/devops) - [Shell scripting](/codesnippets/categories/shell-scripting) - [Automation](/codesnippets/categories/automation) Quick Tip Wrap any flaky command in one reusable retry function and stop re-running red pipelines by hand. The Problem The Problem Some commands fail for reasons that have nothing to [#Bash](/codesnippets/tags/bash)[#Retry](/codesnippets/tags/retry)[#Exponential Backoff](/codesnippets/tags/exponential-backoff)+3 tags [read more](/codesnippets/post/bash-retry-function-exponential-backoff) 6 related posts