Wim Onderbeke

/home

/posts

Home > Posts > How asyncio works

How asyncio works

An in-depth look at how asyncio works in Python

  ·   16 min read

I’ve been using asyncio for a while now and I’ve always been curious about how it works under the hood. How does its event loop run coroutines? How can another coroutine take over during a blocking I/O operation? How does the event loop know when the operating system has completed that I/O operation? In this blog post I want to explore what really happens when you await a coroutine. First let’s see how coroutines declared with the async/await syntax came to be in Python. Afterwards we’ll dive deeper into the asyncio event loop.

From generators to coroutines

PEP 225 introduced generators and the yield statement to Python. Generator are iterators that are often used to generate a sequence of values on-the-fly, instead of storing them all in memory. When we use the yield keyword within a function body, that function turns into a generator function:

def gen_123():
    yield 1
    yield 2
    yield 3

When we call that function, we create a generator object:

<generator object gen_123 at 0x1bb06b8>

This object implements the iterator protocol (__next__() and __iter__() methods), and can therefore be used in for loops or called directly via the next() function:

>>> for number in gen_123():
...     print(number)
...
1
2
3

>>> gen = gen_123()
>>> next(gen)
1
>>> next(gen)
2
>>> next(gen)
3
>>> next(gen)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Whenever __next__() is called on a generator object, the function is evaluated until the next yield statement and then the value of the yield is returned. Afterwards, the generator function is suspended and its stack frame is stored. When the end of the function is reached, or a return statement is encountered, a StopIteration exception is raised.

Returning values from generators

The body of a generator function also allows for a return statement. Returning a value from a generator results in that value being stored inside of the StopIteration exception:

>>> def gen_1_and_return():
...     yield 1
...     return "abc"
...
>>> gen = gen_1_and_return()
>>> next(gen)
1
>>> try:
...     next(gen)
... except StopIteration as exc:
...     print(exc.value)
abc

send() and throw()

A less-known feature of generators is that you can also send data to them or throw exceptions in them using send() and throw():

>>> def send_and_throw():
...     val = yield 1 # Capture the send value
...     print(val)
...     yield 2
...     yield 3
...
>>> gen = send_and_throw()
>>> next(gen)
1
>>> gen.send("abc")
abc
2
>>> gen.throw(Exception())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 4, in test
Exception

The value you send() is passed as the return value from the yield keyword. Calling send(None) is equivalent to calling next() on the generator. throw() allows raising exceptions inside generators, with the exception raised at the same spot yield was called.

Yield from

Yield from allows a generator to delegate its operations to a nested generator. Any next(), send() or throw() call is passed into the nested generator. The most common way to use yield from is to yield values from an iterable:

>>> def gen_123():
...     yield 1
...     yield 2
...     yield 3
...
>>> for i in gen_123():
...     print(i)
1
2
3

Furthermore, yield from will also capture the return value of the nested generator (i.e. value of the StopIteration exception). That return value will also be the final return value of the yield from:

>>> def return_1_and_yield():
...     return 1
...     yield
... 
>>> def print_1_and_error():
...     result = yield from return_1_and_yield()
...     print(result)
... 
>>> gen = print_1_and_error()
>>> next(gen)
1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Generators, coroutines and async/await

Generators in Python are effectively coroutines. They can be suspended, resumed, and you can use them for cooperative multitasking. However, using generators as coroutines has a number of downsides:

  • It is easy to confuse coroutines with “regular” generators, since they share the same syntax.
  • Whether or not a function is a coroutine is determined by a presence of yield or yield from statements in its body, which can lead to unobvious errors when such statements appear in or disappear from the function body during refactoring.

Therefore PEP 492 introduced native coroutines and the async/await syntax to Python. You can define a native coroutine using the await keyword:

>>> async def coro_1():
...     return 1
...
>>> coro_1()
<coroutine object coro_1 at 0x100318b40>

When you call such a function, it returns a native coroutine object. This object behaves in almost the exact same way as a generator. You can still call send() on them, but calling next() will result in an error. This is to prevent them from being used as iterators (like regular generators).

Native coroutines can call each other with the await keyword:

>>> async def coro_2():
...     r = await coro_1()
...     return 1 + r
... 
>>> coro_2().send(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 2

The await keyword does exactly what yield from does but for native coroutines. In fact, await is implemented as yield from with some additional checks to ensure that the object being awaited is not a generator or some other iterable. Similarly to yield from the return value of the inner native coroutine is also captured in the StopIteration.

This is the first important step in understanding asyncio in Python. The coroutines declared with the async/await syntax are just a layer of syntactic sugar above generators and yield from.

The asyncio library

By itself, a coroutine has no concept of yielding control to another coroutine. It can only yield control to the caller at the bottom of a coroutine stack. This caller can then switch to another coroutine and run it. Event loops are responsible for managing and running these coroutines. When a coroutine yields, the event loop decides which coroutine to run next.

asyncio came to the Python standard library around the same time async/await was introduced (see PEP 3156). It provides an event loop and allows running coroutines on its event loop using tasks. Let’s examine these tasks and explore how the event loop executes and schedules them.

From futures to tasks

Before we delve into tasks, let’s first touch upon asyncio.Future objects in Python. Futures simply represent the eventual result of an operation. It is the responsibility of the underlying operation to complete the future. A future is done whenever it has a result or an exception. Scheduling callbacks to run when the future completes is possible via the add_done_callback() method. Accessing the result of the future is done by calling result() on the future. If the future has a result set, this method will return the result value. If it has an exception set, it will raise the exception.

Futures by themself have little to do with the event loop. As we’ll see later, they’re primarily used to bridge low-level callback-based code with high-level async/await code. For scheduling coroutines on the event loop, we use more specialized future objects called tasks.

A task is a future that wraps around a coroutine and is scheduled on the event loop. It keeps track of the operation’s result, but unlike futures, which can handle any operation, tasks are specifically designed to handle coroutines. While futures are completed by their underlying operations, tasks are driven to completion by the event loop. Specifically, the event loop calls the __step() method of the task, which advances the coroutine until control is yielded back to the event loop. We’ll explore this process in more detail in the next section.

The basics of the event loop

The asyncio event loop is an infinite while loop that keeps running until it is stopped.

while True:
    self._run_once()
    if self._stopping:
        break

The event loop doesn’t directly handle futures or tasks. Instead, it models its work using callbacks. It maintains two queues: a “ready queue” for callbacks that are ready to run, and a “scheduled queue” for callbacks that will be ready at a future time. During each iteration, the event loop moves relevant callbacks from the scheduled queue to the ready queue and executes all callbacks in the ready queue.

To schedule a coroutine on the event loop, you can use the loop.create_task() method. This method creates the task, which in turn schedules the task’s __step() method via loop.call_soon(). This call_soon() method adds the task’s __step() method to the loop’s ready queue. In the next iteration, the event loop will execute the __step() method, which advances the task’s coroutine by calling send(None) on it. Here’s what the task’s code looks like at this point:

# Simplified task code for the happy flow
class Task(Future):
    def __init__(self, coro, loop):
        self._coro = coro
        self._loop = loop
        self._loop.call_soon(self.__step)
    
    def __step(self):
        coro = self._coro
        result = coro.send(None)
        ...

The send(None) traverses any await (as an await is nothing more than a yield from) until a yield statement is encountered or a StopIteration exception is raised. Let’s examine these two cases in more detail.

Remember that send(None) is equivalent to calling next() on generators. However, you can’t call next() on native coroutines, as this prevents them from being used as iterators. Nevertheless, it’s still helpful to think of send(None) as next(), since the coroutine advances until the next yield statement—this mental model simplifies the process!

Case 1: A StopIteration is raised

Suppose we have the following setup, where we create a task to let the event loop execute coro_1:

async def coro_2():
    return 1

async def coro_1():
     r = await coro_2()
     return 1 + r
    
loop = asyncio.get_event_loop()
loop.create_task(coro_1())

When the __step() method of the task calls send(None) on the task’s coroutine, the await statement delegates this call to coro_2. Since coro_2 only contains return 1, it raises a StopIteration with the value 1. The await then returns this value. Subsequently, return 1 + r raises another StopIteration with the value 2. This exception bubbles up to the task’s __step() method, where it’s captured and set as the task’s result:

# Simplified task code for the happy flow
# Handles the result of the StopIteration
class Task(Future):
    ...
    
    def __step(self):
        coro = self._coro
        try:
            result = coro.send(None)
        except StopIteration as exc:
            # Task captures the value of the StopIteration
            self.set_result(exc.value)
        ...

Case 2: A yield statement is encountered

There is nothing concurrent about the coroutine’s execution in the case above. The entire coroutine is completed synchronously, no matter how many awaits you include. So how is control ever yielded back to the event loop? The answer is twofold—control can be returned to the event loop by either:

  1. A bare yield statement
  2. Yielding a Future attached to the same event loop as the task

Let’s see how the task’s __step() method handles both cases.

People often have the misconception that control is yielded back to the event loop on every await (as is the case in the Javascript event loop). In Python this is not the case as the await is nothing more than a yield from and thus just delegates its operations to the nested coroutine.

Bare yield

Suppose we have the following scenario with a bare yield in coro_2:

async def coro_2():
    yield
    print("2")

async def coro_1():
    await coro_2()
    print("1")

async def coro_3():
    print("3")
    
loop = asyncio.get_event_loop()
loop.create_task(coro_1())
loop.create_task(coro_3())
# Prints 3, 2, 1

When we create tasks for coro_1 and coro_3, their callbacks are added to the ready queue. The event loop processes callbacks in FIFO order, so coro_1’s __step() callback executes first. This method calls send(None) on coro_2, which returns None due to the bare yield. Upon encountering this None value, __step() reschedules itself by adding itself back to the event loop’s ready queue. The event loop then calls coro_3’s task’s __step() method, printing “3”. Only in the next event loop iteration will coro_1’s task’s __step() method execute again, printing “2” and “1”. Here’s how the task’s code handles bare yields:

# Simplified task code for the happy flow
# Handles the result of the StopIteration and bare yields
class Task(Future):
    ...
    
    def __step(self):
        coro = self._coro
        try:
            result = coro.send(None)
        except StopIteration as exc:
            # Task captures the value of the StopIteration
            self.set_result(exc.value)
        else:
            if result is None:
                # Bare yield relinquishes control for one
                # event loop iteration.
                self._loop.call_soon(self.__step)

A bare yield relinquishes control to the event loop for one iteration. Only in the next iteration of the event loop, will the task continue executing.

Note that the example used in this section is purely for illustration and is actually a SyntaxError. You cannot use yield or yield from expressions in an async function. This prevents confusion between native coroutines and generators. However, if you want to mimic a bare yield, you can await the following BareYield instead:

class BareYield(object):
    def __await__(self):
        yield

Yielding a Future attached to the same event loop as the task

As the Future is already attached to the same event loop we can already expect it be executed at some point in the future. Therefore the task’s __step() method is executed whenever this future completes. This is done via the add_done_callback() method of the future. Yielding a future yields back control the the event loop, and only when the future completes will the task continue executing.

This also explains what happens when you await a future:

class Future:
    ...

    def __await__(self):
        if not self.done():
            # This tells parent task to susped execution until the future
            # is completed.
            yield self
        return self.result()

It yields itself such that the task of the coroutine awaiting the future continues to progress once the future is completed.

Finally, this is what the task’s code looks like when handling StopIterations, bare yields, and yielding futures. These represent the happy paths in the task’s execution. If you’re curious about how tasks are cancelled, how exceptions are handled or how contexts are managed, I’d recommend taking a look at the task’s source code yourself.

# Simplified task code for the happy flow
# Handles the result of the StopIteration, bare yields and yielding futures
class Task(Future):
    ...
    
    def __step(self):
        coro = self._coro
        try:
            result = coro.send(None)
        except StopIteration as exc:
            # Task captures the value of the StopIteration
            self.set_result(exc.value)
        else:
            if result is None:
                # Bare yield relinquishes control for one
                # event loop iteration.
                self._loop.call_soon(self.__step)
            elif isinstance(result, Future):
                # Yielding future relinquishes control to the
                # event loop. Task progresses again 
                # when the future completes.
                result.add_done_callback(
                    self.__step
                )

The scheduled queue

The asyncio event loop also allows to schedule a callback at a given time in the future via the call_later and call_at methods.

def call_later(self, delay, callback, *args, context=None):
    ...

def call_at(self, when, callback, *args, context=None):
    ...

These methods add their callbacks to the event loop’s scheduled queue, along with each callback’s desired execution time. During each event loop iteration, the event loop checks its scheduled queue. If it notices that a callback’s desired execution time has passed, it moves that callback to the ready queue.

Note that the event loop can be blocked by another callback for quite some time and that the event loop therefore executes the scheduled callbacks on a best-effort basis.

I/O multiplexing

We’ve covered the basics of the event loop, including how control returns to it and which queues it maintains internally. However, we haven’t yet explored how control is yielded back to the event loop during blocking operations such as I/O.

Consider network I/O as an example. Socket methods such as send, recv, connect and accept block by default. This means if we send data over a socket, the event loop blocks until the call completes, preventing other tasks from executing. While you can set a socket to non-blocking—allowing these methods to return immediately without completing their action—this raises some questions: How do we determine when these operations are finished? How can we identify which sockets are ready for reading and writing? This same issue applies to other I/O operations, such as file handling.

The solution lies in using an I/O multiplexing mechanism provided by the operating system. Most operating systems offer a way to monitor multiple file descriptors simultaneously to check if I/O is possible on any of them. Python’s selectors module checks which mechanisms are available and automatically uses the most efficient one for the system.

The asyncio event loop leverages this module to provide asynchronous I/O. It offers methods such as loop.sock_recv() and loop.sock_sendall() for performing asynchronous I/O operations. Popular libraries such as httpx and aiohttp build upon these methods to create their own asynchronous APIs.

Let’s examine how the event loop handles a HTTP GET request to www.example.com using the loop’s sock_sendall() method:

async def sendall_coro(sock):
    loop = asyncio.get_event_loop()
    await loop.sock_sendall(sock, b"GET / HTTP/1.1\\r\\nHost:www.example.com\\r\\n\\r\\n")
    
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(("www.example.com", 80))
sock.setblocking(False)

loop = asyncio.get_event_loop()
loop.create_task(sendall_coro(sock))

The event loop executes the __step() method of the task in its ready queue, delegating send(None) to the loop’s sock_sendall() method. This method first attempts to send data over the socket to check if it’s ready for writing.

# n is the amount of bytes we managed to send
try:
    n = sock.send(data)
except (BlockingIOError, InterruptedError):
    n = 0

If the socket isn’t ready yet, we need to monitor when it becomes writable. This is where the selectors module comes into play. It allows us to choose which events we’re interested in (EVENT_READ/EVENT_WRITE) and wait for these events to occur. Here’s how we can register to write events for our socket:

data = # any object, doesn't need to be serializable
file_descriptor = sock.fileno()
sel = selectors.DefaultSelector()
sel.register(file_descriptor, selectors.EVENT_WRITE, data)

When registering for an event, we can pass any arbitrary piece of data, including callbacks. After registration, we can make a select() call with a specified timeout and receive the file descriptors, events, and associated data in return:

# List of events that contain 
# file descriptor, event (READ/WRITE) and data
events = sel.select(timeout)

The sock_sendall() method makes use of the selectors module, by creating a future attached to the loop, registering for the EVENT_WRITE selector event and storing a callback. This callback writes bytes to the socket incrementally and completes the future once the entire payload is written. The sock_sendall() method awaits the created future, such that the task resumes execution after the future completes.

In each iteration, the event loop makes a select call to determine which file descriptors are ready for reading or writing. Since we’ve stored a callback for the EVENT_WRITE selector event, the select call will return that callback when the socket is ready for writing. The event loop then moves these callbacks to the ready queue and executes them. Socket reading follows a similar process—we create a future, await it and store a callback for the EVENT_READ event that sets the read bytes as the future’s result.

The complete picture

Now that we’ve explored the various pieces of the event loop, let’s take a step back and have a look at all the steps in a single iteration:

  1. Remove cancelled callbacks from the scheduled queue
  2. Determine timeout value for the select call
    • If there are callbacks in the ready queue or the event loop is stopping, the timeout is 0. Otherwise the timeout is the time until the first callback in the scheduled queue.
  3. Make the select call
    • This processes the read and write events and moves their respective callbacks to the ready queue.
  4. Move all ready callbacks from the scheduled queue to the ready queue
  5. Execute all the callbacks in the ready queue

Conclusion

In this post, we’ve explored exactly what happens when you await a coroutine. We’ve demonstrated that coroutines are not so different from generators, examined the intricacies of the asyncio event loop, and explored the moments when coroutines relinquish control back to the event loop.

As an engineer, I find great value in peeling back the layers of complex systems like asyncio. It’s fascinating to find out how these tools built upon fundamental Python concepts. Having a deep understanding of the systems we use daily is invaluable when debugging or optimizing our code and makes us more well-rounded engineers as a whole. In a next post, I might delve deeper into the Node.js event loop and see how the two compare.

←
What's WSGI anyway
back to top