Ch 2 — Concurrency and Asynchronous Programming: a Detailed OverviewHow Programming Languages Model Asynchronous Program Flow
How Programming Languages Model Asynchronous Program Flow In the previous chapter, we covered asynchronous program flow, concurrency, and parallelism in general terms. In this chapter, we’ll narrow our scope. Specifically, we’ll look into different ways to model and deal with concurrency in programming languages and libraries. It’s important to keep in mind that threads, futures, fibers, goroutines, promises, etc. are abstractions that give us a way to model an asynchronous program flow. They have different strengths and weaknesses, but they share a goal of giving programmers an easy-to-use (and importantly, hard to misuse), efficient, and expressive way of creating a program that handles tasks in a non-sequential, and often unpredictable, order. The lack of precise definitions is prevalent here as well; many terms have a name that stems from a concrete implementation at some point in time but has later taken on a more general meaning that encompasses different implementations and varieties of the same thing. We’ll first go through a way of grouping different abstractions together based on their similarities before we go on to discuss the pros and cons of each of them. We’ll also go through important definitions that we’ll use throughout the book and discuss OS threads in quite some detail. The topics we discuss here are quite abstract and complicated so don’t feel bad if you don’t understand everything immediately. As we progress through the book and you get used to the different terms and techniques by working through some examples, more and more pieces will fall into place. Specifically, the following topics will be covered:
• Definitions • Threads provided by the operating system • Green threads/stackfull coroutines/fibers
26 How Programming Languages Model Asynchronous Program Flow
• Callback based approaches
• Promises, futures, and async/await
Definitions
We can broadly categorize abstractions over concurrent operations into two groups:
1. Cooperative: These are tasks that yield voluntarily either by explicitly yielding or by calling
a function that suspends the task when it can’t progress further before another operation has
finished (such as making a network call). Most often, these tasks yield to a scheduler of some
sort. Examples of this are tasks generated by async/await in Rust and JavaScript.
2. Non-cooperative: Tasks that don’t necessarily yield voluntarily. In such a system, the scheduler
must be able to pre-empt a running task, meaning that the scheduler can stop the task and
take control over the CPU even though the task would have been able to do work and progress.
Examples of this are OS threads and Goroutines (after GO version 1.14).
Figure 2.1 – Non-cooperative vs. cooperative multitasking
Definitions 27
Note In a system where the scheduler can pre-empt running tasks, tasks can also yield voluntarily as they do in a cooperative system, and it’s rare with a system that only relies on pre-emption.
We can further divide these abstractions into two broad categories based on the characteristics of their implementation:
1. Stackful: Each task has its own call stack. This is often implemented as a stack that’s similar to
the stack used by the operating system for its threads. Stackful tasks can suspend execution at
any point in the program as the whole stack is preserved.
2. Stackless: There is not a separate stack for each task; they all run sharing the same call stack. A
task can’t be suspended in the middle of a stack frame, limiting the runtime’s ability to pre-empt
the task. However, they need to store/restore less information when switching between tasks
so they can be more efficient.
There are more nuances to these two categories that you’ll get a deep understanding of when we implement an example of both a stackful coroutine (fiber) and a stackless coroutine (Rust futures generated by async/await) later in the book. For now, we keep the details to a minimum to just provide an overview.
Threads We keep referring to threads all throughout this book, so before we get too far in, let’s stop and give “thread” a good definition since it’s one of those fundamental terms that causes a lot of confusion. In the most general sense, a thread refers to a thread of execution, meaning a set of instructions that need to be executed sequentially. If we tie this back to the first chapter of this book, where we provided several definitions under the Concurrency vs. Parallelism subsection, a thread of execution is similar to what we defined as a task with multiple steps that need resources to progress. The generality of this definition can be a cause of some confusion. A thread to one person can obviously refer to an OS thread, and to another person, it can simply refer to any abstraction that represents a thread of execution on a system. Threads are often divided into two broad categories:
• OS threads: These threads are created by the OS and managed by the OS scheduler. On Linux,
this is known as a kernel thread.
• User-level threads: These threads are created and managed by us as programmers without the
OS knowing about them.
Now, this is where things get a bit tricky: OS threads on most modern operating systems have a lot of similarities. Some of these similarities are dictated by the design of modern CPUs. One example
28 How Programming Languages Model Asynchronous Program Flow
of this is that most CPUs assume that there is a stack it can perform operations on and that it has a
register for the stack pointer and instructions for stack manipulation.
User-level threads can, in their broadest sense, refer to any implementation of a system (runtime) that
creates and schedules tasks, and you can’t make the same assumptions as you do with OS threads.
They can closely resemble OS threads by using separate stacks for each task, as we’ll see in Chapter 5
when we go through our fiber/green threads example, or they can be radically different in nature, as
we’ll see when we go through how Rust models concurrent operations later on in Part 3 of this book.
No matter the definition, a set of tasks needs something that manages them and decides who gets
what resources to progress. The most obvious resource on a computer system that all tasks need to
progress is CPU time. We call the “something” that decides who gets CPU time to progress a scheduler.
Most likely, when someone refers to a “thread” without adding extra context, they refer to an OS
thread/kernel thread, so that’s what we’ll do going forward.
I’ll also keep referring to a thread of execution as simply a task. I find the topic of asynchronous
programming easier to reason about when we limit the use of terms that have different assumptions
associated with them depending on the context as much as possible.
With that out of the way, let’s go through some defining characteristics of OS threads while we also
highlight their limitations.
Important!
Definitions will vary depending on what book or article you read. For example, if you read about
how a specific operating system works, you might see that processes or threads are abstractions
that represent “tasks”, which will seem to contradict the definitions we use here. As I mentioned
earlier, the choice of reference frame is important, and it’s why we take so much care to define
the terms we use thoroughly as we encounter them throughout the book.
The definition of a thread can also vary by operating system, even though most popular systems
share a similar definition today. Most notably, Solaris (pre-Solaris 9, which was released in
2002) used to have a two-level thread system that differentiated between application threads,
lightweight processes, and kernel threads. This was an implementation of what we call M:N
threading, which we’ll get to know more about later in this book. Just beware that if you read
older material, the definition of a thread in such a system might differ significantly from the
one that’s commonly used today.
Now that we’ve gone through the most important definitions for this chapter, it’s time to talk more
about the most popular ways of handling concurrency when programming.
Threads provided by the operating system 29
Threads provided by the operating system
Note! We call this 1:1 threading. Each task is assigned one OS thread. Since this book will not focus specifically on OS threads as a way to handle concurrency going forward, we treat them more thoroughly here.
Let’s start with the obvious. To use threads provided by the operating system, you need, well, an operating system. Before we discuss the use of threads as a means to handle concurrency, we need to be clear about what kind of operating systems we’re talking about since they come in different flavors. Embedded systems are more widespread now than ever before. This kind of hardware might not have the resources for an operating system, and if they do, you might use a radically different kind of operating system tailored to your needs, as the systems tend to be less general purpose and more specialized in nature. Their support for threads, and the characteristics of how they schedule them, might be different from what you’re used to in operating systems such as Windows or Linux. Since covering all the different designs is a book on its own, we’ll limit the scope to talk about treads, as they’re used in Windows and Linux-based systems running on popular desktop and server CPUs. OS threads are simple to implement and simple to use. We simply let the OS take care of everything for us. We do this by spawning a new OS thread for each task we want to accomplish and write code as we normally would. The runtime we use to handle concurrency for us is the operating system itself. In addition to these advantages, you get parallelism for free. However, there are also some drawbacks and complexities resulting from directly managing parallelism and shared resources.
Creating new threads takes time Creating a new OS thread involves some bookkeeping and initialization overhead, so while switching between two existing threads in the same process is pretty fast, creating new ones and discarding ones you don’t use anymore involves work that takes time. All the extra work will limit throughput if a system needs to create and discard a lot of them. This can be a problem if you have huge amounts of small tasks that need to be handled concurrently, which often is the case when dealing with a lot of I/O.
Each thread has its own stack We’ll cover stacks in detail later in this book, but for now, it’s enough to know that they occupy a fixed size of memory. Each OS thread comes with its own stack, and even though many systems allow this size
30 How Programming Languages Model Asynchronous Program Flow
to be configured, they’re still fixed in size and can’t grow or shrink. They are, after all, the cause of stack
overflows, which will be a problem if you configure them to be too small for the tasks you’re running.
If we have many small tasks that only require a little stack space but we reserve much more than we
need, we will occupy large amounts of memory and possibly run out of it.
Context switching
As you now know, threads and schedulers are tightly connected. Context switching happens when the
CPU stops executing one thread and proceeds with another one. Even though this process is highly
optimized, it still involves storing and restoring the register state, which takes time. Every time that
you yield to the OS scheduler, it can choose to schedule a thread from a different process on that CPU.
You see, threads created by these systems belong to a process. When you start a program, it starts a
process, and the process creates at least one initial thread where it executes the program you’ve written.
Each process can spawn multiple threads that share the same address space.
That means that threads within the same process can access shared memory and can access the same
resources, such as files and file handles. One consequence of this is that when the OS switches contexts
by stopping one thread and resuming another within the same process, it doesn’t have to save and
restore all the state associated with that process, just the state that’s specific to that thread.
On the other hand, when the OS switches from a thread associated with one process to a thread
associated with another, the new process will use a different address space, and the OS needs to take
measures to make sure that process “A” doesn’t access data or resources that belong to process “B”. If
it didn’t, the system wouldn’t be secure.
The consequence is that caches might need to be flushed and more state might need to be saved and
restored. In a highly concurrent system under load, these context switches can take extra time and
thereby limit the throughput in a somewhat unpredictable manner if they happen frequently enough.
Scheduling
The OS can schedule tasks differently than you might expect, and every time you yield to the OS, you’re
put in the same queue as all other threads and processes on the system.
Moreover, since there is no guarantee that the thread will resume execution on the same CPU core as it
left off or that two tasks won’t run in parallel and try to access the same data, you need to synchronize
data access to prevent data races and other pitfalls associated with multicore programming.
Rust as a language will help you prevent many of these pitfalls, but synchronizing data access will
require extra work and add to the complexity of such programs. We often say that using OS threads
to handle concurrency gives us parallelism for free, but it isn’t free in terms of added complexity and
the need for proper data access synchronization.
Threads provided by the operating system 31
The advantage of decoupling asynchronous operations from OS threads Decoupling asynchronous operations from the concept of threads has a lot of benefits. First of all, using OS threads as a means to handle concurrency requires us to use what essentially is an OS abstraction to represent our tasks. Having a separate layer of abstraction to represent concurrent tasks gives us the freedom to choose how we want to handle concurrent operations. If we create an abstraction over concurrent operations such as a future in Rust, a promise in JavaScript, or a goroutine in GO, it is up to the runtime implementor to decide how these concurrent tasks are handled. A runtime could simply map each concurrent operation to an OS thread, they could use fibers/green threads or state machines to represent the tasks. The programmer that writes the asynchronous code will not necessarily have to change anything in their code if the underlying implementation changes. In theory, the same asynchronous code could be used to handle concurrent operations on a microcontroller without an OS if there’s just a runtime for it. To sum it up, using threads provided by the operating system to handle concurrency has the following advantages:
• Simple to understand • Easy to use • Switching between tasks is reasonably fast • You get parallelism for free
However, they also have a few drawbacks:
• OS-level threads come with a rather large stack. If you have many tasks waiting simultaneously
(as you would in a web server under heavy load), you’ll run out of memory pretty fast.
• Context switching can be costly and you might get an unpredictable performance since you
let the OS do all the scheduling.
• The OS has many things it needs to handle. It might not switch back to your thread as fast as
you’d wish.
• It is tightly coupled to an OS abstraction. This might not be an option on some systems.
Example Since we’ll not spend more time talking about OS threads in this book, we’ll go through a short example so you can see how they’re used:
32 How Programming Languages Model Asynchronous Program Flow
ch02/aa-os-threads
use std::thread::{self, sleep};
fn main() {
println!("So, we start the program here!");
let t1 = thread::spawn(move || {
sleep(std::time::Duration::from_millis(200));
println!("The long running tasks finish last!");
});
let t2 = thread::spawn(move || {
sleep(std::time::Duration::from_millis(100));
println!("We can chain callbacks...");
let t3 = thread::spawn(move || {
sleep(std::time::Duration::from_millis(50));
println!("...like this!");
});
t3.join().unwrap();
});
println!("The tasks run concurrently!");
t1.join().unwrap();
t2.join().unwrap();
}
In this example, we simply spawn several OS threads and put them to sleep. Sleeping is essentially the
same as yielding to the OS scheduler with a request to be re-scheduled to run after a certain time has
passed. To make sure our main thread doesn’t finish and exit (which will exit the process) before our
children thread has had time to run we join them at the end of our main function.
If we run the example, we’ll see how the operations occur in a different order based on how long we
yielded each thread to the scheduler:
So, we start the program here!
The tasks run concurrently!
We can chain callbacks...
...like this!
The long-running tasks finish last!
So, while using OS threads is great for a number of tasks, we also outlined a number of good reasons
to look at alternatives by discussing their limitations and downsides. The first alternatives we’ll look
at are what we call fibers and green threads.
Fibers and green threads 33
Fibers and green threads
Note! This is an example of M:N threading. Many tasks can run concurrently on one OS thread. Fibers and green threads are often referred to as stackful coroutines.
The name “green threads” originally stems from an early implementation of an M:N threading model used in Java and has since been associated with different implementations of M:N threading. You will encounter different variations of this term, such as “green processes” (used in Erlang), which are different from the ones we discuss here. You’ll also see some that define green threads more broadly than we do here. The way we define green threads in this book makes them synonymous with fibers, so both terms refer to the same thing going forward. The implementation of fibers and green threads implies that there is a runtime with a scheduler that’s responsible for scheduling what task (M) gets time to run on the OS thread (N). There are many more tasks than there are OS threads, and such a system can run perfectly fine using only one OS thread. The latter case is often referred to as M:1 threading. Goroutines is an example of a specific implementation of stackfull coroutines, but it comes with slight nuances. The term “coroutine” usually implies that they’re cooperative in nature, but Goroutines can be pre-empted by the scheduler (at least since version 1.14), thereby landing them in somewhat of a grey area using the categories we present here. Green threads and fibers use the same mechanisms as an OS, setting up a stack for each task, saving the CPU’s state, and jumping from one task(thread) to another by doing a context switch. We yield control to the scheduler (which is a central part of the runtime in such a system), which then continues running a different task. The state of execution is stored in each stack, so in such a solution, there would be no need for async, await, Future, or Pin. In many ways, green threads mimic how an operating system facilitates concurrency, and implementing them is a great learning experience. A runtime using fibers/green threads for concurrent tasks can have a high degree of flexibility. Tasks can, for example, be pre-empted and context switched at any time and at any point in their execution, so a long-running task that hogs the CPU could in theory be pre-empted by the runtime, acting as a safeguard from having tasks that end up blocking the whole system due to an edge-case or a programmer error. This gives the runtime scheduler almost the same capabilities as the OS scheduler, which is one of the biggest advantages of systems using fibers/green threads.
34 How Programming Languages Model Asynchronous Program Flow
The typical flow goes as follows:
• You run some non-blocking code
• You make a blocking call to some external resource
• The CPU jumps to the main thread, which schedules a different thread to run and jumps to
that stack
• You run some non-blocking code on the new thread until a new blocking call or the task is finished
• The CPU jumps back to the main thread, schedules a new thread that is ready to make progress,
and jumps to that thread
Figure 2.2 – Program flow using fibers/green threads
Each stack has a fixed space
As fibers and green threads are similar to OS threads, they do have some of the same drawbacks as
well. Each task is set up with a stack of a fixed size, so you still have to reserve more space than you
actually use. However, these stacks can be growable, meaning that once the stack is full, the runtime
can grow the stack. While this sounds easy, it’s a rather complicated problem to solve.
We can’t simply grow a stack as we grow a tree. What actually needs to happen is one of two things:
1. You allocate a new piece of continuous memory and handle the fact that your stack is spread
over two disjointed memory segments
2. You allocate a new larger stack (for example, twice the size of the previous stack), move all your
data over to the new stack, and continue from there
Fibers and green threads 35
The first solution sounds pretty simple, as you can leave the original stack as it is, and you can basically context switch over to the new stack when needed and continue from there. However, modern CPUs can work extremely fast if they can work on a contiguous piece of memory due to caching and their ability to predict what data your next instructions are going to work on. Spreading the stack over two disjointed pieces of memory will hinder performance. This is especially noticeable when you have a loop that happens to be just at the stack boundary, so you end up making up to two context switches for each iteration of the loop. The second solution solves the problems with the first solution by having the stack as a contiguous piece of memory, but it comes with some problems as well. First, you need to allocate a new stack and move all the data over to the new stack. But what happens with all pointers and references that point to something located on the stack when everything moves to a new location? You guessed it: every pointer and reference to anything located on the stack needs to be updated so they point to the new location. This is complex and time-consuming, but if your runtime already includes a garbage collector, you already have the overhead of keeping track of all your pointers and references anyway, so it might be less of a problem than it would for a non-garbage collected program. However, it does require a great deal of integration between the garbage collector and the runtime to do this every time the stack grows, so implementing this kind of runtime can get very complicated. Secondly, you have to consider what happens if you have a lot of long-running tasks that only require a lot of stack space for a brief period of time (for example, if it involves a lot of recursion at the start of the task) but are mostly I/O bound the rest of the time. You end up growing your stack many times over only for one specific part of that task, and you have to make a decision whether you will accept that the task occupies more space than it needs or at some point move it back to a smaller stack. The impact this will have on your program will of course vary greatly based on the type of work you do, but it’s still something to be aware of.
Context switching Even though these fibers/green threads are lightweight compared to OS threads, you still have to save and restore registers at every context switch. This likely won’t be a problem most of the time, but when compared to alternatives that don’t require context switching, it can be less efficient. Context switching can also be pretty complex to get right, especially if you intend to support many different platforms.
Scheduling When a fiber/green thread yields to the runtime scheduler, the scheduler can simply resume execution on a new task that’s ready to run. This means that you avoid the problem of being put in the same run queue as every other task in the system every time you yield to the scheduler. From the OS perspective, your threads are busy doing work all the time, so it will try to avoid pre-empting them if it can.
36 How Programming Languages Model Asynchronous Program Flow
One unexpected downside of this is that most OS schedulers make sure all threads get some time to
run by giving each OS thread a time slice where it can run before the OS pre-empties the thread and
schedules a new thread on that CPU. A program using many OS threads might be allotted more time
slices than a program with fewer OS threads. A program using M:N threading will most likely only
use a few OS threads (one thread per CPU core seems to be the starting point on most systems). So,
depending on whatever else is running on the system, your program might be allotted fewer time
slices in total than it would be using many OS threads. However, with the number of cores available
on most modern CPUs and the typical workload on concurrent systems, the impact from this should
be minimal.
FFI
Since you create your own stacks that are supposed to grow/shrink under certain conditions and might
have a scheduler that assumes it can pre-empt running tasks at any point, you will have to take extra
measures when you use FFI. Most FFI functions will assume a normal OS-provided C-stack, so it
will most likely be problematic to call an FFI function from a fiber/green thread. You need to notify
the runtime scheduler, context switch to a different OS thread, and have some way of notifying the
scheduler that you’re done and the fiber/green thread can continue. This naturally creates overhead
and added complexity both for the runtime implementor and the user making the FFI call.
Advantages
• It is simple to use for the user. The code will look like it does when using OS threads.
• Context switching is reasonably fast.
• Abundant memory usage is less of a problem when compared to OS threads.
• You are in full control over how tasks are scheduled and if you want you can prioritize them
as you see fit.
• It’s easy to incorporate pre-emption, which can be a powerful feature.
Drawbacks
• Stacks need a way to grow when they run out of space creating additional work and complexity
• You still need to save the CPU state on every context switch
• It’s complicated to implement correctly if you intend to support many platforms and/or
CPU architectures
• FFI can have a lot of overhead and add unexpected complexity
Callback based approaches 37
Callback based approaches
Note! This is another example of M:N threading. Many tasks can run concurrently on one OS thread. Each task consists of a chain of callbacks.
You probably already know what we’re going to talk about in the next paragraphs from JavaScript, which I assume most know. The whole idea behind a callback-based approach is to save a pointer to a set of instructions we want to run later together with whatever state is needed. In Rust, this would be a closure. Implementing callbacks is relatively easy in most languages. They don’t require any context switching or pre-allocated memory for each task. However, representing concurrent operations using callbacks requires you to write the program in a radically different way from the start. Re-writing a program that uses a normal sequential program flow to one using callbacks represents a substantial rewrite, and the same goes the other way. Callback-based concurrency can be hard to reason about and can become very complicated to understand. It’s no coincidence that the term “callback hell” is something most JavaScript developers are familiar with. Since each sub-task must save all the state it needs for later, the memory usage will grow linearly with the number of callbacks in a task.
Advantages
• Easy to implement in most languages • No context switching • Relatively low memory overhead (in most cases)
Drawbacks
• Memory usage grows linearly with the number of callbacks. • Programs and code can be hard to reason about. • It’s a very different way of writing programs and it will affect almost all aspects of the program since all yielding operations require one callback. • Ownership can be hard to reason about. The consequence is that writing callback-based programs without a garbage collector can become very difficult.
38 How Programming Languages Model Asynchronous Program Flow
• Sharing state between tasks is difficult due to the complexity of ownership rules.
• Debugging callbacks can be difficult.
Coroutines: promises and futures
Note!
This is another example of M:N threading. Many tasks can run concurrently on one OS thread.
Each task is represented as a state machine.
Promises in JavaScript and futures in Rust are two different implementations that are based on the
same idea.
There are differences between different implementations, but we’ll not focus on those here. It’s worth
explaining promises a bit since they’re widely known due to their use in JavaScript. Promises also
have a lot in common with Rust’s futures.
First of all, many languages have a concept of promises, but I’ll use the one from JavaScript in the
following examples.
Promises are one way to deal with the complexity that comes with a callback-based approach.
Instead of:
setTimer(200, () => {
setTimer(100, () => {
setTimer(50, () => {
console.log("I'm the last one");
});
});
});
We can do:
function timer(ms) {
return new Promise((resolve) => setTimeout(resolve, ms));
}
timer(200)
.then(() => timer(100))
.then(() => timer(50))
.then(() => console.log("I'm the last one"));
Coroutines: promises and futures 39
The latter approach is also referred to as the continuation-passing style. Each subtask calls a new one once it’s finished. The difference between callbacks and promises is even more substantial under the hood. You see, promises return a state machine that can be in one of three states: pending, fulfilled, or rejected. When we call timer(200) in the previous example, we get back a promise in the pending state. Now, the continuation-passing style does fix some of the issues related to callbacks, but it still retains a lot of them when it comes to complexity and the different ways of writing programs. However, they enable us to leverage the compiler to solve a lot of these problems, which we’ll discuss in the next paragraph.
Coroutines and async/await Coroutines come in two flavors: asymmetric and symmetric. Asymmetric coroutines yields to a scheduler, and they’re the ones we’ll focus on. Symmetric coroutines yield a specific destination; for example, a different coroutine. While coroutines are a pretty broad concept in general, the introduction of coroutines as objects in programming languages is what really makes this way of handling concurrency rival the ease of use that OS threads and fibers/green threads are known for. You see when you write async in Rust or JavaScript, the compiler re-writes what looks like a normal function call into a future (in the case of Rust) or a promise (in the case of JavaScript). Await, on the other hand, yields control to the runtime scheduler, and the task is suspended until the future/ promise you’re awaiting has finished. This way, we can write programs that handle concurrent operations in almost the same way we write our normal sequential programs. Our JavaScript program can now be written as follows:
async function run() {
await timer(200);
await timer(100);
await timer(50);
console.log("I'm the last one");
}
You can consider the run function as a pausable task consisting of several sub-tasks. On each “await” point, it yields control to the scheduler (in this case, it’s the well-known JavaScript event loop). Once one of the sub-tasks changes state to either fulfilled or rejected, the task is scheduled to continue to the next step.
40 How Programming Languages Model Asynchronous Program Flow
When using Rust, you can see the same transformation happening with the function signature when
you write something such as this:
async fn run() -> () { … }
The function wraps the return object, and instead of returning the type (), it returns a Future with
an output type of ():
Fn run() -> impl Future<Output = ()>
Syntactically, Rust’s futures 0.1 was a lot like the promise example we just showed, and the Rust futures
we use today have a lot in common with how async/await works in JavaScript..
This way of rewriting what look like normal functions and code into something else has a lot of
benefits, but it’s not without its drawbacks.
As with any stackless coroutine implementation, full pre-emption can be hard, or impossible, to
implement. These functions have to yield at specific points, and there is no way to suspend execution
in the middle of a stack frame in contrast to fibers/green threads. Some level of pre-emption is possible
by having the runtime or compiler insert pre-emption points at every function call, for example, but
it’s not the same as being able to pre-empt a task at any point during its execution.
Pre-emption points
Pre-emption points can be thought of as inserting code that calls into the scheduler and asks
it if it wishes to pre-empt the task. These points can be inserted by the compiler or the library
you use before every new function call for example.
Furthermore, you need compiler support to make the most out of it. Languages that have metaprogramming
abilities (such as macros) can emulate much of the same, but this will still not be as seamless as it will
when the compiler is aware of these special async tasks.
Debugging is another area where care must be taken when implementing futures/promises. Since the
code is re-written as state machines (or generators), you won’t have the same stack traces as you do
with normal functions. Usually, you can assume that the caller of a function is what precedes it both
in the stack and in the program flow. For futures and promises, it might be the runtime that calls the
function that progresses the state machine, so there might not be a good backtrace you can use to see
what happened before calling the function that failed. There are ways to work around this, but most
of them will incur some overhead.
Advantages
• You can write code and model programs the same way you normally would
• No context switching
Summary 41
• It can be implemented in a very memory-efficient way • It’s easy to implement for various platforms
Drawbacks
• Pre-emption can be hard, or impossible, to fully implement, as the tasks can’t be stopped in
the middle of a stack frame
• It needs compiler support to leverage its full advantages
• Debugging can be difficult both due to the non-sequential program flow and the limitations
on the information you get from the backtraces.
Summary You’re still here? That’s excellent! Good job on getting through all that background information. I know going through text that describes abstractions and code can be pretty daunting, but I hope you see why it’s so valuable for us to go through these higher-level topics now at the start of the book. We’ll get to the examples soon. I promise! In this chapter, we went through a lot of information on how we can model and handle asynchronous operations in programming languages by using both OS-provided threads and abstractions provided by a programming language or a library. While it’s not an extensive list, we covered some of the most popular and widely used technologies while discussing their advantages and drawbacks. We spent quite some time going in-depth on threads, coroutines, fibers, green threads, and callbacks, so you should have a pretty good idea of what they are and how they’re different from each other. The next chapter will go into detail about how we do system calls and create cross-platform abstractions and what OS-backed event queues such as Epoll, Kqueue, and IOCP really are and why they’re fundamental to most async runtimes you’ll encounter out in the wild.