Concurrency seems to be a topic most programming languages default as an advanced topic that only experienced developers should mingle with. In some other languages and frameworks they try to make the concurrency decision for you and hide it under the hood (Until the thread pools are exhausted for no apparent reason). NodeJS, however, encourages its users to think about concurrency before writing even a single line of code.
There are two types of tasks that today’s hardware has to take some effort and time to get done: intensive computations and read/write to a medium that is not memory. Computations are CPU bound and reads/writes are I/O bound. If we only have one CPU core, only one computation can be run at a given moment. Similarly if there’s only one floppy disk drive we can only execute one I/O operation at a time.
On the most basic level, it doesn’t make sense to stop the user from using the CPU while some I/O operations are executing (If your OS freezes completely every time you download a movie from flash drive to your hard disk, you should probably upgrade your OS by now). Today, PCs are more powerful — They all have a few CPU cores, and enough I/O bandwidth to have hundreds or even thousands of I/O operations at the same time. Despite the progress of hardwares, lots of poorly designed applications still do only one thing at a time. NodeJS is here to make using all those hardware resources as easy as possible.
For many other programming languages, invoking an I/O operation halts the entire process. It has to wait for the result before moving to the next line of the code. During the wait other I/O or CPU resources might just be idle. In NodeJS, the APIs around I/O functions are very different from those traditional APIs. Instead of taking in parameters and spitting out results, I/O functions in NodeJS are famous for not returning anything meaningful. They do ask you to pass in a callback function and promise to invoke it with the results of the I/O operations once its done. And your code will proceed to the next line as if the I/O function call never happened.
If your code keeps running, who’s taking care of the I/O operation? Simple, another thread. But NodeJS is said to be single-threaded! Yes and no. NodeJS is single-threaded for YOUR code (including all those callbacks). Each time you request an I/O operation, NodeJS spawns a new thread for it and hands over the results to your thread via event loop.
This is what non-blocking I/O means: it runs I/O operations in parallel with your code. What happens if your code initialized an I/O operation, then started doing some intensive computations that take time to be done? Well the code that’s designed to take care of the I/O result has to wait after the computations finish.
You’ll get a callback after the sausages are done. But if you can’t pause your online game, they will be sitting there getting cold
Why treat the two types of tasks differently? Why not make executing CPU bound tasks the same way as I/O operations? Because the characteristic of parallel I/O operations versus parallel computations are very different. I/O operations are very parallelizable. A web server can easily handle thousands of network connections at a given moment. CPU bound computation tasks, on the other hand, can only be run in parallel by the number of CPU cores. There’s not much gain to setup non-blocking computations only to have them run in sequential order in the end. Furthermore, if your main code does not rely on the result of the computation right away, maybe you should consider run the computation on a separate process, or even a standalone microservice!
And this distinction is where some traditional languages and frameworks failed to convey to their users. For example, some web server frameworks spin up a new thread for every HTTP request. This is fine when most of the requests are I/O bound tasks. If they are CPU bound, having those many threads will not speed up the responses, and all the threading overhead could actually slow down the whole server.
We have a microservice that takes in HTTP requests for players queuing up a ladder. If there are already players in the queue, the service tries to match two players up for a game. The matching algorithm takes players’ skill levels, their game histories, and wait time in queue as inputs. Once the players waited too long in queue we will try to match them with larger skill gaps. At a first glance, we could drop a few setInterval and invoke the matchmaking algorithm on given time intervals within the NodeJS web server.
The downside of this implementation is that once the matchmaking algorithm is running, the web server basically cannot respond to anything else. Rather than trying to squeezing more performance out of this architecture, we asked ourselves whether the matchmaking computation should be part of the http request/response cycle or not. When we were clear that the answer is a NO we moved the matchmaking code into a separate application.
There’s more to this story. We designed this application so that we can run multiple instances in parallel, and the processes do not have to be aware of each other. A stateless application!
I’ve always liked the motto: Write programs that do one thing and do it well. Along with the rise of microservices, NodeJS’s single-threadedness made us constantly ask ourselves whether the responsibility of a piece of code is too much. With a good separation of responsibilities we run into less and less situations where we have to push our tooling to their extremes.
