Computer latency – What is the cost of latency

Usually, when we think about improving the performance of a program, our mind wonders towards  the Big O notation. As good as that model may be, there are more aspects of performance that we have to consider. One of the important assumptions that we make when we use the Big O notation, is that all operations are created equal and take the same time to execute. I’m sorry to burst your bubble, but the operations we use are not the same. The time they take to execute varies widely, therefore, if you are aware of the different operation types, you can truly improve the performance of your code.

CPU’s have got faster

This is the evolution of the speed of the average CPU in Millions of operations per second. Source here. You may have seen it, or you may be familiar with Moore’s law. Getting back to the graph above, you might notice that CPU’s speed has been increasing steadily for a while now. Unfortunately however, programs need more than CPUs to run. They also need RAM, storage and most of times a network connection.

For example: While CPUs on average perform 12 times more operations per second in 2017 compared to 2007, the average internet speed for USA has increased merely 6 times (source here). The point I am trying to make here is that CPUs have gotten faster and the rest of the computer has a hard time keeping up.

Overview of computer latency

Here is an overview of the time it takes to do different types of tasks. To make things easier to understand I will turn those into dollars so you have a better understanding of the resources you are spending.

• 1 cpu cycle – 0.4 ns – $1 – a cup of coffee
• Layer 1 cache – 0.9 ns – $2 – cheap menu at McDonald
• Layer 2 cache – 2.8 ns – $7 –  your average stapler
• Layer 3 cache – 28 ns – $70 – Netflix subscription for 6 months
• Reference main memory – 100ns – $250 – a cheap smartphone
• Reading 1 MB from an SSD – 1 ms – $2 500 000 – a really nice house
• Reading 1 MB from an HDD – 10 ms – $25 000 000 – a fortune 500 company s yearly profits
• Send packet San Francisco to Hong Kong – 141 ms – $352 500 000 – a bit more than the GDP of Micronesia in 2017

Another point that might be worth mentioning here is that the further we get from the CPU, the higher the risk that something will go wrong. And so many things can go wrong with a computer programs.

What to do with this information

The purpose of all of this is to give you the broader picture when you design and/or implement your programs. Sometimes, mischievous traps are hidden behind the most benign database call. Be mindful where you spend your resources or you might find yourself running out of budget too soon.