How Many Threads Do You Need?

Multithreading is important. But is it important for you and your applications? Often when I discuss this issue with fellow developers, the answer seems to be driven more by their background and less by their need. Developers with a C or C++ background, for instance, seem more likely to use threading extensively, while developers who come from languages such as Visual Basic 6 or Visual FoxPro tend to dismiss threading as a technique that their business applications don’t need.

So how much threading do you really need? The answer might surprise you, but before we can get to the bottom of the issue, let me tell you what multithreading is, how it works, and when it is beneficial (both in today’s world as well as in tomorrow’s scenarios).

The Basics

Multithreading represents the ability for the computer to execute several things at once. Like multitasking really, except multitasking generally refers to the ability to run multiple programs at the same time, while multithreading refers to the ability for a single program to execute multiple “things” at the same time.

Of course, when you look under the hood, you discover that a processor can only execute one thing at a time. Everything that goes beyond this simple technical fact is “smoke and mirrors.” For an operating system to support multithreading (or multitasking for that matter), it must implement some mechanism that allows one task or thread to do a little bit of work, and then quickly switch to the next thread or task. Once that thread has been given some processing time, the OS moves on to the next thread, and so on. This switching happens so frequently and so quickly that the illusion of things happening at the same time works quite well.

But, of course, it remains an illusion. Not only is the processor quite busy doing a single thing at a time, the mechanism that switches from one thread of execution to another introduces some overhead. The entire system would work by far the most efficiently if only a single thread was executing on a processor. Nevertheless, there are significant advantages to running multiple threads and tasks that make it worthwhile to go to all the trouble of thread switching. As a result, Windows routinely runs a very large number of threads. The machine I am writing this article on is currently running 490 threads in 42 processes, despite the fact that all I seem to be running is Microsoft Word.

The Simple Explanation for the Need to Multithread

The need for multithreading ranges from simple necessity to “luxuries” such as enhancing the user’s experience. On the need-side, consider the core Windows operating system and how it executes programs. Without multithreading or multitasking, I would not be able to run Word and open the Windows start menu at the same time. However, that is not an aspect that many developers (besides those working at Microsoft perhaps) worry about. What is more interesting, for instance, is Word’s ability to run the spelling checker in the background while I type my article. I do not have to stop writing to run the spelling checker, nor does the spelling checker interrupt my flow of work in any way. I can simply type away on the main thread, and the verification of my input is performed on a secondary (probably lower priority) thread.

The same applies to practically all applications available today. Often, developers tell me they do not believe they need multithreading because they only write business applications. I have to admit that I do not quite follow their argument. Business application can benefit from multithreading in many ways. For instance, a business application loads data. Whenever data operations may take a while, threading is beneficial, because without threading, the application appears to be hung. Windows flags it as “not responding,” which may result in the user being annoyed (at best) or inappropriately terminating the application (at worst). So which data operations could potentially take a while? All of them! Even very fast data operations may encounter scenarios where the database server is too busy to perform the operation speedily. Or perhaps the server is blocking data access due to another ongoing operation. Furthermore, data processing speed also depends on bandwidth. Unknown components such as Internet connections may result in utterly unpredictable performance results. Modern applications need to be able to handle such situations gracefully.

In non-threaded scenarios, all code executes on the main thread (the user interface thread). This means that all operations need to complete before that thread can continue to handle user interface updates and interactions. If your UI has a button that queries data (directly or through a middle tier) on the same thread, then nothing in the UI can happen until that the application completes the operation. If it takes 30 seconds to run the query, then the user is completely stuck for 30 seconds. The user does not even have the option to terminate the operation.

You can imagine a similar scenario when a user launches a browser that loads a Web page. While this happens, the user can move and resize the browser, or with browsers like Internet Explorer 7, even open a new tab and interact with a different page, while the other page is still loading. And once the browser loads some of the information, it starts to render the top of the page, which allows the user to start reading and interacting with it even before it is completely loaded. And if it takes too long to load the page, the user can simply decide to close the tab, or to navigate elsewhere. If loading were to happen on the main thread, however, this would not be possible. Instead, the user would always have to wait for the loading to complete before any interaction or any rendering would be possible. Imagine having to wait for a timeout to occur before you can fix an incorrectly typed URL. Imagine having to wait for a large page to load completely before you can navigate elsewhere after having realized you went to the wrong page. Imagine not being able to close your favorite sports site until all the video clips have loaded, when your boss unexpectedly walks in on you.

The same problems apply to database applications. You do not want your user to have to wait for a very large query to execute. They may have run it by accident, or they may not want to wait after they realize how long it is taking. Maybe they want to interact with other parts of your application while data loads in the background. Or maybe it is simply undesirable for people to terminate your application through the task manager because they think it has crashed, when it fact it is simply working too hard to notify Windows that it is still alive.

There are many other scenarios where multithreading is beneficial in database applications. One of my developers once told me that he thought multithreading is not useful in database applications. At that time, it was the developer’s task to implement an auto-complete feature in some of the textboxes of an application. His application performed the auto-complete functionality on the UI thread. This meant that whenever the user hit a key, some code would run to try to find possible suggestions for the input provided so far. Depending on how much work had to be done, this resulted in a more or less noticeable delay in between keystrokes and made for a very bad user experience. The developer eventually agreed to give multithreading a try, and after having realized that multithreading is a quite approachable technology in .NET, became an immediate convert. Using multithreading, the user experience and the user’s perception of the system’s performance was drastically improved.

But the story doesn’t end here. Moving forward, multithreading is not just a nifty feature for those who want to write professional applications, but it turns into an absolute necessity.

Multi-Processor Systems

A single processor can only execute one thing at a time. Multiple processors, on the other hand, can do multiple things in parallel. Therefore, multiprocessor systems can execute multiple threads at the same time, which moves us beyond a simple illusion into areas with true performance gains. But aren’t multiprocessor systems still rare? Absolutely not! If you don’t have one of your own, you’ve no doubt heard of systems that have 2, 4, 8, or 16 processors. In fact, dual-core systems have become quite common today, and when someone buys a new computer, it is somewhat likely that it is dual-core. Dual-core systems really are multiprocessor systems, with 2 (or more) processors united on a single chip. And then there is the graphics card, which also has its own processor, the GPU (Graphics Processing Unit).

However, I am also thinking about older systems as multiprocessor. Including the scummy old box that is running Windows 98 that you haven’t used in years. It is likely that this computer only has a single processor. Nevertheless, I consider it a multiprocessor system in many scenarios, especially when it runs business applications. Why is that? Because computers are rarely islands. Today, practically all computers set up in business scenarios are networked. They access data that is likely to be stored on systems like SQL Server. And guess what? When you execute a query against a SQL Server database, more than one processor is involved. A client box initiates the call, but the actual execution of the query happens on the server (and its processor). But here’s the trouble: Even though two processors are involved in the operation, the client machine has to wait until the server is done processing the query and sends back the result. What a waste! Even though two processors are involved in the operation, only one of them is used at a time! Using multithreading, you could achieve a drastic improvement in performance and overall user experience. With single-threading, to the user the application may appear to have crashed.

In a service-oriented world, you no doubt encounter similar scenarios all the time. Whenever you call anything that executes on a different machine, multithreading makes sense, no matter whether you make a call to a Web service, .NET Remoting, WCF, or any other cross-machine call, such as loading a dynamic XML file.

But wait, there is more! As we look into the future, we have to ask ourselves how we should architect tomorrow’s applications in order to work well on the hardware of that time. We know that computers will get faster (Moore’s Law tells us so), and we also know that software running on those computers will be more resource intensive as we implement more and more sophisticated things (experience tells us so). The one thing that is not as obvious is how computers will get faster and how software can take advantage of the better hardware. We know at this point that Moore’s Law will not hold up based on processing power and processor frequency alone. Processors simply won’t keep getting drastically faster as they have in the past. I am not predicting this. This effect is already a reality today. So how then can we keep up with the needs of faster hardware? The answer is “multi-core.” Today’s dual-core systems are already taking this path, but it will not end there. 128-core systems already exist today, even if they are not common yet.

The problem is that a single-threaded application runs no faster on a multi-core system than on a single-core machine. For your application to keep up with this development, it will have to implement multithreading.

Threading for the Masses

So why isn’t everyone using threading already? Well, a lot of people are. But a lot of people are not, mainly because the difficulty (real or perceived) of implementing threading. If you are an unmanaged C++ programmer, threading is hard. However, if you are a .NET developer, then threading is much easier to implement. .NET 1.x introduced a relatively simple but powerful threading model that supports true free-threading. Developers could do this explicitly or through a mechanism know as the “thread pool,” which enables the developer to simply execute a certain method call on another thread. .NET 2.0 offers background worker objects to make threading even easier. These objects can start a certain operation on a secondary thread, and fire an event when the operation is complete, so it is easy for the developer to incorporate the result. This makes it possible to execute operations such as data queries on a secondary thread. When the query is complete, an event fires, which can be used to bind the resulting dataset to a grid on the UI (to name one example).

Nevertheless, the fact remains that creating multithreaded applications is not quite as easy as creating single-threaded applications. One might argue that 90% of the complexity has been eliminated by .NET, but there simply are a few things that present logistical issues that a developer must handle. In the example above, a query that executed on a background worker thread created a dataset. While it is relatively easy to implement such a scenario, it is not as easy as simply retrieving a dataset as a return value from a single-threaded method. Another example is a simple for-loop. Imagine a for loop that runs 100 times, incrementing a property on an object. If two threads run at the same time executing the same loop, involving the same property, neither loop will run 100 times. Instead, each loop is likely to run 50 times. Or perhaps some different number of times, depending on each thread’s priority and the exact timing of the start of the loop in each thread. Both loops together will iterate 100 times exactly, but each loop individually will run some lower number of times.

And to make matters worse, this problem will only occur in some rare cases based on coincidence, making it very hard to diagnose the problem. In fact, if someone attaches a debugger to the process to step through the code, the effect may not occur at all, since the developer is likely to step through one thread at a time. Clearly, developers need better tools. Luckily though, this scenario is not all that likely to occur if your application uses different threads to perform different distinct tasks. For instance, if you use one thread as the UI thread, and another thread to load data, then I’d say it is less likely that the operations could conflict. However, a similar effect can still occur when the UI thread attempts to refresh a grid bound to a dataset at the same time that the background thread is updating the data in the dataset. For these scenarios, there are well-defined and well-understood ways of circumventing the problem. Background threads are never supposed to update the user interface directly. Instead, they are supposed to use the Invoke() method, which all Windows Form UI controls have. Similarly, if you use the background worker component, the problem can not occur at all, since that component is handling all such thread synchronization issues for you.

So really, multithreading is not all that hard anymore. Nevertheless, it has to get simpler yet! I have been part of discussions where developers and architects argue that the next step after object-oriented programming and service-oriented architecture should be thread-oriented programming. I think that’s not that outlandish of an idea. If we need to write multithreaded code so our software runs well on multi-core systems, then our compilers and development environments should make it easy for us to do so. .NET has taken big steps towards this goal, but developers still have to worry about low-level details of threading, rather than focusing on the business issue at hand.

At a recent developers event, I ended up spending a few hours at an airport waiting for a delayed plane together with my friend Jeffrey Richter (co-founder of Wintellect). We had an excellent discussion about how to make multithreading easier for the masses. The background worker object is great, but I would much rather see an implementation scenario where we could write a few lines of code that query data and then assign it as a data source, and the compiler figures out that the first few lines run on the UI thread, the execution of the query should run on the background thread, and the assignment of the result needs to be synchronized back to the main thread. This could happen implicitly in many scenarios, or it could be controlled by the developer in a declarative way. Perhaps certain methods could be flagged with an [Asynchronous()] attribute, for instance, instructing the compiler to use multithreading.

It certainly is conceivable that the compiler or even the runtime can make that distinction and thus create the resulting code appropriately multithreaded. Similarly, compilers, tools, and the Common Language Runtime could do a much better job at finding classic threading issues such as racing (as illustrated in the loop example above) or deadlocks and handle those scenarios in appropriate ways.

The developer should not even have to be aware that multithreading is used in such scenarios. The code should just run better and smoother, resulting in better user experiences and in better use of hardware. Developers no longer have to worry about multitasking and processes anymore; we should be able to get to the same point for multithreading. Sure, some people may want to deal with the low-level details of threading, just like some scenarios may require developers to worry about processes or App Domains. But for the majority of scenarios, multithreading should just work, or at the very least, be extremely simple to use.

Luckily, a lot of very smart people are thinking about this stuff, so we are likely to see drastically simplified threading scenarios in the not too distant future. For the time being however, we need to do a bit of extra work to support multithreading. But if you haven’t done so yet, give it a try! You might find that it is much easier to implement than you think.