In the pipelining chapter, we saw how overlapping the stages of multiple instructions within a single CPU dramatically improves performance without needing a faster clock speed. Parallel processing takes this idea a significant step further, moving beyond overlapping stages of a single processor and instead using multiple processing units, working genuinely at the same time, to execute multiple instructions or even multiple entire programs simultaneously.
As CPU clock speeds have approached certain physical and thermal limits over the years, simply making a single processor run faster has become increasingly difficult. Parallel processing offers an alternative path to improved performance, not by making one processor faster, but by using several processors, or several processing cores within a single chip, to share the overall workload between them.
In this tutorial, you will learn about multiprocessing and multicore systems, explore Flynn's classification, a widely used framework for categorizing different types of parallel computer architectures, and understand the practical performance benefits and challenges that come with running computations in parallel.
Parallel processing refers to the simultaneous use of multiple processing units to solve a computational problem, dividing the overall workload into smaller pieces that can be worked on at the same time, rather than one after another in strict sequence. This stands in contrast to the traditional single-processor systems we have studied throughout most of this series, where instructions are ultimately processed one after another, even with the overlapping benefits pipelining provides.
CS Engineering Gyan's simulated system needs to process analytics for four different videos Sequential processing (single processor): Process video 1's analytics, then video 2's, then video 3's, then video 4's, one after another Parallel processing (four processors): Processor 1 handles video 1's analytics Processor 2 handles video 2's analytics Processor 3 handles video 3's analytics Processor 4 handles video 4's analytics All four run simultaneously, finishing in roughly the time it would take to process just one video sequentially
Multiprocessing refers to a computer system that contains two or more separate, complete processors, all sharing access to the same main memory and working together to execute programs. Each processor in a multiprocessing system is capable of independently fetching, decoding, and executing its own instructions, effectively giving the system multiple independent instruction cycles running at once.
A multiprocessing server handles requests from CS Engineering Gyan's website Processor A handles an incoming request to load the C++ notes page Processor B simultaneously handles a completely separate request to load the DBMS notes page Both requests are processed at the same time, rather than one waiting for the other to finish first
Multiprocessing systems can generally be categorized as either symmetric, where all processors share equal access to memory and are treated identically by the operating system, or asymmetric, where certain processors are assigned specific, specialized roles rather than being treated interchangeably.
A multicore system takes the idea of multiprocessing and places multiple independent processing cores directly onto a single physical CPU chip, rather than using entirely separate physical processors. Each core within a multicore CPU is capable of independently fetching, decoding, and executing instructions, functioning much like a complete processor of its own, while still sharing certain resources, such as cache memory, with the other cores on the same chip.
A quad-core CPU inside a laptop used for editing CS Engineering Gyan's video content Core 1 handles video rendering calculations Core 2 handles audio processing for the same video Core 3 handles background file saving operations Core 4 remains available for other running applications, such as a web browser open alongside the video editor All four cores operate simultaneously within the same physical CPU chip
Multicore systems have become extremely common in modern computing devices, from smartphones to laptops to servers, since placing multiple cores on a single chip is generally more practical and power-efficient than using entirely separate physical processors for every system.
Flynn's classification is a widely used framework for categorizing computer architectures based on two factors: how many instruction streams the system processes, and how many data streams it works on at the same time. This framework identifies four broad categories, each describing a different relationship between instructions and data.
| Category | Full Name | Description |
|---|---|---|
| SISD | Single Instruction, Single Data | A traditional single-processor system executing one instruction on one data item at a time |
| SIMD | Single Instruction, Multiple Data | The same single instruction is applied simultaneously to multiple different data items at once |
| MISD | Multiple Instruction, Single Data | Multiple different instructions operate on the exact same single data item at once |
| MIMD | Multiple Instruction, Multiple Data | Multiple independent instructions operate on multiple different data items simultaneously |
CS Engineering Gyan's simulated system needs to increase the brightness of every pixel in a video frame by the same fixed amount SIMD approach: The exact same "add brightness value" instruction is applied simultaneously across many pixels at once, rather than processing each individual pixel with its own separate instruction This is exactly the kind of repetitive, uniform operation that SIMD architectures, commonly found in graphics processors, are specifically designed to handle efficiently
A multicore CPU processing several unrelated tasks Core 1 executes instructions related to video rendering Core 2 executes entirely different instructions related to network communication Each core works with its own distinct instructions and its own distinct data, completely independently of the other This independent, flexible arrangement is a clear example of the MIMD category, and describes how most modern multicore and multiprocessor systems actually operate
Of these four categories, SISD represents traditional single-processor systems, while MIMD is by far the most common category found in modern multicore and multiprocessor systems, since it offers the greatest flexibility, allowing each processing unit to work on entirely independent tasks.
Parallel processing improves overall performance by dividing a large workload into smaller, independent pieces that can genuinely be worked on at the same time, rather than relying entirely on making a single processor execute instructions faster. This becomes especially valuable for tasks that can naturally be broken down into separate, independent pieces of work, such as processing separate video files, handling multiple simultaneous website requests, or performing the same calculation across a large, uniform dataset.
Not every task benefits equally from parallel processing. Some computations are inherently sequential, meaning each step genuinely depends on the result of the previous step, making it impossible to meaningfully divide the work across multiple processors. Even for tasks that can be divided, coordinating and combining the results from multiple processors introduces its own overhead, meaning parallel processing does not always provide a perfectly proportional speedup relative to the number of processors used.
A calculation where each step depends entirely on the previous step's result Step 1: Calculate value A Step 2: Calculate value B, using value A as an input Step 3: Calculate value C, using value B as an input Since each step genuinely requires the result of the step before it, these calculations cannot be meaningfully split across multiple processors to run simultaneously, since Step 2 has nothing to work with until Step 1 actually finishes
This example highlights an important limitation: parallel processing offers the greatest benefit for tasks that can genuinely be divided into independent pieces, while strictly sequential tasks see little to no benefit from having additional processors available.
| Advantages | Limitations |
|---|---|
| Allows large workloads to be divided and completed significantly faster using multiple processing units. | Strictly sequential tasks see little to no benefit from additional processors. |
| Multicore designs offer improved performance without relying solely on increasing clock speed. | Coordinating work and combining results across multiple processors introduces additional overhead. |
| Flynn's classification provides a clear framework for understanding different parallel architecture designs. | Not every problem can be naturally or efficiently divided into independent, parallel pieces. |
| Mistake | Correct Practice |
|---|---|
| Confusing multiprocessing with multicore systems. | Remember that multiprocessing involves separate physical processors, while multicore places multiple cores on a single chip. |
| Assuming every task automatically becomes faster with more processors available. | Understand that strictly sequential tasks, where each step depends on the previous one, see little benefit from additional processors. |
| Confusing SIMD with MIMD in Flynn's classification. | Remember that SIMD applies one instruction to multiple data items, while MIMD runs multiple independent instructions on multiple data items. |
| Believing parallel processing is simply a faster version of pipelining. | Understand that pipelining overlaps stages within a single processor, while parallel processing uses genuinely separate processing units working simultaneously. |
Parallel processing extends the performance improvements we studied in the pipelining chapter by using genuinely separate processing units, whether through multiprocessing or multicore designs, to work on different parts of a problem simultaneously, rather than overlapping stages within a single processor. Flynn's classification, with its four categories, SISD, SIMD, MISD, and MIMD, provides a clear, structured way to describe exactly how different parallel architectures relate instructions to the data they operate on.
We also saw that parallel processing is not a universal solution, since strictly sequential tasks, where each step genuinely depends on the one before it, cannot be meaningfully divided across multiple processors, and coordinating work across processors always introduces some additional overhead. Understanding both the strengths and limitations of parallel processing completes an important part of the performance-focused story that began with pipelining in the previous chapter.
With parallel processing covered, you have now completed a comprehensive journey through Computer Organization and Architecture, from number systems and logic gates, through registers, the instruction cycle, and the CPU, all the way to memory, buses, interrupts, and finally the performance-oriented topics of pipelining, RISC versus CISC, and parallel processing. Together, these chapters provide a solid foundation for understanding exactly how a computer works at the hardware level, from the smallest logic gate to a complete, modern multicore system.