Back in the instruction cycle chapter, we walked through fetch, decode, execute, and store as four separate stages that a single instruction passes through, one after another, before the next instruction's cycle begins. That approach works correctly, but it leaves a surprising amount of the CPU's hardware sitting idle at any given moment, since only one stage is actually doing work while the other three stages wait their turn.
Pipelining is a technique that addresses this inefficiency by allowing multiple instructions to be processed simultaneously, with each instruction sitting at a different stage of the instruction cycle at the same time. Rather than waiting for one instruction to completely finish all four stages before starting the next one, a pipelined CPU can begin fetching a new instruction the very moment the previous instruction moves on to its decode stage.
In this tutorial, you will learn how pipelining works using a stage-by-stage timeline example, understand how pipelining improves overall instruction throughput and speedup, and explore the three major types of pipeline hazards, structural, data, and control hazards, that can complicate and sometimes slow down a pipelined design.
In a simple, non-pipelined CPU, each instruction must completely finish all four stages, fetch, decode, execute, and store, before the next instruction can even begin its fetch stage. This means that while one instruction is being decoded, for example, the fetch hardware sits completely idle, doing no useful work at all.
Non-pipelined execution timeline (each stage takes 1 clock cycle) Instruction 1: Fetch, Decode, Execute, Store (cycles 1-4) Instruction 2: Fetch, Decode, Execute, Store (cycles 5-8) Instruction 3: Fetch, Decode, Execute, Store (cycles 9-12) Total time for 3 instructions: 12 clock cycles
This example clearly shows the inefficiency: even though each individual stage only takes one clock cycle, most of the CPU's stage-specific hardware remains unused during most of this timeline, since only one stage is ever active at any given moment.
Pipelining overlaps the stages of multiple instructions, so that while one instruction is being decoded, the very next instruction can simultaneously begin its fetch stage, using hardware that would otherwise be sitting idle. This overlapping continues across every stage, dramatically improving how efficiently the CPU's hardware is utilized over time.
Pipelined execution timeline (same 4 stages, overlapped) Cycle 1: Instruction 1 - Fetch Cycle 2: Instruction 1 - Decode | Instruction 2 - Fetch Cycle 3: Instruction 1 - Execute | Instruction 2 - Decode | Instruction 3 - Fetch Cycle 4: Instruction 1 - Store | Instruction 2 - Execute | Instruction 3 - Decode Cycle 5: Instruction 2 - Store | Instruction 3 - Execute Cycle 6: Instruction 3 - Store Total time for 3 instructions: 6 clock cycles
Compare this to the non-pipelined example, which took 12 clock cycles for the exact same three instructions. By overlapping the four stages, pipelining completes all three instructions in just 6 clock cycles, a significant improvement achieved simply by keeping every stage of the CPU's hardware busy as much as possible.
Speedup is a common way of measuring exactly how much faster a pipelined design performs compared to an equivalent non-pipelined design, calculated by dividing the non-pipelined execution time by the pipelined execution time.
Non-pipelined time for 3 instructions: 12 clock cycles Pipelined time for the same 3 instructions: 6 clock cycles Speedup = Non-pipelined time / Pipelined time Speedup = 12 / 6 = 2 This means the pipelined design completed the same work twice as fast as the non-pipelined design
As more and more instructions are processed, the speedup achieved by pipelining generally continues to improve, gradually approaching the total number of pipeline stages as an ideal upper limit, assuming everything proceeds smoothly without any disruptions.
The examples so far have assumed that pipelining always proceeds perfectly smoothly, with every instruction flowing cleanly through each stage without any complications. In real CPU designs, however, certain situations can disrupt this smooth flow, forcing the pipeline to pause or delay certain instructions. These disruptive situations are known as pipeline hazards, and they generally fall into three categories: structural, data, and control hazards.
A structural hazard occurs when two or more instructions need to use the exact same piece of hardware at the same time, but that hardware is only capable of handling one request at a time. This kind of conflict forces one of the instructions to wait, temporarily breaking the smooth overlap that pipelining relies on.
CS Engineering Gyan's simulated CPU has only a single shared memory unit used for both fetching instructions and accessing data Instruction 1 needs to access memory during its Store stage Instruction 3 needs to access memory during its Fetch stage, at exactly the same clock cycle Since only one memory access can happen at a time, one of these instructions must be delayed by one clock cycle, creating a temporary pause, often called a pipeline stall
Structural hazards are generally addressed by adding additional, separate hardware resources, such as using entirely separate memory units for instructions and data, reducing the chances of two stages needing the exact same resource simultaneously.
A data hazard occurs when an instruction depends on the result of a previous instruction that has not yet finished producing that result, since both instructions are currently overlapping within the pipeline at the same time.
Instruction 1: ADD R1, R2, R3 (calculates R1 = R2 + R3) Instruction 2: SUB R4, R1, R5 (needs the new value of R1) In a pipeline, Instruction 2 might reach its Decode or Execute stage before Instruction 1 has actually finished storing its updated result into R1 If Instruction 2 uses the old, outdated value of R1 instead of waiting for the correct updated value, it will produce an incorrect result
Data hazards are commonly handled using a technique called forwarding, or bypassing, where the result of an instruction is passed directly to a later instruction that needs it, without waiting for that result to be fully written back through the normal store stage first. In cases where forwarding alone cannot resolve the dependency in time, the pipeline may need to insert a brief stall instead.
A control hazard occurs specifically around branch or jump instructions, since the pipeline typically does not know for certain which instruction should be fetched next until the branch instruction itself has actually been resolved. This uncertainty can force the pipeline to either guess, or wait, before continuing to fetch further instructions.
Instruction 1: CMP R1, R2 (compares R1 and R2) Instruction 2: JUMP IF EQUAL, 5000 (branch depends on the comparison result) While Instruction 2 is still working through the pipeline, the CPU does not yet know for certain whether the jump will actually be taken If the pipeline continues fetching instructions in normal sequence, but the jump ends up being taken, those fetched instructions turn out to be incorrect and must be discarded
Control hazards are commonly addressed using branch prediction, where the CPU makes an educated guess about whether a branch will be taken, allowing it to continue fetching instructions along the predicted path. If the prediction later turns out to be wrong, the incorrectly fetched instructions are discarded, and the pipeline restarts from the correct location instead.
| Hazard Type | Cause | Common Solution |
|---|---|---|
| Structural Hazard | Two instructions need the same hardware resource at once | Adding separate, dedicated hardware resources |
| Data Hazard | An instruction needs a result that isn't ready yet | Forwarding, or inserting a pipeline stall |
| Control Hazard | The pipeline doesn't yet know which instruction to fetch next | Branch prediction |
Nearly every modern CPU relies on pipelining as a fundamental part of its design, since it offers a significant performance improvement without necessarily requiring an increase in raw clock speed. By keeping every stage of the CPU's hardware busy as consistently as possible, pipelining allows far more instructions to be processed within the same amount of time compared to a simple, non-pipelined design.
| Advantages | Limitations |
|---|---|
| Significantly improves overall instruction throughput compared to non-pipelined execution. | Pipeline hazards can introduce stalls that reduce the ideal performance improvement. |
| Keeps CPU hardware resources busy far more consistently than non-pipelined designs. | Handling hazards like data dependencies and branches adds design complexity. |
| Forms the foundation for many further performance improvements in modern CPU design. | Incorrect branch predictions require discarding and restarting fetched instructions. |
| Mistake | Correct Practice |
|---|---|
| Confusing structural hazards with data hazards. | Remember that structural hazards involve competing for shared hardware, while data hazards involve waiting on a needed result. |
| Assuming pipelining always achieves the full theoretical speedup in practice. | Remember that hazards and stalls typically reduce real-world speedup below the ideal maximum. |
| Believing branch prediction always guesses correctly. | Understand that incorrect predictions require discarding fetched instructions and restarting from the correct location. |
| Assuming pipelining changes what each individual instruction does. | Remember that pipelining only changes how instructions are scheduled and overlapped, not the actual operations they perform. |
Pipelining dramatically improves CPU performance by overlapping the fetch, decode, execute, and store stages of multiple instructions, keeping far more of the CPU's hardware busy at any given moment compared to non-pipelined execution. We measured this improvement using speedup, and saw through a direct timeline comparison exactly how much faster a pipelined design can complete the same set of instructions.
We also examined the three major categories of pipeline hazards, structural, data, and control, each disrupting the pipeline's smooth flow in a different way, along with the common techniques, additional hardware resources, forwarding, and branch prediction, used to address them. Together, these concepts explain both why pipelining offers such a significant performance benefit, and why real-world pipelined CPU designs require careful handling of these hazards to achieve that benefit reliably.
With pipelining covered, you are now ready to explore RISC versus CISC architectures, where we will compare two fundamentally different philosophies of instruction set design, and see how each approach relates to the performance concepts covered throughout this series so far.