What GPU Cores Do That CPU Cores Cannot: The Architecture

The GPU was born to push pixels. Today it processes data at scales and speeds that have nothing to do with graphics. But the fundamental reason GPUs outperform CPUs on specific tasks is architectural, not accidental. A modern GPU contains thousands of small processors working in parallel on identical operations, while a CPU contains a handful of powerful cores optimized for sequential logic. That difference determines what each is built to do.

The Core Architectural Divide

CPUs and GPUs solve the opposite problem. A CPU excels at conditional logic: if this, then that. It runs program instructions sequentially, with deep pipelines and large caches to minimize stalls when the next instruction depends on the previous one. It predicts branches and speculatively executes code. A GPU abandons all of this. It assumes you will run the same instruction on thousands of data points at once, and it builds an architecture where millions of transistors are devoted to raw compute, not to branching and caching complexity.

NVIDIA’s H100 GPU, released in 2022, contains 16,896 CUDA cores. Each core is simpler than a CPU core; it cannot run an entire program alone. The cores operate in lockstep, so a single instruction executes across hundreds of them simultaneously. A modern CPU like the Intel Xeon has 64 cores across a single chip. The difference in throughput is not marginal.

Parallel Execution at Massive Scale

GPUs achieve performance by doing the same operation billions of times in parallel. Where a CPU would loop through an array in sequence, a GPU assigns each element to a different processor. The speedup depends on how parallelizable the problem is.

Data Parallelism

Matrix multiplication, image filtering, and vector operations are data-parallel: the same computation applies to many independent data points. A GPU divides this work across its cores so all of them compute simultaneously. On the H100, a single instruction can affect 32 floating-point operations per clock cycle. At 2.7 GHz, that amounts to 86 billion floating-point operations per second for this operation type. A CPU of the same era achieves a fraction of that on the same task because it processes elements sequentially.

Sequential Dependencies

If your program is inherently sequential, where B cannot compute until A finishes, a GPU gains little advantage. It will sit idle waiting for dependencies to resolve. CPUs have mechanisms to tolerate such gaps. GPUs do not. A single branch statement can serialize execution across all cores, forcing them to wait while one path executes.

Memory Bandwidth

Raw compute speed means nothing if the GPU cannot feed data to its cores fast enough. Memory bandwidth is the rate at which data moves between processor and RAM. This is where GPUs dominate CPUs most visibly.

The H100 has a memory bandwidth of approximately 3.4 terabytes per second when using HBM3 memory. An Intel Xeon with DDR5 RAM achieves roughly 100 gigabytes per second. The GPU is 34 times faster. This matters because deep learning, graphics rendering, and scientific simulation are all bandwidth-limited. The bottleneck is not compute but data delivery.

A typical deep learning training loop reads billions of parameters from memory into the GPU, computes gradients, and writes updates back. The model lives in the GPU’s memory to keep this pipeline fast. If you stored the model in system RAM and transferred it to the GPU for each operation, you would spend more time moving data than computing.

Specialized Compute Units

Modern GPUs include hardware blocks for operations that AI and graphics demand repeatedly. The H100 includes Tensor Cores, which are specialized circuits for matrix multiplication and convolution. A single Tensor Core can multiply two matrices in one clock cycle under certain conditions. This is not software but hardwired logic.

CPUs lack equivalent structures because they are generalists. They execute whatever program a developer writes. GPUs assume you will do the same operation on many inputs, so they optimize that path into silicon. When the assumed operation matches your actual workload, the gain is enormous. When it does not, the general-purpose cores sit underutilized.

Energy Efficiency Per Operation

A GPU trades responsiveness and flexibility for throughput and energy efficiency on specific workloads. Running a matrix multiplication on a CPU’s general-purpose cores consumes more power per operation than running the same multiplication on a GPU’s tensor cores. The GPU’s architectural specialization leaves little room for wasted transistors.

For training a large language model, the GPU’s efficiency is decisive. A single H100 can perform more FLOPS per watt than any CPU. This directly affects operational costs: a data center training models 24/7 on GPUs spends less on electricity than one using CPUs for the same task. At scale, this compounds into millions of dollars annually.

Real-World Applications and Their GPU Requirements

Deep Learning and Model Training

Deep learning is the most visible use case. Training models like GPT-4 or image generation systems requires thousands of GPUs working in parallel. OpenAI and other labs rent GPU clusters from cloud providers because they cost tens of millions of dollars to build and cannot be shared profitably as physical hardware. An H100 costs between 15,000 and 40,000 dollars depending on quantity and vendor.

Video Encoding and Processing

Video encoding and processing also offload to GPUs. NVIDIA’s NVENC hardware accelerator can encode video in real time at resolutions and bitrates that would require a server room of CPUs. Streaming platforms like Twitch and YouTube use GPU encoding to reduce infrastructure costs.

Graphics Rendering

Graphics rendering is where the GPU started. Modern video games render frames at 60 or 120 Hz by exploiting a GPU’s ability to execute the same shader, a small graphics program, across millions of pixels in parallel. Without a GPU, rendering a game frame would take seconds on a CPU.

When CPUs Outperform GPUs

GPUs excel at throughput; CPUs excel at latency. If you need a result in 10 milliseconds, a CPU with fewer cores but deeper caches and branch prediction often wins. If you can tolerate 100 milliseconds of latency but need to process a billion data points in that window, a GPU is the answer.

Web servers, databases, and interactive applications run on CPUs. They spend most of their time waiting for network requests or disk I/O. A GPU would sit idle. The CPU’s complexity is an asset here: it can jump between many tasks with low overhead, switching to a different operation while waiting for I/O to complete.

The choice between CPU and GPU is not about which is faster. It is about which matches the problem. Understanding the architectural differences makes that choice clear.