High-Performance Computing and Parallel Processing | 高性能计算与并行处理

📚 High-Performance Computing and Parallel Processing | 高性能计算与并行处理

High-performance computing (HPC) refers to the use of supercomputers and parallel processing techniques to solve complex computational problems that exceed the capacity of a single standard computer. At its core lies parallel processing — the simultaneous execution of multiple computations using multiple processors, cores, or machines.

高性能计算（HPC）指利用超级计算机和并行处理技术来解决超出单台标准计算机处理能力的复杂计算问题。其核心是并行处理——即使用多个处理器、多核心或多台机器同时执行多个计算任务。

This revision guide breaks down the key concepts, architectures, programming models, and performance laws that appear in computer science examinations. Whether you are preparing for A-level, IB, or university entrance tests, mastering these fundamentals will give you a clear edge.

本复习指南系统梳理了计算机考试中涉及的关键概念、体系结构、编程模型与性能定律。无论你正在备考 A-level、IB 还是大学入学考试，扎实掌握这些基础内容都将为你带来显著优势。

1. Why Parallel Processing? | 为什么要并行处理？

For decades, computer performance improved through increasing clock frequencies and transistor density — a trend described by Moore’s law. However, this approach has hit fundamental physical limits. Higher clock speeds generate excessive heat and consume more power, a problem known as the “power wall.” Meanwhile, the “memory wall” refers to the growing gap between CPU speed and memory access speed.

几十年来，计算机性能的提升主要依靠提高时钟频率和晶体管密度——这一趋势由摩尔定律所描述。然而，这一路线已经触及基本的物理极限。更高的时钟频率会产生过多热量并消耗更多功耗，这一问题被称为”功耗墙”。而”内存墙”则指 CPU 速度与内存访问速度之间不断扩大的差距。

Parallel processing offers a way forward: instead of making a single processor faster, we use many processors working together. This approach is more energy-efficient and can scale to massive problem sizes. Modern computers — from smartphones to data centres — are all parallel to some degree.

并行处理提供了一条出路：与其让单个处理器变得更快，不如让多个处理器协同工作。这种方法更节能，并且能够扩展到超大规模的问题。现代计算机——从智能手机到数据中心——都在一定程度上采用了并行处理。

2. Levels of Parallelism | 并行的层次

Parallelism can be exploited at multiple levels within a computer system. Understanding these levels is essential for exam questions that ask you to identify where parallelism occurs.

并行性可以在计算机系统的多个层次上加以利用。理解这些层次对于考试中要求你判断并行发生在何处的题目至关重要。

Bit-level parallelism: Increasing the word size reduces the number of instructions needed to process large data. For example, a 64-bit processor can process 64 bits of data in one operation, whereas a 16-bit processor would need four operations.
Instruction-level parallelism (ILP): A single processor executes multiple instructions simultaneously through techniques such as pipelining, superscalar execution, and out-of-order execution.
Data-level parallelism (DLP): The same operation is applied to many data elements simultaneously. Vector processors and GPUs excel at this type of parallelism.
Task-level parallelism (TLP): Different tasks or threads run on different processors or cores, each performing independent work that contributes to the overall result.

位级并行：增大字长可以减少处理大数据所需的指令数量。例如，64 位处理器一次操作可以处理 64 位数据，而 16 位处理器则需要执行 4 次操作。
指令级并行（ILP）：单个处理器通过流水线、超标量执行和乱序执行等技术同时执行多条指令。
数据级并行（DLP）：同一条操作同时应用于多个数据元素。向量处理器和 GPU 尤其擅长此类并行。
任务级并行（TLP）：不同的任务或线程运行在不同的处理器或核心上，各自执行独立的工作并共同贡献于最终结果。

In modern systems, all four levels coexist. A CPU core uses ILP internally, while multiple cores in a chip exploit TLP, and SIMD (Single Instruction, Multiple Data) units within each core exploit DLP.

在现代系统中，这四种层次同时并存。CPU 核心内部利用指令级并行，芯片中的多个核心利用任务级并行，而每个核心内部的 SIMD（单指令多数据）单元则利用数据级并行。

3. Flynn’s Taxonomy | Flynn 分类法

Michael Flynn proposed a classification of parallel computer architectures in 1966 based on two dimensions: the number of instruction streams and the number of data streams. This taxonomy remains a cornerstone of computer architecture exams.

1966 年，迈克尔·弗林提出了基于两个维度——指令流数量和数据流数量——的并行计算机体系结构分类法。这一分类法至今仍是计算机体系结构考试的核心内容。

Classification	Instruction Streams	Data Streams	Example
SISD	One	One	Traditional single-core CPU
SIMD	One	Multiple	GPU, vector processors
MISD	Multiple	One	Fault-tolerant systems (rare)
MIMD	Multiple	Multiple	Multi-core CPUs, clusters

SISD (Single Instruction, Single Data) is the classic von Neumann architecture. SIMD executes one instruction on many data elements simultaneously — ideal for graphics and scientific computing. MISD (Multiple Instruction, Single Data) is rarely implemented in practice but appears in some fault-tolerant and pipeline designs. MIMD is the most common form of parallel processing today, found in multi-core processors, distributed systems, and cloud computing clusters.

SISD（单指令单数据）是经典的冯·诺依曼体系结构。SIMD 在多个数据元素上同时执行一条指令——非常适合图形处理和科学计算。MISD（多指令单数据）在实践中很少实现，但出现在某些容错和流水线设计中。MIMD 是当今最常见的并行处理形式，广泛应用于多核处理器、分布式系统和云计算集群中。

When answering exam questions, remember this key point: GPUs are fundamentally SIMD machines, while modern multi-core CPUs are MIMD machines that also contain SIMD units.

回答考试题目时务必记住关键一点：GPU 本质上是 SIMD 机器，而现代多核 CPU 是同时包含 SIMD 单元的 MIMD 机器。

4. Shared-Memory vs Distributed-Memory | 共享内存 vs 分布式内存

Parallel systems are also classified by how memory is organised. This distinction determines which programming model can be used and influences scalability and cost.

并行系统还根据内存的组织方式进行分类。这一区别决定了可以使用哪种编程模型，并影响系统的扩展性和成本。

Shared-memory systems feature multiple processors accessing a single, globally accessible memory space via a bus or interconnect. All processors can read and write the same variables, which simplifies programming. However, they suffer from contention and cache-coherence problems, and are limited in scalability. Multi-core CPUs are a classic example.

共享内存系统的特点是多个处理器通过总线或互连网络访问一个全局可访问的统一内存空间。所有处理器都可以读写相同的变量，这简化了编程。然而，它们面临争用和缓存一致性问题，并且可扩展性受限。多核 CPU 是典型例子。

Distributed-memory systems connect independent nodes via a network, and each node has its own private memory. Processors communicate explicitly by passing messages. This approach scales to thousands or millions of cores, but programming is more complex. Supercomputers and clusters use this model.

分布式内存系统通过网络连接多个独立节点，每个节点拥有自己的私有内存。处理器之间通过显式传递消息进行通信。这种方法可以扩展到数千甚至数百万个核心，但编程更加复杂。超级计算机和集群采用这种模型。

A hybrid design — clusters of shared-memory nodes — is increasingly common. Each node internally uses shared memory, while nodes communicate via message passing.

混合设计——由共享内存节点组成的集群——正变得越来越普遍。每个节点内部使用共享内存，而节点之间通过消息传递进行通信。

5. Parallel Programming Models: OpenMP and MPI | 并行编程模型：OpenMP 与 MPI

Two programming models dominate parallel computing: OpenMP for shared-memory systems and MPI (Message Passing Interface) for distributed-memory systems. Exam questions frequently ask you to compare them.

两种编程模型主导着并行计算领域：OpenMP 用于共享内存系统，MPI（消息传递接口）用于分布式内存系统。考试题目经常要求你比较这两种模型。

OpenMP is a set of compiler directives, libraries, and environment variables that extend C, C++, or Fortran. The programmer annotates code regions that should run in parallel, and the runtime system manages threads automatically. For example:

OpenMP 是一组扩展 C、C++ 或 Fortran 的编译器指令、库和环境变量。程序员标注出应并行运行的代码区域，运行时系统自动管理线程。例如：

#pragma omp parallel for
for (i = 0; i < N; i++) { … }

This directive tells the compiler to distribute loop iterations across multiple threads. OpenMP uses a fork-join model: the master thread forks a team of threads, they execute the parallel region, then join back together.

这条指令告诉编译器将循环迭代分配到多个线程上。OpenMP 采用 fork-join 模型：主线程派生出一个线程组，各线程执行并行区域，然后再汇合到一起。

MPI is a message-passing library standard. Processes are independent and communicate by calling functions such as MPI_Send and MPI_Recv. Common collective operations include MPI_Bcast (broadcast) and MPI_Reduce. MPI programs use the Single Program, Multiple Data (SPMD) pattern, where every process runs the same program but on different data.

MPI 是一个消息传递库标准。各进程相互独立，通过调用 MPI_Send 和 MPI_Recv 等函数进行通信。常见的集合操作包括 MPI_Bcast（广播）和 MPI_Reduce（归约）。MPI 程序使用单程序多数据（SPMD）模式，即每个进程运行相同的程序但处理不同的数据。

Feature	OpenMP	MPI
Memory model	Shared memory	Distributed memory
Communication	Implicit (shared variables)	Explicit (message passing)
Ease of use	Easier, incremental	Harder, more control
Scalability	Limited to one machine	Scales across thousands of nodes

In many HPC applications, OpenMP and MPI are combined: OpenMP handles intra-node parallelism while MPI handles inter-node communication.

在许多高性能计算应用中，OpenMP 和 MPI 会结合使用：OpenMP 负责节点内的并行，MPI 负责节点间的通信。

6. GPUs and Vector Processing | GPU 与向量处理

Graphics Processing Units (GPUs) have become the workhorse of high-performance computing. A GPU contains thousands of small, simple cores designed for massive SIMD parallelism. They are particularly effective for matrix operations, deep learning, and scientific simulations.

图形处理单元（GPU）已成为高性能计算的主力军。一个 GPU 包含数千个为大规模 SIMD 并行而设计的小型简单核心。它们在矩阵运算、深度学习和科学模拟中尤其高效。

The GPU programming model uses a host (CPU) and a device (GPU). The CPU transfers data to GPU memory, launches a kernel — a function executed in parallel by many threads — and then copies results back. CUDA (NVIDIA) and OpenCL are the standard programming frameworks.

GPU 编程模型采用主机（CPU）加设备（GPU）的结构。CPU 将数据传输到 GPU 内存，启动一个内核——由大量线程并行执行的函数——然后将结果拷贝回来。CUDA（NVIDIA）和 OpenCL 是标准的编程框架。

Vector processing is a related technique in which a single instruction operates on a vector (an array) of data. A vector processor contains deep pipelines for each arithmetic unit. The Cray-1 (1976) was the first successful vector supercomputer. Modern CPUs include vector extensions such as SSE and AVX.

向量处理是一项相关技术，指单个指令对数据向量（数组）进行操作。向量处理器为每个运算单元配置了深度流水线。Cray-1（1976 年）是第一台成功的向量超级计算机。现代 CPU 包含 SSE 和 AVX 等向量扩展。

Key GPU-specific terms you must know: threads (individual execution units), warps (groups of 32 threads executing in lockstep), and memory hierarchy (global, shared, and local memory).

你必须掌握的关键 GPU 术语：线程（单个执行单元）、线程束（以锁步模式执行的 32 个线程组）和内存层次（全局内存、共享内存和局部内存）。

7. Amdahl’s Law and Speed-up | Amdahl 定律与加速比

Amdahl’s law is the single most important formula in parallel computing. It quantifies the maximum speed-up obtainable when only part of a program can be parallelised.

Amdahl 定律是并行计算中最重要的公式。它量化了当程序中只有一部分可以被并行化时所能获得的最大加速比。

Let P be the fraction of execution time that can be parallelised, and N the number of processors. The speed-up S is given by:

设 P 为可以被并行化的执行时间占比，N 为处理器数量。加速比 S 由下式给出：

S(N) = 1 / ((1 − P) + P/N)

As N approaches infinity, the maximum speed-up converges to:

当 N 趋于无穷大时，最大加速比收敛于：

S(∞) = 1 / (1 − P)

This means that if 90% of a program is parallelisable (P = 0.9), the absolute maximum speed-up is 10×, no matter how many processors you add. The serial portion (1 − P) becomes the bottleneck. This is a classic exam point: doubling processors does not double performance.

这意味着，如果一个程序中 90% 可以被并行化（P = 0.9），那么无论增加多少处理器，绝对最大加速比仅为 10 倍。串行部分（1 − P）成为瓶颈。这是一个经典考点：处理器数量翻倍并不会使性能翻倍。

Efficiency is defined as E = S(N) / N, measuring how well processors are utilised. Perfect linear speed-up gives E = 1 (100%). In practice, efficiency decreases as N grows due to communication overhead and load imbalance.

效率定义为 E = S(N) / N，用于衡量处理器的利用程度。完美的线性加速比对应 E = 1（100%）。实际上，随着 N 增大，通信开销和负载不均会导致效率下降。

8. Designing Parallel Algorithms | 并行算法设计

Designing a parallel algorithm requires a systematic approach. The widely taught framework — PCAM — consists of four stages: Partitioning, Communication, Agglomeration, and Mapping.

设计并行算法需要系统化的方法。广泛使用的框架——PCAM——包含四个阶段：划分、通信、聚合和映射。

Partitioning breaks the problem into smaller tasks and data pieces. Two strategies exist: domain decomposition splits the data, while functional decomposition splits the computation into different tasks.

划分将问题分解为更小的任务和数据块。有两种策略：域分解分割数据，而功能分解将计算拆分为不同的任务。

Communication determines how tasks exchange data. The Goal is to minimise communication, as it is typically far slower than computation. In a stencil computation, for example, each task computes interior values independently but must receive boundary values from neighbouring tasks.

通信确定任务之间如何交换数据。目标是尽量减少通信，因为通信通常远慢于计算。例如，在模板计算中，每个任务独立计算内部值，但必须从相邻任务接收边界值。

Agglomeration combines small tasks into larger ones to reduce communication overhead and improve efficiency. Finally, Mapping assigns tasks to processors, balancing load and minimising inter-processor communication.

聚合将小任务合并为较大的任务，以减少通信开销并提高效率。最后，映射将任务分配到处理器上，实现负载均衡并最小化处理器间通信。

Load balancing is critical: if one processor finishes much earlier than others, it sits idle while the system waits — a waste of resources. Static load balancing assigns fixed distributions, while dynamic load balancing redistributes work at runtime.

负载均衡至关重要：如果一个处理器远早于其他处理器完成工作，它就会空转等待系统，造成资源浪费。静态负载均衡分配固定的任务分布，而动态负载均衡在运行时重新分配工作。

9. Measuring and Improving Performance | 性能测量与优化

Besides speed-up and efficiency, several other metrics help evaluate parallel systems. Scalability measures how well performance improves as both problem size and processor count increase. Gustafson’s law observes that in practice, when more processors are available, scientists tend to solve larger problems rather than the same problem faster.

除了加速比和效率之外，还有其他几个指标可以帮助评估并行系统。可扩展性衡量当问题规模和处理器数量同时增加时性能的提升程度。Gustafson 定律指出，在实际应用中，当更多处理器可用时，科学家倾向于解决更大的问题，而不是更快地解决同一个问题。

Gustafson’s law expresses the scaled speed-up as:

Gustafson 定律将规模化加速比表示为：

S(N) = N − P × (N − 1)

Common performance bottlenecks in parallel programs include communication overhead, idle time from load imbalance, and serial sections that cannot be parallelised. Optimisation techniques include overlapping computation with communication, using non-blocking communication, and reducing synchronisation points.

并行程序中的常见性能瓶颈包括通信开销、负载不均导致的空闲时间以及无法并行化的串行部分。优化技术包括将计算与通信重叠、使用非阻塞通信以及减少同步点。

Profiling tools such as Intel VTune, NVIDIA Nsight, and the open-source tool gprof help identify hot spots — the code regions that consume the most execution time. Remember the optimisation principle: first measure, then optimise.

Intel VTune、NVIDIA Nsight 以及开源工具 gprof 等性能分析工具可以帮助识别热点——即消耗最多执行时间的代码区域。记住优化原则：先测量，再优化。

10. Applications, Challenges, and the Future | 应用、挑战与未来

High-performance computing powers many fields: weather forecasting, molecular dynamics, computational fluid dynamics, quantum chemistry, genome sequencing, financial risk modelling, and deep learning. In each case, the underlying calculations are inherently parallel — thousands of grid points, atoms, or data samples can be processed simultaneously.

高性能计算推动着众多领域的发展：天气预报、分子动力学、计算流体力学、量子化学、基因组测序、金融风险建模和深度学习。在每种情况下，底层计算本质上都是并行的——数千个网格点、原子或数据样本可以同时被处理。

Despite its successes, parallel processing faces persistent challenges. Energy consumption is a major concern for exascale systems. Programming complexity remains high, and fault tolerance becomes harder as systems grow — when one node in a million fails, the entire job may crash. New research explores domain-specific architectures, quantum computing, and neuromorphic chips as alternatives.

尽管取得了巨大成功，并行处理仍面临持续不断的挑战。能耗是百亿亿次（exascale）系统面临的主要问题。编程复杂性依然很高，而随着系统规模扩大，容错性变得更加困难——当百万节点中有一个节点发生故障，整个作业可能崩溃。新的研究正在探索领域专用架构、量子计算和神经形态芯片作为替代方案。

For your exam, remember the hierarchy: transistors → cores → nodes → clusters. And keep in mind that the trend is toward heterogeneous computing — combining CPUs, GPUs, and specialised accelerators to achieve the best performance per watt.

考试时请记住这个层级：晶体管 → 核心 → 节点 → 集群。同时牢记，当前趋势是异构计算——将 CPU、GPU 和专用加速器结合起来，以实现最佳的性能功耗比。

11. Key Exam Points Summary | 考试要点总结

To help you revise efficiently, here is a concise checklist of the most frequently tested concepts in this topic:

为了帮助你高效复习，以下是最常考概念的精简清单：

Flynn’s taxonomy: SISD, SIMD, MISD, MIMD — be able to give examples of each.
Shared memory vs distributed memory: advantages and limitations of each.
OpenMP (thread-based, #pragma) vs MPI (process-based, message passing).
Amdahl’s law: compute maximum speed-up given P; explain why serial sections limit scaling.
Differences between GPUs and CPUs: thousands of cores, SIMD, memory hierarchy.
PCAM design methodology and load balancing.

Flynn 分类法：SISD、SIMD、MISD、MIMD——能够举出每种类型的例子。
共享内存 vs 分布式内存：各自的优点和局限。
OpenMP（基于线程，使用 #pragma）vs MPI（基于进程，使用消息传递）。
Amdahl 定律：给定 P 值计算最大加速比；解释为什么串行部分限制扩展。
GPU 与 CPU 的区别：数千核心、SIMD、内存层次。
PCAM 设计方法论与负载均衡。

Practice this calculation type: if P = 0.75 on 8 processors, Amdahl’s law gives S = 1 / (0.25 + 0.75/8) = 2.91. Notice that efficiency E = 2.91 / 8 ≈ 0.36 — only 36% utilisation, illustrating the cost of the serial portion.

请练习这类计算：若 P = 0.75，处理器数量为 8，Amdahl 定律给出 S = 1 / (0.25 + 0.75/8) = 2.91。注意效率 E = 2.91 / 8 ≈ 0.36——仅 36% 的利用率，这直观地说明了串行部分带来的代价。

High-performance computing and parallel processing are not just exam topics — they are the foundation of modern computing infrastructure. By understanding why parallelism matters, how systems are classified, how programs are written, and how performance is measured, you will be well prepared for both examinations and practical problem-solving.

高性能计算与并行处理不仅是考试主题，更是现代计算基础设施的基石。通过理解并行为何重要、系统如何分类、程序如何编写以及性能如何衡量，你将为考试和实际解决问题做好充分准备。

Published by TutorHao | Computer Science Revision Series | aleveler.com

更多咨询请联系16621398022（同微信）

High-Performance Computing and Parallel Processing | 高性能计算与并行处理