Parallel computing

Parallel computing divides a computational task among multiple processing elements so parts can execute simultaneously. Its performance depends on how work is divided, coordinated, and recombined.

Task parallelism: A parallel programming approach that runs distinct tasks concurrently, often on different data. Shows how independent operations can run side by side rather than splitting one operation across data.
Concurrency: The structuring of multiple activities so they can make progress during overlapping periods. Parallel computing is one way to execute concurrent activities, using simultaneous processing.
High-performance computing: The use of powerful computing systems to solve computationally demanding problems. Parallel computing is the main way supercomputers combine processing capacity.
Serial computing: A form of computation in which instructions execute sequentially on a single processing stream. It is the baseline against which parallel speedup and overhead are measured.
Data parallelism: A parallel programming approach that applies the same operation simultaneously to multiple data elements. Illustrates splitting a large dataset into pieces processed by the same computation.
Central processing unit: The processor that executes instructions and controls operations in a computer. A CPU may provide multiple cores or instruction units for parallel work.
Message Passing Interface: A standard library specification for communication between processes in parallel programs. MPI enables parallel applications to exchange data across distributed-memory systems.
Distributed computing: Computation performed by independent networked computers that coordinate to complete tasks. It emphasizes autonomous machines and network coordination, while parallel computing emphasizes simultaneous execution.
Synchronization: Coordination that controls the timing or ordering of concurrent operations. Parallel tasks often must wait or coordinate to preserve correctness.
Graphics processing unit: A processor designed to perform many similar operations in parallel, especially for graphics and data workloads. GPUs provide large numbers of processing units for data-parallel computations.

Linked from 28 pages