GPUs (graphics processing unit) are computer hardware used to handle computer graphics and image processing. Because they are optimised for parallel computations and can process multiple simultaneous computations, they’re good for deep learning work. See CUDA.
The idea is that GPUs are able to make many parallel computations at the expense of thread speed. So compared to a CPU, each GPU thread will be much slower but we can do millions of operations in parallel.
i.e., 4 operations in parallel really fast (CPU), or 1 million operations in parallel really slow (GPU)
Resources
- *Programming Massively Parallel Processors, by David B. Kirk and Wen-mei W. Hwu