GPU使用进程的查看和管理

在跑深度学习的时候出现问题

Check failed: error == cudaSuccess (2 vs. 0)  out of memory

修改batch_size的大小错误依旧存在，最后发现是一些进程在占用GPU所导致。关闭这些进程，错误消失。

查看GPU使用情况

nvidia-smi

显示如下：aicrobo@ubuntu:~$ nvidia-smiThu Mar 29 19:52:48 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.48 Driver Version: 367.48 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 106... Off | 0000:01:00.0 On | N/A |
| 32% 41C P2 27W / 120W | 1705MiB / 3012MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 990 C python 61MiB |
| 0 1257 G /usr/bin/X 155MiB |
| 0 27293 C ../../../build/tools/caffe 1273MiB |
| 0 28878 C ../../build/tools/caffe 105MiB |
| 0 29220 C ../../build/tools/caffe 105MiB |
+-----------------------------------------------------------------------------+可以看出当前有６个进程在占用GPU，运行下面的命令关闭相应的进程：kill -9 PID此处的PID为上图中PID列所对应的序列号。
注意：kill -９命令用来强制退出