CAFFE提取特征并可视化

使用CAFFE（ http://caffe.berkeleyvision.org ）运行CNN网络，并提取出特征，将其存储成lmdb以供后续使用，亦可以对其可视化。

使用已训练好的模型进行图像分类

主要步骤

1.在caffe_root下运行./scripts/download_model_binary.py models/bvlc_reference_caffenet获得预训练的CaffeNet。

2.在ipython里（或python，但需要把部分代码注释掉）运行以下代码来加载网络。

./models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Make sure that caffe is on the python path:

caffe_root = '../'  # this file is expected to be in {caffe_root}/examples
import sys
sys.path.insert(0, caffe_root + 'python')

import caffe

plt.rcParams['figure.figsize'] = (10, 10)
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

import os
if not os.path.isfile(caffe_root + 'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel'):
print("Downloading pre-trained CaffeNet model...")
!../scripts/download_model_binary.py ../models/bvlc_reference_caffenet

3.设置网络为测试阶段，并加载网络模型prototxt和数据平均值mean_npy。

./models/bvlc_reference_caffenet/deploy.prototxt
./models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel
./python/caffe/imagenet/ilsvrc_2012_mean.npy

caffe.set_mode_cpu()
net = caffe.Net(caffe_root + 'models/bvlc_reference_caffenet/deploy.prototxt',
caffe_root + 'models/bvlc_reference_caffenet/bvlc_reference_caffenet.caffemodel',
caffe.TEST)

# input preprocessing: 'data' is the name of the input blob == net.inputs[0]

transformer = caffe.io.Transformer({'data': net.blobs['data'].data.shape})
transformer.set_transpose('data', (2,0,1))
transformer.set_mean('data', np.load(caffe_root + 'python/caffe/imagenet/ilsvrc_2012_mean.npy').mean(1).mean(1)) # mean pixel
transformer.set_raw_scale('data', 255)  # the reference model operates on images in [0,255] range instead of [0,1]
transformer.set_channel_swap('data', (2,1,0))  # the reference model has channels in BGR order instead of RGB

4.加载测试图片，并预测分类结果。

./examples/images/cat.jpg

# set net to batch size of 50

net.blobs['data'].reshape(50,3,227,227)

net.blobs['data'].data[...] = transformer.preprocess('data', caffe.io.load_image(caffe_root + 'examples/images/cat.jpg'))
out = net.forward()
print("Predicted class is #{}.".format(out['prob'].argmax()))

5.加载标签，并输出top_k

./data/ilsvrc12/synset_words.txt

# load labels

imagenet_labels_filename = caffe_root + 'data/ilsvrc12/synset_words.txt'
try:
labels = np.loadtxt(imagenet_labels_filename, str, delimiter='\t')
except:
!../data/ilsvrc12/get_ilsvrc_aux.sh
labels = np.loadtxt(imagenet_labels_filename, str, delimiter='\t')

# sort top k predictions from softmax output

top_k = net.blobs['prob'].data[0].flatten().argsort()[-1:-6:-1]
print labels[top_k]

提取特征并可视化

如果提取特征之后不作存储直接可视化的话，可按以下步骤。

1.网络的特征存储在net.blobs，参数和bias存储在net.params，以下代码输出每一层的名称和大小。这里亦可手动把它们存储下来。

[(k, v.data.shape) for k, v in net.blobs.items()]
[(k, v[0].data.shape) for k, v in net.params.items()]

2.可视化，以下是辅助函数

# take an array of shape (n, height, width) or (n, height, width, channels)

# and visualize each (height, width) thing in a grid of size approx. sqrt(n) by sqrt(n)

def vis_square(data, padsize=1, padval=0):
data -= data.min()
data /= data.max()

# force the number of filters to be square
n = int(np.ceil(np.sqrt(data.shape[0])))
padding = ((0, n ** 2 - data.shape[0]), (0, padsize), (0, padsize)) + ((0, 0),) * (data.ndim - 3)
data = np.pad(data, padding, mode='constant', constant_values=(padval, padval))

# tile the filters into an image
data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple(range(4, data.ndim + 1)))
data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])

plt.imshow(data)
plt.show()

根据每一层的名称，选择需要可视化的层，可以可视化filters（参数）和output（特征）

# the parameters are a list of [weights, biases]

filters = net.params['conv1'][0].data
vis_square(filters.transpose(0, 2, 3, 1))

feat = net.blobs['conv1'].data[0, :36]
vis_square(feat, padval=1)

# There are 256 filters, each of which has dimension 5 x 5 x 48. We show only the first 48 filters, with each channel shown separately, so that each filter is a row.

filters = net.params['conv2'][0].data
vis_square(filters[:48].reshape(48**2, 5, 5))

# rectified, only the first 36 of 256 channels

feat = net.blobs['conv2'].data[0, :36]
vis_square(feat, padval=1)

feat = net.blobs['conv3'].data[0]
vis_square(feat, padval=0.5)

feat = net.blobs['conv4'].data[0]
vis_square(feat, padval=0.5)

feat = net.blobs['conv5'].data[0]
vis_square(feat, padval=0.5)

feat = net.blobs['pool5'].data[0]
vis_square(feat, padval=1)

feat = net.blobs['fc6'].data[0]
plt.subplot(2, 1, 1)
plt.plot(feat.flat)
plt.subplot(2, 1, 2)
_ = plt.hist(feat.flat[feat.flat > 0], bins=100)

feat = net.blobs['fc7'].data[0]
plt.subplot(2, 1, 1)
plt.plot(feat.flat)
plt.subplot(2, 1, 2)
_ = plt.hist(feat.flat[feat.flat > 0], bins=100)

feat = net.blobs['prob'].data[0]
plt.plot(feat.flat)