项目12:人脸面部关键点检测

近年来,深度学习模型尤其是卷积神经网络,在语义分割、分类、目标检 测等视觉任务上取均得了优秀的成绩。人脸检测与关键点检测与应用是计算机视觉中 两个较为重要的研究问题,其主要目标是从人脸的姿态和面部表情中?取出丰富的信 息。人脸检测是一项分类任务,目的是给定图像后,找出图像中所有的人脸,并将其 标定。人脸关键点检测是一项回归任务,目的是给定人脸图像后,将该图像中所有人 脸的五官以及轮廓的某些位置进行标定。该项任务是很多实际应用场景下的基础工作, 例如美颜技术、人脸分析与表情识别、表情重建等。人脸关键点检测也可称为关键点标定或关键点定位,是给指定的人脸图像的面部 关键点区域进行识别检测,包括面部轮廓、鼻子、眼睛、眉毛、嘴巴等。本章需要学习的内容是kaggle提供的关键点数据集进行关键点模型的训练和预测。

1. 准备工作

1.1 构建项目

在指定的磁盘路径创建存放当前项目的目录,linux或macos可使用mkdir命令创建文件夹目录,Windows直接使用图形化界面右键新建文件夹即可,project12,并且在项目内构建dataset文件夹,用于存放数据集。

    (dlwork) jingyudeMacBook-Pro:~ jingyuyan$ mkdir project12

    (dlwork) jingyudeMacBook-Pro:~ jingyuyan$ cd project12

    (dlwork) jingyudeMacBook-Pro:project12 jingyuyan$ mkdir dataset

1.2 下载和解压数据集

我们可以通过kaggle官网进行数据集的下载https://www.kaggle.com/c/facial-keypoints-detection/data, 鉴于该网站在国外,下载数据集可能不太方便,本文附录提供了下载方法,请读者自行下载,下载完毕放将数据集文件facial-keypoints-detection.zip放入dataset目录下并解压,进入facial-keypoints-detection文件夹中,对test.zip和train.zip文件进行解压,最终得到的目录如下:

project12/
├───demo12.ipynb   
├───dataset/
   ├──facial-keypoints-detection.zip
   └──facial-keypoints-detection/
       ├───IdLookupTable.csv
       ├───SampleSubmission.csv
       ├───test.csv
       └── ....

2. 处理数据集

该数据集由7049张96*96的灰度图像组成。其中每个图像都标有15个关键点的坐标(x,y),改数据集的关键点较为不平整,有的关键点高达7000多个标签可以进行训练,而有的标签只有2000多个。

2.1 对数据集进行预处理

利用pandas的DataFrame对数据集进行预处理后,利用numpy将其转换成ndarray。

import os
import sys
from pandas import DataFrame
from sklearn.utils import shuffle
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
train_cvs = './dataset/facial-keypoints-detection/training.csv'
test_cvs = './dataset/facial-keypoints-detection/test.csv'
lookup_cvs = './dataset/facial-keypoints-detection/IdLookupTable.csv'

定义加载数据集的函数,has_label用于识别训练集和测试集,因为该任务是回归任务,所以我们的测试不选择使用label

def load_data(cvs_file, has_label=True):
    rc = pd.read_csv(cvs_file)
    rc['Image'] = rc['Image'].apply(lambda im: np.fromstring(im, sep=' '))
    rc = rc.dropna()
    x = np.vstack(rc['Image'].values) / 255.
    x = x.astype(np.float32)
    y = None
    if has_label:
        y = rc[rc.columns[:-1]].values
        y = (y - 48) / 48
        x, y = shuffle(x, y, random_state=42) 
        y = y.astype(np.float32)

    return x, y

通过load_data生成训练集

x_img_train, y_label_train = load_data(train_cvs)

查看训练集的各个属性,我们将数据一维化(96*96=9216)后,将有效的2140个训练集合并形成新的集合

x_img_train.shape
(2140, 9216)

查看第一项的y_label_train,会发现输出了30个浮点数,分别代表15个关键点的(x, y)坐标位置。

y_label_train[0], len(y_label_train[0])
(array([ 0.3816111 , -0.21757638, -0.40208334, -0.21338195,  0.21397223,
        -0.20919445,  0.56600696, -0.21338195, -0.20930555, -0.2008125 ,
        -0.5739097 , -0.18404861,  0.167875  , -0.37682638,  0.6707778 ,
        -0.33072916, -0.16739583, -0.37263888, -0.70382637, -0.23852777,
         0.03376389,  0.22246528,  0.4193264 ,  0.5116389 , -0.38531944,
         0.5158264 ,  0.02538195,  0.4403889 ,  0.03376389,  0.8259514 ],
       dtype=float32), 30)

2.1 分析数据集

为了更加清晰的分析数据集,我们先定义plot_img和plot_data函数对数据集进行显示处理。

def plot_img(img, label, axis, c=['c'], s=10, cmap='gray'):
    img = img.reshape(96, 96)
    axis.imshow(img, cmap=cmap) 
    axis.scatter(label[0::2] * 48 + 48, label[1::2] * 48 + 48, marker='o', s=s, c=c)

def plot_data(x, y, begin=0, title=None):
    fig = plt.figure(figsize=(6, 6))
    fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)
    plt.title(title)
    for i in range(16):
        ax = fig.add_subplot(4, 4, i + 1, xticks=[], yticks=[])
        plot_img(x[begin+i], y[begin+i], ax)

    plt.show()

我们显示训练集中,从第60项数据开始往后的16项数据。

plot_data(x_img_train, y_label_train, begin=60)

png

可以看到,改数据集的label数据在数据集的人脸中标注了16个关键点,我们利用DataFrame读取训练集显示出这些关键点原始的数据

rc = pd.read_csv(train_cvs)
rc[:1]
left_eye_center_x left_eye_center_y right_eye_center_x right_eye_center_y left_eye_inner_corner_x left_eye_inner_corner_y left_eye_outer_corner_x left_eye_outer_corner_y right_eye_inner_corner_x right_eye_inner_corner_y ... nose_tip_y mouth_left_corner_x mouth_left_corner_y mouth_right_corner_x mouth_right_corner_y mouth_center_top_lip_x mouth_center_top_lip_y mouth_center_bottom_lip_x mouth_center_bottom_lip_y Image
0 66.033564 39.002274 30.227008 36.421678 59.582075 39.647423 73.130346 39.969997 36.356571 37.389402 ... 57.066803 61.195308 79.970165 28.614496 77.388992 43.312602 72.935459 43.130707 84.485774 238 236 237 238 240 240 239 241 241 243 240 23...

1 rows × 31 columns

从表中可以发现,这些关键点数据都有标注相应的意义,例如left_eye_center_x、right_eye_center_x、right_eye_inner_corner_x等等。

查看数据集的数量

print(rc.count())
left_eye_center_x            7039
left_eye_center_y            7039
right_eye_center_x           7036
right_eye_center_y           7036
left_eye_inner_corner_x      2271
left_eye_inner_corner_y      2271
left_eye_outer_corner_x      2267
left_eye_outer_corner_y      2267
right_eye_inner_corner_x     2268
right_eye_inner_corner_y     2268
right_eye_outer_corner_x     2268
right_eye_outer_corner_y     2268
left_eyebrow_inner_end_x     2270
left_eyebrow_inner_end_y     2270
left_eyebrow_outer_end_x     2225
left_eyebrow_outer_end_y     2225
right_eyebrow_inner_end_x    2270
right_eyebrow_inner_end_y    2270
right_eyebrow_outer_end_x    2236
right_eyebrow_outer_end_y    2236
nose_tip_x                   7049
nose_tip_y                   7049
mouth_left_corner_x          2269
mouth_left_corner_y          2269
mouth_right_corner_x         2270
mouth_right_corner_y         2270
mouth_center_top_lip_x       2275
mouth_center_top_lip_y       2275
mouth_center_bottom_lip_x    7016
mouth_center_bottom_lip_y    7016
Image                        7049
dtype: int64

3. 搭建简单的神经网络进行预测

我们先使用相对简单的神经网络模型进行拟合,尝试少量的参数是否能满足我们预期的效果,这样也比较节约时间,后期我们可以根据训练的结果继续调整模型结构和一些参数,以达到我们的需求。

3.1 搭建模型

该模型比较简单,仅仅只有一个隐藏层即可。

from keras.models import Sequential
from keras.layers import Dense, Activation
from keras.optimizers import SGD

# 设置模型参数和训练参数
# 输出神经元
OUTPUT_NUM = 30
# 模型输入层数量
INPUT_SHAPE = (9216,)
# 验证集划分比例
VALIDATION_SPLIT = 0.2
# 训练周期,这边设置10个周期即可
EPOCHS = 120
# 单批次数据量
BATCH_SIZE = 64
# 训练LOG打印形式
VERBOSE = 1
# 损失函数
LOSS = 'mean_squared_error'
# 训练集
x_img_train, y_label_train = load_data(train_cvs)
Using TensorFlow backend.

我们设置训练120批次,单批次传入数据量为64,验证集按8:2划分,输出的30个神经元表示30个(x, y)坐标。

model = Sequential()
model.add(Dense(100, activation='relu', input_shape=INPUT_SHAPE))
model.add(Dense(OUTPUT_NUM)) 

sgd = SGD(lr=0.01, momentum=0.9, nesterov=True)
model.compile(loss='mean_squared_error', optimizer=sgd) 

model.summary()
Model: "sequential_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 100)               921700    
_________________________________________________________________
dense_2 (Dense)              (None, 30)                3030      
=================================================================
Total params: 924,730
Trainable params: 924,730
Non-trainable params: 0
_________________________________________________________________

构建的模型如下

1

3.2 训练模型

设置好模型参数后开始进行训练,虽然需要训练120轮,但是只有一个隐藏层,参数只有924,730个,时间不会太长。

history = model.fit(train_x, lable_y,
                    batch_size=BATCH_SIZE,
                    epochs=EPOCHS,
                    validation_split=VALIDATION_SPLIT # 保留20%用來驗證
                   )
Train on 1712 samples, validate on 428 samples
Epoch 1/120
1712/1712 [==============================] - 3s 2ms/step - loss: 0.1144 - val_loss: 0.0195
Epoch 2/120
1712/1712 [==============================] - 1s 369us/step - loss: 0.0147 - val_loss: 0.0141
Epoch 3/120
1712/1712 [==============================] - 1s 313us/step - loss: 0.0123 - val_loss: 0.0116
Epoch 4/120
1712/1712 [==============================] - 1s 305us/step - loss: 0.0111 - val_loss: 0.0110
Epoch 5/120
1712/1712 [==============================] - 1s 495us/step - loss: 0.0105 - val_loss: 0.0105
Epoch 6/120
1712/1712 [==============================] - 1s 673us/step - loss: 0.0099 - val_loss: 0.0102
Epoch 7/120
1712/1712 [==============================] - 1s 465us/step - loss: 0.0094 - val_loss: 0.0096
Epoch 8/120
1712/1712 [==============================] - 1s 414us/step - loss: 0.0091 - val_loss: 0.0101
Epoch 9/120
1712/1712 [==============================] - 1s 575us/step - loss: 0.0089 - val_loss: 0.0101
Epoch 10/120
1712/1712 [==============================] - 1s 589us/step - loss: 0.0085 - val_loss: 0.0089
Epoch 11/120
1712/1712 [==============================] - 1s 499us/step - loss: 0.0084 - val_loss: 0.0086
Epoch 12/120
1712/1712 [==============================] - 1s 684us/step - loss: 0.0080 - val_loss: 0.0089
Epoch 13/120
1712/1712 [==============================] - 1s 400us/step - loss: 0.0079 - val_loss: 0.0082
Epoch 14/120
1712/1712 [==============================] - 1s 567us/step - loss: 0.0076 - val_loss: 0.0087
Epoch 15/120
1712/1712 [==============================] - 1s 387us/step - loss: 0.0075 - val_loss: 0.0081
Epoch 16/120
1712/1712 [==============================] - 1s 351us/step - loss: 0.0074 - val_loss: 0.0081
Epoch 17/120
1712/1712 [==============================] - 1s 354us/step - loss: 0.0072 - val_loss: 0.0081
Epoch 18/120
1712/1712 [==============================] - 1s 345us/step - loss: 0.0070 - val_loss: 0.0077
Epoch 19/120
1712/1712 [==============================] - 1s 297us/step - loss: 0.0070 - val_loss: 0.0076
Epoch 20/120
1712/1712 [==============================] - 1s 295us/step - loss: 0.0068 - val_loss: 0.0075
Epoch 21/120
1712/1712 [==============================] - 1s 349us/step - loss: 0.0067 - val_loss: 0.0075
Epoch 22/120
1712/1712 [==============================] - 1s 352us/step - loss: 0.0067 - val_loss: 0.0073
Epoch 23/120
1712/1712 [==============================] - 1s 349us/step - loss: 0.0065 - val_loss: 0.0075
Epoch 24/120
1712/1712 [==============================] - 1s 398us/step - loss: 0.0064 - val_loss: 0.0073
Epoch 25/120
1712/1712 [==============================] - 1s 315us/step - loss: 0.0064 - val_loss: 0.0070
Epoch 26/120
1712/1712 [==============================] - 1s 371us/step - loss: 0.0063 - val_loss: 0.0070
Epoch 27/120
1712/1712 [==============================] - 1s 340us/step - loss: 0.0062 - val_loss: 0.0069
Epoch 28/120
1712/1712 [==============================] - 0s 267us/step - loss: 0.0061 - val_loss: 0.0069
Epoch 29/120
1712/1712 [==============================] - 1s 292us/step - loss: 0.0061 - val_loss: 0.0068
Epoch 30/120
1712/1712 [==============================] - 0s 284us/step - loss: 0.0060 - val_loss: 0.0068
Epoch 31/120
1712/1712 [==============================] - 1s 312us/step - loss: 0.0060 - val_loss: 0.0067
Epoch 32/120
1712/1712 [==============================] - 0s 285us/step - loss: 0.0058 - val_loss: 0.0066
Epoch 33/120
1712/1712 [==============================] - 1s 308us/step - loss: 0.0058 - val_loss: 0.0066
Epoch 34/120
1712/1712 [==============================] - 1s 358us/step - loss: 0.0057 - val_loss: 0.0065
Epoch 35/120
1712/1712 [==============================] - 1s 312us/step - loss: 0.0056 - val_loss: 0.0068
Epoch 36/120
1712/1712 [==============================] - 1s 351us/step - loss: 0.0056 - val_loss: 0.0066
Epoch 37/120
1712/1712 [==============================] - 0s 289us/step - loss: 0.0055 - val_loss: 0.0065
Epoch 38/120
1712/1712 [==============================] - 0s 280us/step - loss: 0.0055 - val_loss: 0.0064
Epoch 39/120
1712/1712 [==============================] - 0s 287us/step - loss: 0.0054 - val_loss: 0.0065
Epoch 40/120
1712/1712 [==============================] - 0s 282us/step - loss: 0.0054 - val_loss: 0.0063
Epoch 41/120
1712/1712 [==============================] - 1s 310us/step - loss: 0.0053 - val_loss: 0.0062
Epoch 42/120
1712/1712 [==============================] - 1s 360us/step - loss: 0.0053 - val_loss: 0.0061
Epoch 43/120
1712/1712 [==============================] - 1s 375us/step - loss: 0.0053 - val_loss: 0.0062
Epoch 44/120
1712/1712 [==============================] - 1s 307us/step - loss: 0.0052 - val_loss: 0.0061
Epoch 45/120
1712/1712 [==============================] - 1s 802us/step - loss: 0.0052 - val_loss: 0.0060
Epoch 46/120
1712/1712 [==============================] - 1s 430us/step - loss: 0.0051 - val_loss: 0.0060
Epoch 47/120
1712/1712 [==============================] - 1s 381us/step - loss: 0.0050 - val_loss: 0.0062
Epoch 48/120
1712/1712 [==============================] - 1s 319us/step - loss: 0.0050 - val_loss: 0.0059
Epoch 49/120
1712/1712 [==============================] - 0s 283us/step - loss: 0.0050 - val_loss: 0.0059
Epoch 50/120
1712/1712 [==============================] - 1s 296us/step - loss: 0.0049 - val_loss: 0.0060
Epoch 51/120
1712/1712 [==============================] - 1s 298us/step - loss: 0.0049 - val_loss: 0.0058
Epoch 52/120
1712/1712 [==============================] - 1s 316us/step - loss: 0.0049 - val_loss: 0.0064
Epoch 53/120
1712/1712 [==============================] - 1s 412us/step - loss: 0.0049 - val_loss: 0.0058
Epoch 54/120
1712/1712 [==============================] - 1s 335us/step - loss: 0.0048 - val_loss: 0.0057
Epoch 55/120
1712/1712 [==============================] - 1s 592us/step - loss: 0.0048 - val_loss: 0.0058
Epoch 56/120
1712/1712 [==============================] - 1s 366us/step - loss: 0.0047 - val_loss: 0.0056
Epoch 57/120
1712/1712 [==============================] - 1s 384us/step - loss: 0.0047 - val_loss: 0.0056
Epoch 58/120
1712/1712 [==============================] - 1s 302us/step - loss: 0.0046 - val_loss: 0.0056
Epoch 59/120
1712/1712 [==============================] - 1s 357us/step - loss: 0.0046 - val_loss: 0.0055
Epoch 60/120
1712/1712 [==============================] - 1s 300us/step - loss: 0.0046 - val_loss: 0.0058
Epoch 61/120
1712/1712 [==============================] - 1s 459us/step - loss: 0.0045 - val_loss: 0.0054
Epoch 62/120
1712/1712 [==============================] - 1s 439us/step - loss: 0.0045 - val_loss: 0.0054
Epoch 63/120
1712/1712 [==============================] - 1s 347us/step - loss: 0.0045 - val_loss: 0.0056
Epoch 64/120
1712/1712 [==============================] - 1s 309us/step - loss: 0.0045 - val_loss: 0.0054
Epoch 65/120
1712/1712 [==============================] - 0s 292us/step - loss: 0.0044 - val_loss: 0.0053
Epoch 66/120
1712/1712 [==============================] - 1s 339us/step - loss: 0.0044 - val_loss: 0.0053
Epoch 67/120
1712/1712 [==============================] - 1s 359us/step - loss: 0.0043 - val_loss: 0.0053
Epoch 68/120
1712/1712 [==============================] - 0s 287us/step - loss: 0.0043 - val_loss: 0.0053
Epoch 69/120
1712/1712 [==============================] - 1s 304us/step - loss: 0.0043 - val_loss: 0.0052
Epoch 70/120
1712/1712 [==============================] - 1s 311us/step - loss: 0.0043 - val_loss: 0.0053
Epoch 71/120
1712/1712 [==============================] - 1s 361us/step - loss: 0.0043 - val_loss: 0.0053
Epoch 72/120
1712/1712 [==============================] - 1s 300us/step - loss: 0.0042 - val_loss: 0.0052
Epoch 73/120
1712/1712 [==============================] - 1s 326us/step - loss: 0.0042 - val_loss: 0.0052
Epoch 74/120
1712/1712 [==============================] - 1s 358us/step - loss: 0.0042 - val_loss: 0.0051
Epoch 75/120
1712/1712 [==============================] - 1s 369us/step - loss: 0.0041 - val_loss: 0.0051
Epoch 76/120
1712/1712 [==============================] - 1s 390us/step - loss: 0.0041 - val_loss: 0.0051
Epoch 77/120
1712/1712 [==============================] - 1s 339us/step - loss: 0.0041 - val_loss: 0.0051
Epoch 78/120
1712/1712 [==============================] - 0s 281us/step - loss: 0.0040 - val_loss: 0.0050
Epoch 79/120
1712/1712 [==============================] - 0s 252us/step - loss: 0.0040 - val_loss: 0.0050
Epoch 80/120
1712/1712 [==============================] - 1s 318us/step - loss: 0.0040 - val_loss: 0.0050
Epoch 81/120
1712/1712 [==============================] - 1s 309us/step - loss: 0.0040 - val_loss: 0.0053
Epoch 82/120
1712/1712 [==============================] - 1s 362us/step - loss: 0.0040 - val_loss: 0.0053
Epoch 83/120
1712/1712 [==============================] - 0s 267us/step - loss: 0.0040 - val_loss: 0.0049
Epoch 84/120
1712/1712 [==============================] - 1s 295us/step - loss: 0.0039 - val_loss: 0.0049
Epoch 85/120
1712/1712 [==============================] - 1s 330us/step - loss: 0.0039 - val_loss: 0.0049
Epoch 86/120
1712/1712 [==============================] - 1s 299us/step - loss: 0.0039 - val_loss: 0.0048
Epoch 87/120
1712/1712 [==============================] - 0s 268us/step - loss: 0.0038 - val_loss: 0.0048
Epoch 88/120
1712/1712 [==============================] - 0s 259us/step - loss: 0.0038 - val_loss: 0.0048
Epoch 89/120
1712/1712 [==============================] - 1s 295us/step - loss: 0.0038 - val_loss: 0.0048
Epoch 90/120
1712/1712 [==============================] - 0s 276us/step - loss: 0.0038 - val_loss: 0.0052
Epoch 91/120
1712/1712 [==============================] - 1s 301us/step - loss: 0.0038 - val_loss: 0.0048
Epoch 92/120
1712/1712 [==============================] - 1s 358us/step - loss: 0.0038 - val_loss: 0.0047
Epoch 93/120
1712/1712 [==============================] - 1s 326us/step - loss: 0.0037 - val_loss: 0.0048
Epoch 94/120
1712/1712 [==============================] - 0s 280us/step - loss: 0.0037 - val_loss: 0.0047
Epoch 95/120
1712/1712 [==============================] - 1s 307us/step - loss: 0.0037 - val_loss: 0.0046
Epoch 96/120
1712/1712 [==============================] - 1s 298us/step - loss: 0.0037 - val_loss: 0.0047
Epoch 97/120
1712/1712 [==============================] - 1s 313us/step - loss: 0.0036 - val_loss: 0.0046
Epoch 98/120
1712/1712 [==============================] - 1s 335us/step - loss: 0.0036 - val_loss: 0.0047
Epoch 99/120
1712/1712 [==============================] - 0s 262us/step - loss: 0.0036 - val_loss: 0.0046
Epoch 100/120
1712/1712 [==============================] - 1s 371us/step - loss: 0.0036 - val_loss: 0.0046
Epoch 101/120
1712/1712 [==============================] - 1s 496us/step - loss: 0.0036 - val_loss: 0.0045
Epoch 102/120
1712/1712 [==============================] - 1s 492us/step - loss: 0.0036 - val_loss: 0.0047
Epoch 103/120
1712/1712 [==============================] - 1s 466us/step - loss: 0.0036 - val_loss: 0.0046
Epoch 104/120
1712/1712 [==============================] - 1s 323us/step - loss: 0.0035 - val_loss: 0.0046
Epoch 105/120
1712/1712 [==============================] - 0s 286us/step - loss: 0.0035 - val_loss: 0.0045
Epoch 106/120
1712/1712 [==============================] - 1s 297us/step - loss: 0.0035 - val_loss: 0.0045
Epoch 107/120
1712/1712 [==============================] - 1s 348us/step - loss: 0.0035 - val_loss: 0.0045
Epoch 108/120
1712/1712 [==============================] - 0s 252us/step - loss: 0.0034 - val_loss: 0.0044
Epoch 109/120
1712/1712 [==============================] - 1s 316us/step - loss: 0.0034 - val_loss: 0.0044
Epoch 110/120
1712/1712 [==============================] - 0s 285us/step - loss: 0.0034 - val_loss: 0.0044
Epoch 111/120
1712/1712 [==============================] - 0s 292us/step - loss: 0.0034 - val_loss: 0.0044
Epoch 112/120
1712/1712 [==============================] - 0s 278us/step - loss: 0.0033 - val_loss: 0.0043
Epoch 113/120
1712/1712 [==============================] - 1s 345us/step - loss: 0.0033 - val_loss: 0.0044
Epoch 114/120
1712/1712 [==============================] - 0s 279us/step - loss: 0.0033 - val_loss: 0.0043
Epoch 115/120
1712/1712 [==============================] - 1s 383us/step - loss: 0.0033 - val_loss: 0.0044
Epoch 116/120
1712/1712 [==============================] - 1s 452us/step - loss: 0.0033 - val_loss: 0.0044
Epoch 117/120
1712/1712 [==============================] - 1s 422us/step - loss: 0.0033 - val_loss: 0.0043
Epoch 118/120
1712/1712 [==============================] - 1s 324us/step - loss: 0.0033 - val_loss: 0.0043
Epoch 119/120
1712/1712 [==============================] - 1s 310us/step - loss: 0.0033 - val_loss: 0.0043
Epoch 120/120
1712/1712 [==============================] - 1s 577us/step - loss: 0.0032 - val_loss: 0.0043

定义绘制函数,绘制出训练结果。由于训练结果数值较小,所以我们设置函数的y轴浮动区间在(0.001, 0.01)之间。

def show_train_history(train_history,train,validation):
    plt.plot(train_history.history[train])
    plt.plot(train_history.history[validation])
    plt.title('Train histoty')
    plt.grid()
    plt.ylabel(train)
    plt.xlabel('Epoch')
    plt.ylim(0.001, 0.01)
    plt.legend(['train','validation',],loc = 'upper left')
    plt.yscale('log')
    name = train + '.png'
    plt.savefig(name)
    plt.show()
show_train_history(history,'loss','val_loss')

png

可以看到,最终的训练误差在0.32左右,验证误差在0.43左右。

3.3 测试模型

对测试集进行预测,并从第60个开始显示,显示16个人脸关键点标定的结果

x_img_test, _ = load_data(test_cvs, has_label=False)
pred_y = model.predict(x_img_test) 
plot_data(x_img_test, y_pred, begin=60)

png

从结果上来看,虽然大体上关键点能标对,但是有些细节部分不是那么精确,例如图上多处嘴巴、眉毛、眼睛等位置的标注出现了偏移的现象。可以多观察几组结果看看。

plot_data(x_img_test, y_pred, begin=400)

png

该模型仅仅只使用一个隐藏层,接下来我们建立更加复杂的模型提高精确度。

3.4 保存模型

养成习惯把训练好的模型存在来,下一节还会用到。

model.save('./model/model.h5')
model.save_weights('./model/model_weights.h5')
model_json = model.to_json()
with open('./model/model.json', 'w') as file:
    file.write(model_json)

4. 搭建更加精确的卷积神经网络模型进行预测

4.1 定义数据扩充方法

对数据进行扩充,利用水平镜像的方法对数据扩充类ImageDataGenerator进行添加新的扩充方法

import numpy as np
import matplotlib.pyplot as plt
from keras.models import Sequential
from keras.optimizers import SGD
from keras.layers import Dense, Dropout, Flatten
from keras.layers import Conv2D, MaxPooling2D
from keras.callbacks import EarlyStopping, LearningRateScheduler
from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import train_test_split
from keras import callbacks
class FlippedImageDataGenerator(ImageDataGenerator):
    flip_indices = [(0, 2), (1, 3), (4, 8), (5, 9),
                    (6, 10), (7, 11), (12, 16), (13, 17),
                    (14, 18), (15, 19), (22, 24), (23, 25)]

    def next(self):
        X_batch, y_batch = super(FlippedImageDataGenerator, self).next()
        batch_size = X_batch.shape[0]
        indices = np.random.choice(batch_size, batch_size / 2, replace=False)
        X_batch[indices] = X_batch[indices, :, :, ::-1]

        if y_batch is not None:
            y_batch[indices, ::2] = y_batch[indices, ::2] * -1

            for a, b in self.flip_indices:
                y_batch[indices, a], y_batch[indices, b] = (y_batch[indices, b], y_batch[indices, a])

        return X_batch, y_batch

4.2 建立模型

建立一个6个卷积层的深度模型,设置训练周期为100,使用数据扩充的方法对数据进行扩充处理。

# 设置模型参数和训练参数
# 输出神经元
OUTPUT_NUM = 30
# 模型输入层数量
INPUT_SHAPE = (96, 96, 1)
# 训练周期,这边设置100个周期即可
EPOCHS = 100
# 训练LOG打印形式
VERBOSE = 1
# 损失函数
LOSS = 'mean_squared_error'
# 训练集
x_img_train, y_label_train = load_data(train_cvs)
x_img_train = x_img_train.reshape(-1, 96, 96, 1)
def Model2():
    model = Sequential()
    model.add(Conv2D(32, (3, 3), 
                     padding='same', activation='relu', 
                     kernel_initializer='he_normal', input_shape=INPUT_SHAPE))

    model.add(Conv2D(32, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))

    model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(64, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))

    model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(128, (3, 3), activation='relu'))
    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.2))

    model.add(Flatten())
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(0.5))

    model.add(Dense(30))

    return model

model2 = Model2()

model2.summary()

sgd = SGD(lr=0.01, momentum=0.9, nesterov=True)
model2.compile(loss=LOSS, optimizer=sgd) 
WARNING: Logging before flag parsing goes to stderr.
W0115 01:13:11.906343 4495103424 deprecation_wrapper.py:119] From /Users/jingyuyan/anaconda3/envs/dlwork/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.



Model: "sequential_2"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 96, 96, 32)        320       
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 94, 94, 32)        9248      
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 47, 47, 32)        0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 47, 47, 32)        0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 47, 47, 64)        18496     
_________________________________________________________________
conv2d_4 (Conv2D)            (None, 45, 45, 64)        36928     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 22, 22, 64)        0         
_________________________________________________________________
dropout_2 (Dropout)          (None, 22, 22, 64)        0         
_________________________________________________________________
conv2d_5 (Conv2D)            (None, 22, 22, 128)       73856     
_________________________________________________________________
conv2d_6 (Conv2D)            (None, 20, 20, 128)       147584    
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 10, 10, 128)       0         
_________________________________________________________________
dropout_3 (Dropout)          (None, 10, 10, 128)       0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 12800)             0         
_________________________________________________________________
dense_3 (Dense)              (None, 128)               1638528   
_________________________________________________________________
dropout_4 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_4 (Dense)              (None, 30)                3870      
=================================================================
Total params: 1,928,830
Trainable params: 1,928,830
Non-trainable params: 0
_________________________________________________________________

构建的模型如图所示:

2

4.3 开始训练

训练过程比较漫长,建议读者这里使用GPU进行训练,训练完毕后记得保存模型权重。使用callbacks.ModelCheckpoint函数设置训练过程中保存最佳效果的模型。

x_train, x_val, y_train, y_val = train_test_split(x_img_train, y_label_train, test_size=0.2, random_state=42)
# 设置保存最佳模型
cbks = [callbacks.ModelCheckpoint("best_model2.h5", save_best_only=True)]
# 设置数据扩充生成器
flipgen = FlippedImageDataGenerator()
# 开始训练
history = model2.fit_generator(flipgen.flow(x_train, y_train),
                               steps_per_epoch=len(x_train),
                               epochs=EPOCHS,
                               callbacks=cbks,
                               validation_data=(x_val, y_val)
                              )
Epoch 1/100
1712/1712 [==============================] - 27s 16ms/step - loss: 0.0108 - val_loss: 0.0046
Epoch 2/100
1712/1712 [==============================] - 26s 15ms/step - loss: 0.0046 - val_loss: 0.0039
Epoch 3/100
1712/1712 [==============================] - 26s 15ms/step - loss: 0.0041 - val_loss: 0.0037
Epoch 4/100
1712/1712 [==============================] - 25s 15ms/step - loss: 0.0039 - val_loss: 0.0034
Epoch 5/100

...........

1712/1712 [==============================] - 25s 15ms/step - loss: 0.0014 - val_loss: 0.0010
Epoch 99/100
1712/1712 [==============================] - 25s 15ms/step - loss: 0.0014 - val_loss: 0.0010
Epoch 100/100
1712/1712 [==============================] - 25s 15ms/step - loss: 0.0014 - val_loss: 0.0010

把模型和权重保存

model2.save('model/model2.h5')
model2.save_weights('model/model2_weight.h5')

4.4 训练过程评估

建立显示show_train_history函数,查看训练过程的损失。可以发现损失已经降到了0.001左右,相对比前一个模型,损失降低了不少。

def show_train_history(train_history,train,validation):
    plt.plot(train_history.history[train])
    plt.plot(train_history.history[validation])
    plt.title('Train histoty')
    plt.grid()
    plt.ylabel(train)
    plt.xlabel('Epoch')
    plt.ylim(0.001, 0.01)
    plt.legend(['train','validation',],loc = 'upper left')
    plt.yscale('log')
    name = train + '.png'
    plt.savefig(name)
    plt.show()
show_train_history(history,'loss','val_loss')

loss

4.5 对模型进行预测

使用上小节训练的模型对测试集进行预测,如果没有自行训练模型的读者,可以附录获取下载已经训练好的模型的方法。使用load_weights加载预训练模型。

from keras.models import load_model
from keras.models import model_from_json
model2 = Model2()
model2.load_weights('./model/model2_weight.h5')
model_json = model2.to_json()
with open('./model/model2.json', 'w') as file:
    file.write(model_json)

使用之前定义好的加载函数加载测试集

import os
import sys
from pandas import DataFrame
from sklearn.utils import shuffle
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

train_cvs = './dataset/facial-keypoints-detection/training.csv'
test_cvs = './dataset/facial-keypoints-detection/test.csv'
lookup_cvs = './dataset/facial-keypoints-detection/IdLookupTable.csv'
def load_data(cvs_file, has_label=True):
    rc = pd.read_csv(cvs_file)
    rc['Image'] = rc['Image'].apply(lambda im: np.fromstring(im, sep=' '))
    rc = rc.dropna()
    x = np.vstack(rc['Image'].values) / 255.
    x = x.astype(np.float32)
    y = None
    if has_label:
        y = df[df.columns[:-1]].values
        y = (y - 48) / 48
        x, y = shuffle(x, y, random_state=42) 
        y = y.astype(np.float32)

    return x, y
x_img_test_4d , _ = load_data(test_cvs, has_label=False)
x_img_test_4d = x_img_test.reshape(-1, 96, 96, 1)
y_pred_2 = model2.predict(x_img_test_4d)

使用之前定义的plot_data函数绘制结果。

def plot_img(img, label, axis, c=['c'], s=10, cmap='gray'):
    img = img.reshape(96, 96)
    axis.imshow(img, cmap=cmap) 
    axis.scatter(label[0::2] * 48 + 48, label[1::2] * 48 + 48, marker='o', s=s, c=c)

def plot_data(x, y, begin=0, title=None, cmap='gray'):
    fig = plt.figure(figsize=(6, 6))
    fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)
    plt.title(title)
    for i in range(16):
        ax = fig.add_subplot(4, 4, i + 1, xticks=[], yticks=[])
        plot_img(x[begin+i], y[begin+i], ax, cmap=cmap)

    plt.show()
plot_data(x_img_test_4d, y_pred_2, begin=40, title='Model_2')

png

我们使用上一个模型对同一组测试数据进行预测,查看两个模型对比效果。

with open('./model/model.json', 'r') as file:
    model_json = file.read()
model = model_from_json(model_json)
model = load_model('model/model.h5')
x_img_test , _ = load_data(test_cvs, has_label=False)
y_pred_1 = model.predict(x_img_test)
plot_data(x_img_test, y_pred_1, begin=40, title='Model_1')

png

仔细的读者会发现,在同一组测试数据下Model_2的预测效果比Model_1更加的精准。

我们单独对单张人脸进行测试,随机选择一张人脸。

fig = plt.figure(figsize=(8, 8))
idx = 93
point_size = 30
fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)

# model 1
ax = fig.add_subplot(1, 2, 1, xticks=[], yticks=[])
ax.set_title('Model_1')
plot_img(x_img_test[idx], y_pred_1[idx], ax, s = point_size)

# model 2
ax = fig.add_subplot(1, 2, 2, xticks=[], yticks=[])
ax.set_title('Model_2')
plot_img(x_img_test[idx], y_pred_2[idx], ax, c=['r'], s= point_size)

plt.show()

png

显然,Model_1的关键点基本都是漂移的状态,相对Model_2的会精准许多。

5. 自定义测试集预测

在测试过数据集提供的测试集后,我们来尝试一下使用自己准备的数据集进行预测。使用我们附录中提供的faces1000测试集进行预测,下载文件后放入文件目录,或者也可自行安放路径。faces1000是作者自己收集整理的一个小批量的人脸图片集,主要用于一些简单的可视化测试,其中包含985张有效的人脸图片。数据集是通过人脸检测器筛查出来的,所以每张图片只有一个完整的人脸,很适合用于本次关键点预测的实验。

首先,定义read_directory函数,读取文件夹中的人脸图片。

from keras.models import load_model
from keras.models import model_from_json
import os
import cv2
# 定义图片读取函数
def read_directory(directory): 
    imgs = []
    for filename in os.listdir(directory):
        img = cv2.imread(os.path.join(directory, filename))
        imgs.append(img)
    return imgs

从文件夹中读取已经预训练好的Model2模型

with open('./model/model2.json', 'r') as file:
    model_json = file.read()
model2 = model_from_json(model_json)
model2.load_weights('./model/model2_weight.h5')

由于我们的自由图片集是彩色三通道的图片格式,所以我们修改之前的图像显示函数,得到show_img和show_img_list函数。

def show_img(img, axis, c=['c'], s=10, label=None):
    img = img.reshape(96, 96, 3)
    axis.imshow(img)
    if label is not None:
        axis.scatter(label[0::2] * 48 + 48, label[1::2] * 48 + 48, marker='o', s=s, c=c)

def show_img_list(x, begin=0, title=None, y=None):
    fig = plt.figure(figsize=(8, 8))
    fig.subplots_adjust(left=0, right=1, bottom=0, top=1, hspace=0.05, wspace=0.05)
    plt.title(title)
    for i in range(25):
        ax = fig.add_subplot(5, 5, i + 1, xticks=[], yticks=[])
        if y is not None:
            show_img(x[begin+i],ax, c=['w'], s=15, label=y[begin+i],)
        else:
            show_img(x[begin+i], ax, label=None)

    plt.show()

显示从189开始的原图查看效果。

# 显示原图时,不需要传入关键点数据,y=None即可
show_img_list(faces1000_imgs, y=None,begin=189)

png

读取faces1000文件夹下所有图片,使用imgs列表存储。

faces1000_path = './faces1000/'
imgs = read_directory(faces1000_path)

将图片转换为和神经网络输入尺寸一致,并划分三通道图和单通道图,三通道图用于展示,单通道图用于预测。

faces1000_imgs = []
faces1000_imgs_gray = []
for img in imgs:
    if img is not None:
        im = cv2.resize(img, (96, 96))
        im = cv2.cvtColor(im, cv2.COLOR_BGR2RGB)
        im_gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
        faces1000_imgs.append(im)
        faces1000_imgs_gray.append(im_gray)

将单通道图和三通道图的测试集进行数据转换。

faces1000_imgs = np.concatenate([faces1000_imgs])
faces1000_imgs = faces1000_imgs.reshape(faces1000_imgs.shape[0], 96, 96, 3)
faces1000_img_test = np.concatenate([faces1000_imgs_gray])
faces1000_img_test = faces1000_img_test.reshape(faces1000_img_test.shape[0], 96, 96, 1)

别忘了将需要传入模型的faces1000_img_test数据归一化处理

faces1000_img_test = faces1000_img_test/ 255.

预测faces1000_img_test集合

res = model2.predict(faces1000_img_test)

显示第189开始的后25个人脸

show_img_list(faces1000_imgs, y=res, begin=189)

png

测试单个人脸

idx = 365
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(1, 2, 2, xticks=[], yticks=[])
show_img(faces1000_imgs[idx], label=res[idx], axis=ax, s=35, c=['r'])
plt.show()

png

可以发现,经过小批量的测试,头部姿态稍正的人脸,模型都能成功预测,而头部有偏移的人脸经过预测会出现偏移的情况。解决这个问题可能需要对数据再次进行增强,例如添加角度偏移的数据增强方法,有兴趣的读者可以继续研究。

结论

本章主要从两个模型,一浅一深的向读者展现深度神经网络下人脸关键点回归任务的实验,关键点在人脸领域是一个比较重要的技术支撑,当下比较火爆的AI换脸、特效相机和表情分析等功能都离不开它,希望读者们可以继续深入研究人脸关键点的其他实验,如3D人脸、68以上关键点和更加稠密的关键点检测。


版权声明:如无特殊说明,文章均为本站原创,转载请注明出处

本文链接:http://tunm.top/article/learning_12/