The Beauty of Capturing Faces: Rating the Quality of Digital Portraits

This is the main idea and brief content of essay:
The Beauty of Capturing Faces: Rating the Quality of Digital Portraits.

Abstract

automatic portrait beauty assessment
not only aesthetic scores but with the traits of the subject portrayed
a set of visual features and analyze relations with beauty
classifier to separate beautiful portrait outperforms generic aesthetic classifiers

Introduction

Dataset
Features
Feature analysis: race, gender, age uncorrelated with photographic beauty
Aesthetic prediction: develop a predictive models to classify

Sec. II related work

Sec. III portrait database

Sec. 4. visual feature
Sec. 5. analyze the relations
Sec. 6. classification experiment

Preliminary works: illumination, landmark representation, affective properties, and experimented small dataset.
Design visual features to describe image quality and portrait-specific property.

Related work

Tag-based VS content-aware sampling strategy
Preliminary use traditional aesthetic features designed for a general case, and apply them to the topic-specific contexts
Both metadata-based filtering and face analysis
Number:250,000----->21719----->10141
Face++：position, orientation, demographics, coordinates of facial landmarks, smile and glasses

non-semantic fuzzy concepts of image analysis:

memorability
emotions
interesting
privacy
beauty

Compositional and aesthetic criteria constitute a separate subject of study in the photographic literature

We build a rich large-scale portrait aesthetic database
We build portrait-specific aesthetic visual features design face-specific aesthetic feature with non-face feature, such as illumination, sharpness, manipulation detection, image quality, emotion and memorability. and we show their combined effectiveness
for aesthetic assessment

Now we have non-feature :
illumination , sharpness, manipulation detection, image quality, emotion and memorability
and portrait-specific aesthetic visual features
face expression, face pose, and face position features
hand-crafted features together with low-level generic feature
spatial composition rules
together with specific background contrast features and face brightness and size feature
illumination, demographics, face landmark properties, affective dimension, semantics and post-processing

Large scale portrait dataset

AVA dataset: more than 250,000 images annotated with an aesthetic score, a challenge title and semantic textual tags

enhanced metadata-based filtering: including portrait and portraiture or portraits
face detection-based filtering: Face++ to detect face from 21,719 to 10,141
subject properties: position, orientation, demographics(race, gender, age), coordinates of facial landmarks(eyes, nose and mouth in relative coordinates), smile and glasses

Features for portrait aesthetic assessment

5 main photographic dimensions :

compositional rules
scene semantics
portrait-specific features ： aspect, soft biometrics, demographics , sharpness, illumination
basic quality metrics
fuzzy properties

Compositional rules

focus on single subject, capture essential property of image and collect a set of existing features

lighting

raw brightness channel information might not be enough to capture portrait light patterns
Light Pattern Feature: k-means

sharpness

HSV RGB HSL UAV
Overall Sharpness: Sobel-Tenengrad method
Camera shake: camera motion estimation algorithm of Chakrabarti

traditional compositional features

color features: color names, HSV, pleasure, arousal, dominance metrics, the Itten color histograms
spatial arrangement features: Symmetry(edge) ; Symmetry(HOG), Number of Circles and Rules of Thirds
Texture features: smoothness , order and entropy of the image : GLCM ， Image Order and Level of Detail

Semantics and Scene Content

object bank features that retains the maximum probability of a pixel in the image to be part of one of the 208 objects in the object bank

basic quality metrics

noise
contrast quality
exposure quality: YCbCr
JPEG quality
Image Manipulations: amount of splicing manipulation and compute the amount of median filtering manipulation

Portrait-specific features

face++ description: position, orientation, demographics age gender landmark coordinate expression other properties glasses
landmark sharpness: gradient magnitude
landmark statistics:for each landmark, we get Hue and Brightness
face/background contrasts: consider Face and background as separate images and compute the Lighting Contrast to F,B and get F/B

Fuzzy properties

emotion： SVM emotion classifier
originality: originality classifier from Support Vector Regression
memorability: memorability classifier trained with the Saliency Moments Features SVR
uniqueness: euclidean distance between average spectrum of the image and the spectrum of each image

What make a portrait beautiful

Feature groups for Portrait aesthetic

perform regression analysis using LASSO

Portrait-specific Features correlate the most , with 0.33
Semantic Features with its 190 features detector, achieve 0.211
Compositional Features achieve 0.29
Basic Quality and Fuzzy Property much lower correlation score

Single Features for Portrait Aesthetics

face sharpness and light are of crucial importance
contrast in sharpness between face and background strongly correlates with portrait beauty
light pattern are fundamental
contrast in colors and in gray levels show positive correlation
appealing portraits should have a homogeneous, smooth composition without disturbing distortions
amount of median filtering is negative
exposure quality is negatively
beautiful portraits have little colors such as green, purple, magenta
originality and uniqueness is positive
gender, eye color, glasses, age, race show little correlation