Wee5-3PP attachment 2

Algorithms for PP attachment

Alg0

Dumb one, to label all labels as a default label, say low

Alg1: random baseline

A random unsupervised baseline would have to label each instance in the test data with a random label, 0 or 1.

Practically, it is the lower bound of all the other algorithms

The evaluation is performed on the test set, not training set, and test set should never be used until you completely finished your algorithm, and you could use development set to tune the parameters or add features

Observations

Data set has the characteristics of domain specification

Too complicate method will lead to overfit

Upper Bound

Usually, human performance is used for the upper bound. For PP attachment, using the 4 previous features, human accuracy is around 88%. So we could expect an upper bound of 87%

Using linguistic knowledge

For example, we know the preposition of is much more likely to be associated with low attachment than high attachment (98.7% with 5000+ instances in training set). Therefore, the feature prep_of is very valuable(informative and frequent)

Alg2

If  the prep is 'of', label the tuple as 'low'
Else
If the prep is 'to', label the tuple as 'high'
Else
label the tuple as 'low'(default)

The accuracy will be around 60%, proving that using linguistic knowledge will improve the algorithm

Alg2a

If  the prep is 'of', label the tuple as 'low'
Else
If the prep is 'to', label the tuple as 'high'
Else
label the tuple as 'high'(default)

After looking at prep ‘of’ and ‘to’, the remaining data set contains more high attachment than low attachment on average, and the accuracy will be around 74%.

Alg3

We will set rules for every case in the training set, too specific.

Remark: Due to inconsistent human labelers, more context is needed.