Evaluation

Important Notions

• Empirical evaluation of accuracy:
  • Independent test data
  • Supervised or unsupervised
  • Gold standard evaluation
• Descriptive statistics/measures of accuracy:
  • Accuracy rate (percent correct): 1/n ∑_ix_i
  • Recall: true positives / (true positives + false negatives)
  • Precision: true positives / (true positives + false positives)
  • Logprob: 1/n - ∑_i log P(x_i)
• Statistical inference:
  • Confidence intervals
  • Hypothesis testing (significance)

Slides for lecture 10

Suggested Reading

• Manning, C. D. & Schütze, H. (1999) Foundations of Statistical Natural Language Processing. MIT Press. Sections 6.2.3, 10.6.1., 12.1.8, 15.1.2.