Language Modeling

Important Notions

• String probabilities: P(w₁,...,w_n)
• Types of models:
  • N-gram models
  • N-class models
  • Probabilistic context-free grammars (PCFG)
• Parameter estimation (training):
  • Maximum likelihood estimation (MLE)
  • Sparse data
  • Smoothing techniques:
    • Additive smoothing
    • Held-out estimation
    • Good-Turing reestimation
    • Back-off smoothing
    • Linear interpolation
• Evaluation methods:
  • Probability
  • Entropy and related measures

Slides for lecture 5

Suggested Reading

• Manning, C. D. & Schütze, H. (1999) Foundations of Statistical Natural Language Processing. MIT Press. Chapter 6 (except 6.2.4, 6.2.6, 6.3.3-4).

Project

• Modeling a blocks world language (English data)