Research · June 2001

Conditional random fields

At ICML 2001 John Lafferty (Carnegie Mellon), Andrew McCallum and Fernando Pereira (WhizBang! Labs, Pereira also the University of Pennsylvania) presented conditional random fields: a model of the probability of a whole label sequence given an observation sequence, normalised over the sequence rather than at each state. On part-of-speech tagging of the Penn treebank the per-word error was 5.55 per cent against 5.69 for a hidden Markov model and 6.37 for a maximum entropy Markov model; with spelling features, 4.27 against 4.81.

Why it matters

A sequence could be labelled using many overlapping features of the input, which a generative HMM cannot represent, and without the label bias of discriminative models normalised state by state, which the paper says can be biased towards states with few successor states. It is the step from the generative model of the 1989 tutorial to discriminative sequence labelling. An editorial assessment.

Figure 4 gives per-word error and error on out-of-vocabulary words: HMM 5.69 and 45.99 per cent, MEMM 6.37 and 54.61, CRF 5.55 and 48.05, MEMM with spelling features 4.81 and 26.99, CRF with them 4.27 and 23.76. Without spelling features CRFs beat the HMM overall but are worse on unseen words. Read in full from the authors' postprint in the University of Pennsylvania repository, which says the definitive version is in the ICML 2001 proceedings, pages 282-289. What the record does not claim: the venue and closing date of the conference, which no source read names, or 28 June as the day of the talk.

Event record

Event date

June 2001

Timeline date

Event date

Verification

Sources gathered automatically · September 24, 2026

Lines

Learning theory

ID

evt-0659

Proceedings of the 18th International Conference on Machine Learning (ICML 2001), pages 282-289. The University of Pennsylvania repository gives 28 June 2001 as the date of presentation, but it is the only source for the day: the ACM page sits behind a Cloudflare check and the conference site has no archived copy. Hence month precision.

Sources

primary Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
University of Pennsylvania, ScholarlyCommons (Departmental Papers, CIS) · Published June 2001

Related events

Builds on The tutorial account of hidden Markov models
CRFs are framed as an answer to a limit of hidden Markov models, which cannot represent many overlapping features of the observation; on the same Penn treebank data the paper compares CRFs with HMMs.
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Related AdaBoost
Section 6 of the paper: a CRF can be trained with the same exponential loss that AdaBoost uses.
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

Records that link to this one

Enables Maximum entropy for language processing
The conditional random fields paper names the model of Berger et al. among the local conditional models that suffer from the label bias CRFs remove, and builds its training on the iterative scaling of Della Pietra et al.
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data