ERNIE 2.0: pre-training that adds tasks
On 29 July 2019 a Baidu group led by Yu Sun posted ERNIE 2.0, a scheme in which new pre-training tasks are added to a language model one after another and learned together with the earlier ones so that what was learned is not lost. The large model scored 83.6 on the hidden GLUE test set, 3.1 per cent above BERT, and beat BERT on all ten test tasks.
Why it matters
Pre-training stopped being a pair of fixed tasks such as guessing a masked word: new signals - names, sentence order, discourse relations - can be added to it at any point. The result was measured on a test whose labels the benchmark holds, not the authors.
Seven tasks of three kinds: word, structure and meaning. The English corpus is Wikipedia, BookCorpus, Reddit and the Discovery data for discourse relations, the Chinese one from Baidu's search engine. The transformer settings are BERT's; the base model scores 80.6 on GLUE. In Chinese it was tested on nine tasks against BERT and ERNIE 1.0. XLNet was compared on the development set only, as XLNet reported no test results. What the record does not claim: the figure of 16 tasks in the abstract - the paper's own tables add up differently; the day ERNIE topped the GLUE leaderboard could not be read, as the leaderboard is drawn by script.