Cloze-driven Pretraining of Self-attention Networks

Open paper

Benchmark	Model	Rank	Results
constituency-parsing-on-penn-treebank	CNN Large + fine-tune	–	F1 score: 95.6
named-entity-recognition-ner-on-conll-2003	CNN Large + fine-tune	–	F1: 93.5
sentiment-analysis-on-sst-2-binary	CNN Large	–	Accuracy: 94.6