DeepStruct: Pretraining of Language Models for Structure Prediction

Benchmark Model Rank Results
dialogue-state-tracking-on-multiwoz-2-1DeepStruct multi-task#1Joint Acc: 53.5
dialogue-state-tracking-on-multiwoz-2-1DeepStruct multi-task w/ finetune#2Joint Acc: 54.2
event-extraction-on-ace2005DeepStruct multi-task#1Argument Cl: 63.9Argument Id: 67.5Trigger Cl: 69.2Trigger Id: 72.7
event-extraction-on-ace2005DeepStruct multi-task w/ finetune#2Argument Cl: 56.2Argument Id: 59.4Trigger Cl: 69.8Trigger Id: 73.5
joint-entity-and-relation-extraction-on-2DeepStruct multi-task w/ finetune#1Entity F1: 90.7Relation F1: 78.3
joint-entity-and-relation-extraction-on-2DeepStruct multi-task#2Entity F1: 88.4Relation F1: 72.8
joint-entity-and-relation-extraction-on-2Deepstruct zero-shot#3Entity F1: 48.3Relation F1: 25.8
joint-entity-and-relation-extraction-on-7DeepStruct multi-task#1Entity F1: 90.2Relation F1: 58.9
joint-entity-and-relation-extraction-on-7DeepStruct multi-task w/ finetune#2Entity F1: 90.0Relation F1: 66.8
joint-entity-and-relation-extraction-on-7Deepstruct zero-shot#3Entity F1: 31.8Relation F1: 5.3
joint-entity-and-relation-extraction-on-ade-1DeepStruct multi-task w/ finetune#1Entity F1: 91.1Relation F1: 83.8
joint-entity-and-relation-extraction-on-ade-1DeepStruct multi-task#2Entity F1: 90.5Relation F1: 83.6
joint-entity-and-relation-extraction-on-ade-1Deepstruct zero-shot#3Entity F1: 60.7Relation F1: 10.6
joint-entity-and-relation-extraction-on-nytDeepStruct multi-task w/ finetune#3Entity F1: 95.9Relation F1: 93.3
joint-entity-and-relation-extraction-on-nytDeepStruct multi-task#4Entity F1: 95.4Relation F1: 93.7
joint-entity-and-relation-extraction-on-nytDeepstruct zero-shot#5Entity F1: 60.5Relation F1: 28.6
named-entity-recognition-on-conll03DeepStruct multi-task#2F1: 93.1
named-entity-recognition-on-conll03DeepStruct multi-task w/ finetune#3F1: 93.0
named-entity-recognition-on-conll03Deepstruct zero-shot#4F1: 44.4
named-entity-recognition-on-geniaDeepStruct multi-task w/ finetune#1F1: 80.8
named-entity-recognition-on-geniaDeepStruct multi-task#3F1: 80.2
named-entity-recognition-on-geniaDeepstruct zero-shot#14F1: 47.2
named-entity-recognition-on-ontonotesDeepStruct multi-task w/ finetune#1F1: 87.8
named-entity-recognition-on-ontonotesDeepStruct multi-task#2F1: 87.6
named-entity-recognition-on-ontonotesDeepstruct zero-shot#3F1: 2.5
open-information-extraction-on-nytDeepstruct zero-shot#1F1: 28.9
open-information-extraction-on-nytDeepStruct multi-task#2F1: 43.6
open-information-extraction-on-nytDeepStruct multi-task w/ finetune#3F1: 45.0
open-information-extraction-on-oie2016DeepStruct multi-task w/ finetune#2F1: 71.3
open-information-extraction-on-oie2016Deepstruct multi-task#3F1: 71.2
open-information-extraction-on-oie2016Deepstruct zero-shot#5F1: 28.1
open-information-extraction-on-penn-treebankDeepStruct multi-task w/ finetune#1F1: 45.1
open-information-extraction-on-penn-treebankDeepstruct zero-shot#2F1: 51
open-information-extraction-on-penn-treebankDeepStruct multi-task#3F1: 54.5
open-information-extraction-on-webDeepstruct zero-shot#1F1: 43.8
open-information-extraction-on-webDeepStruct multi-task w/ finetune#2F1: 49.1
open-information-extraction-on-webDeepStruct multi-task#3F1: 50.8
relation-classification-on-fewrel-1DeepStruct multi-task w/ finetune#1F1 (10-way 1-shot): 97.8F1 (10-way 5-shot): 99.8
relation-classification-on-fewrel-1DeepStruct multi-task#2F1 (10-way 1-shot): 92.2F1 (10-way 5-shot): 94.6
relation-classification-on-fewrel-1Deepstruct zero-shot#3F1 (10-way 1-shot): 67.6F1 (10-way 5-shot): 66.4
relation-classification-on-tacred-1Deepstruct zero-shot#1F1: 36.1
relation-classification-on-tacred-1DeepStruct multi-task#15F1: 74.9
relation-classification-on-tacred-1DeepStruct multi-task w/ finetune#17F1: 76.8
relation-extraction-on-tacredDeepStruct multi-task w/ finetune#2F1: 76.8