LinkBERT: Pretraining Language Models with Document Links

Benchmark Model Rank Results
document-classification-on-hocBioLinkBERT (large)#1F1: 88.1Micro F1: 84.87
named-entity-recognition-ner-on-bc5cdrBioLinkBERT (large)#5F1: 90.22
named-entity-recognition-ner-on-jnlpbaBioLinkBERT (large)#6F1: 80.06
named-entity-recognition-ner-on-ncbi-diseaseBioLinkBERT (large)#9F1: 88.76
named-entity-recognition-on-bc2gmBioLinkBERT (large)#8F1: 85.18
named-entity-recognition-on-bc5cdr-chemicalBioLinkBERT (large)#9F1: 94.04
named-entity-recognition-on-bc5cdr-diseaseBioLinkBERT (large)#5F1: 86.39
pico-on-ebm-picoBioLinkBERT (large)#1Macro F1 word level: 74.19
pico-on-ebm-picoBioLinkBERT (base)#2Macro F1 word level: 73.97
question-answering-on-bioasqBioLinkBERT (large)#1Accuracy: 94.8
question-answering-on-bioasqBioLinkBERT (base)#3Accuracy: 91.4
question-answering-on-blurbBioLinkBERT (large)#1Accuracy: 83.5
question-answering-on-blurbBioLinkBERT (base)#2Accuracy: 80.81
question-answering-on-medqa-usmleBioLinkBERT (base)#17Accuracy: 40.0
question-answering-on-newsqaLinkBERT (large)#13F1: 72.6
question-answering-on-pubmedqaBioLinkBERT (large)#17Accuracy: 72.2
question-answering-on-pubmedqaBioLinkBERT (base)#18Accuracy: 70.2
question-answering-on-squad11LinkBERT (large)#5EM: 87.45F1: 92.7
question-answering-on-triviaqaLinkBERT (large)#32F1: 78.2
relation-extraction-on-chemprotBioLinkBERT (large)#3F1: 79.98Micro F1: 79.98
relation-extraction-on-ddiBioLinkBERT (large)#2F1: 83.35Micro F1: 83.35
relation-extraction-on-gadBioLinkBERT (large)#2F1: 84.90Micro F1: 84.90
semantic-similarity-on-biossesBioLinkBERT (large)#1Pearson Correlation: 0.9363
semantic-similarity-on-biossesBioLinkBERT (base)#2Pearson Correlation: 0.9325
text-classification-on-blurbBioLinkBERT (large)#1F1: 84.88
text-classification-on-blurbBioLinkBERT (base)#2F1: 84.35