CodeBERT: A Pre-Trained Model for Programming and Natural Languages

Benchmark Model Rank Results
code-documentation-generation-on-codesearchnet-goCodeBERT (MLM)#1Smoothed BLEU-4: 26.79
code-documentation-generation-on-codesearchnet-goCodeBERT (MLM+RTD)#2Smoothed BLEU-4: 26.66
code-documentation-generation-on-codesearchnet-gopre-train w/ code only#3Smoothed BLEU-4: 26.39
code-documentation-generation-on-codesearchnet-goRoBERTa#4Smoothed BLEU-4: 26.09
code-documentation-generation-on-codesearchnet-goCodeBERT (RTD)#5Smoothed BLEU-4: 26.02
code-documentation-generation-on-codesearchnet-goseq2seq#6Smoothed BLEU-4: 23.48
code-documentation-generation-on-codesearchnet-javascriptTransformer#1Smoothed BLEU-4: 25.61
code-documentation-generation-on-codesearchnet-javascriptCodeBERT (MLM+RTD)#3Smoothed BLEU-4: 9.54
code-documentation-generation-on-codesearchnet-javascriptCodeBERT (RTD)#4Smoothed BLEU-4: 8.73
code-documentation-generation-on-codesearchnet-javascriptCodeBERT (MLM)#5Smoothed BLEU-4: 8.51
code-documentation-generation-on-codesearchnet-javascriptpre-train w/ code only#6Smoothed BLEU-4: 8.3
code-documentation-generation-on-codesearchnet-javascriptseq2seq#7Smoothed BLEU-4: 6.88
code-documentation-generation-on-codesearchnet-javascriptRoBERTa#8Smoothed BLEU-4: 5.72
code-documentation-generation-on-codesearchnet-javaCodeBERT (MLM+RTD)#2Smoothed BLEU-4: 14.56
code-documentation-generation-on-codesearchnet-javaCodeBERT (MLM)#3Smoothed BLEU-4: 13.59
code-documentation-generation-on-codesearchnet-javaRoBERTa#4Smoothed BLEU-4: 13.2
code-documentation-generation-on-codesearchnet-javapre-train w/ code only#5Smoothed BLEU-4: 13.07
code-documentation-generation-on-codesearchnet-javaCodeBERT (RTD)#6Smoothed BLEU-4: 12.72
code-documentation-generation-on-codesearchnet-javaTransformer#7Smoothed BLEU-4: 12.57
code-documentation-generation-on-codesearchnet-javaseq2seq#8Smoothed BLEU-4: 11.42
code-documentation-generation-on-codesearchnet-phpCodeBERT (MLM+RTD)#2Smoothed BLEU-4: 21.32
code-documentation-generation-on-codesearchnet-phpCodeBERT (MLM)#3Smoothed BLEU-4: 21
code-documentation-generation-on-codesearchnet-phppre-train w/ code only#4Smoothed BLEU-4: 20.71
code-documentation-generation-on-codesearchnet-phpCodeBERT (RTD)#5Smoothed BLEU-4: 20.25
code-documentation-generation-on-codesearchnet-phpRoBERTa#6Smoothed BLEU-4: 19.9
code-documentation-generation-on-codesearchnet-phpseq2seq#7Smoothed BLEU-4: 18.4
code-documentation-generation-on-codesearchnet-phpTransformer#8Smoothed BLEU-4: 18.25
code-documentation-generation-on-codesearchnet-pythonCodeBERT (MLM)#2Smoothed BLEU-4: 15.48
code-documentation-generation-on-codesearchnet-pythonCodeBERT (MLM+RTD)#3Smoothed BLEU-4: 15.41
code-documentation-generation-on-codesearchnet-pythonpre-train w/ code only#4Smoothed BLEU-4: 15.12
code-documentation-generation-on-codesearchnet-pythonRoBERTa#5Smoothed BLEU-4: 14.92
code-documentation-generation-on-codesearchnet-pythonTransformer#6Smoothed BLEU-4: 13.44
code-documentation-generation-on-codesearchnet-pythonseq2seq#7Smoothed BLEU-4: 13.04
code-documentation-generation-on-codesearchnet-rubyCodeBERT (MLM+RTD)#2Smoothed BLEU-4: 8.46
code-documentation-generation-on-codesearchnet-rubyCodeBERT (MLM)#3Smoothed BLEU-4: 7.95
code-documentation-generation-on-codesearchnet-rubyTransformer#4Smoothed BLEU-4: 7.87
code-documentation-generation-on-codesearchnet-rubypre-train w/ code only#5Smoothed BLEU-4: 7.36
code-documentation-generation-on-codesearchnet-rubyRoBERTa#6Smoothed BLEU-4: 7.26
code-documentation-generation-on-codesearchnet-rubyseq2seq#7Smoothed BLEU-4: 6.96
type-prediction-on-manytypes4typescriptCodeBERT#4Average Accuracy: 61.72Average Precision: 59.34Average Recall: 59.80…
code-search-on-codesearchnetCodeBERT#6Overall: 76.0Go: 69.3Ruby: 70.6Python: 84.0Java: 86.8…