FAQs

Q: Can I use extra data?
A: Yes. The distributed train and dev corpora were recorded using the same protocol as the test corpora, and are therefore expected to be better-matched to the test data than any other source. Competing teams may find, however, that system performance benefits from the use of other datasets; if so, teams are encouraged to describe the other datasets used, and the method of their use.

Q: Is there a private test set?
A: Yes. During the competition, submitted systems will be tested using the test1 set, and results will be immediately published. After the end of the competition period, each team’s ultimate submission will be tested using the test2 set. Winners will be determined based on scores achieved on the test2 set.

Q: How do I benchmark on the leaderboard?
A: Submit your system using the provided template (link) through the CodaBench Challenge page (link) to receive official evaluation and appear on the leaderboard.

Q: Do I need to open-source my code?
A: No. Code submitted to the challenge remains the intellectual property of the submitting team unless separately open-sourced.