II UNL Olympiad

From UNL Wiki
Revision as of 13:53, 20 March 2013 by Martins (Talk | contribs)
Jump to: navigation, search

The UNL Olympiad is a series of competitions organised by the UNDL Foundation in order to foster the development of UNL-driven resources (dictionaries, grammars and corpora). The second edition of the Olympiad is devoted to the development of grammars for the corpus UC-A1. The competition is open to any participant, and the deadline is July 1st, 2013.

Contents

Important dates

  • July 1st, 2013: Deadline for submitting the files

Modalities

The competition is organised in two modalities:

  • Best UNLization Grammar for <LANGUAGE>
  • Best NLization Grammar for <LANGUAGE>

Where <LANGUAGE> is one of the languages participating in this Olympiad (see the complete list below).
Candidates may participate in one or two modalities, i.e., they may work with the UNLization grammar, with the NLization grammar, or with both.
Candidates may also participate in one or more languages, provided that they belong to the list below.

Prizes

Prizes are awarded to the best grammars of each modality (UNLization and NLization) for each language[1]:

  • 1st place: Gold Medal
  • 2nd place: Silver Medal
  • 3rd place: Bronze Medal

Additionally, the 10 best UNLization grammars and the 10 best NLization grammars will be awarded with USD500.00 each.

Registration

Candidates must be registered at the UNLweb. Participation is open and free, and the registration to the Olympiad is done by sending the following files to r.martins@undlfoundation.org until 23:59:59 (UTC) of July 1st, 2013.

  • For the participants working with the UNLization grammar (IAN):
    • UCA1_<LID>.txt, with the human translation, to the target language, of the sentences of the Corpus UC-A1;
    • <LID>_unl_dic.txt, with the natural language analysis dictionary used to UNLize the translated version of the Corpus UC-A1;
    • <LID>_unl_tgrammar.txt, with the transformation grammar used to UNLize the translated version of the Corpus UC-A1;
    • <LID>_unl_dgrammar.txt, with the disambiguation grammar used to UNLize the translated version of the Corpus UC-A1;
    • <LID>_unl_output.txt, with the output provided by IAN
  • For the participants working with the NLization grammar (EUGENE)
    • UCA1_<LID>.txt, with the human translation, to the target language, of the sentences of the Corpus UC-A1;
    • unl_<LID>_dic.txt, with the natural language generation dictionary used to NLize the UNL version of Corpus UC-A1;
    • unl_<LID>_tgrammar.txt, with the transformation grammar used to NLize the UNL version of the Corpus UC-A1;
    • unl_<LID>_dgrammar.txt, with the disambiguation grammar used to NLize the UNL version of the Corpus UC-A1;
    • unl_<LID>_output.txt, with the output provided by EUGENE

Where <LID> must be replaced by the three-character language according to ISO 639-3.[2].
All files must be provided in UTF-8.

Requisites

The competition is free and open to any participant, but it is limited to the set of languages described below.
The files must comply with the following requisites:

  • The corpus must comply with the translation standards of the target language and should not be artificially translated in order to provoke better results.
  • The input corpus used in UNLization will be used as the reference corpus used to evaluate the NLization output.
  • The dictionary files must comply with the Dictionary Specs and may only bring features present in the Tagset. They should not contain temporary words.
  • The grammar files must comply with the Grammar Specs and must be as generic possible. They should not target the specific sentences of the corpus, but the general structures presented there.
  • The F-measure of the grammars must be equal or greater than 0.8.[3]
  • The files must be original. Grammars whose similarity proves to go beyond any reasonable doubt will be discarded, unless provided by the same author (for different languages).

Evaluation

Grammars will be evaluated and ranked according to the following criteria:

  • Best F-measure
  • Scalability (i.e., extendibility, or the capacity of being reused to other corpora), in case of grammars with the same F-Measure
  • Date of submission, in case of grammars with the same F-Measure and equally scalable

Languages

The II UNL Olympiad will be dedicated to the development of grammars for the following languages[4]

  • Afrikaans
  • Armenian
  • Bengali
  • Bulgarian (NLization only)
  • Chinese
  • Croatian
  • Estonian
  • German
  • Greek (Modern)
  • Japanese
  • Kashmiri
  • Khmer
  • Laotian
  • Latin
  • Malay
  • Oriya (NLization only)
  • Punjabi
  • Serbian
  • Sinhala
  • Slovenian
  • Tamil
  • Telugu
  • Thai
  • Vietnamese


Candidates may participate in one or more languages above.


Notes

  1. This means that for each language there will be awarded up to 6 prizes: Best UNLization Grammar, Second Best UNLization Grammar, Third Best UNLization Grammar, Best NLization Grammar, Second Best UNLization Grammar and Third Best NLization Grammar
  2. For instance, the files to be provided by Russian (code = "rus") must be UCA1_rus.txt, rus_unl_dic.txt, rus_unl_tgrammar.txt, etc.
  3. The F-measure may be calculate at UNLWEB>UNLARIUM>GRAMMAR>[LOCALE]>F-MEASURE
  4. The choice of the languages was motivated by three criteria: 1) Languages for which we do not have the basic grammars yet; 2) Languages that have participated in the recent UNL Schools; and 3) Languages that have already started the project MIR-A1.
Software