NL Reference Corpus

From UNL Wiki
(Difference between revisions)
Jump to: navigation, search
(→Files)
(→Methodology)
Line 10: Line 10:
 
As a natural language corpus, the NC varies for each language. It is derived from a base corpus to be compiled and processed according to the following criteria:
 
As a natural language corpus, the NC varies for each language. It is derived from a base corpus to be compiled and processed according to the following criteria:
 
#The '''Base Corpus''' must have at least 5,000,000 tokens (strings isolated by blank space and other word boundary markers). It must be representative of the contemporary standard use of the written language, and should include documents from as many different genres and domains as possible.  
 
#The '''Base Corpus''' must have at least 5,000,000 tokens (strings isolated by blank space and other word boundary markers). It must be representative of the contemporary standard use of the written language, and should include documents from as many different genres and domains as possible.  
−
#The Base Corpus must be '''segmented''' (in sentences) and '''tagged''' for POS.
+
#The Base Corpus must be '''segmented''' (in sentences).
−
#The segmented corpus is used to calculate the '''average sentence length''' (ASL), which is the median of the length (in words) of all sentences.
+
#The Segmented Corpus must be '''tokenized''' (according to the natural language dictionary exported from the UNLarium).
−
#The tagged corpus is used to extract the [[LSS|linear sentence structures]] (LSS), which are sequences of POS, and to calculate their frequency of occurrence.
+
#The Tokenized Corpus must be '''annotated''' for lexical category, in order to generate the [[LSS|linear sentence structures]] (LSS).
−
#The average sentence length (ASL) and the linear sentence structures (LSS) are used to generate the '''NC templates''', as follows:
+
#The Annotated Corpus must be subdivided into 6 different subsets, according to the number of tokens:
−
#*NC-A1 =         1,000 most frequent LSS where length <= 15th percentile (very small sentences)
+
##*A1 = length <= 15th percentile (very small sentences)
−
#*NC-A2 = NC-A1 + 1,000 most frequent LSS where 15th percentile < length <= 30th percentile (small sentences)
+
##*A2 = 15th percentile < length <= 30th percentile (small sentences)
−
#*NC-B1 = NC-A2 + 1,000 most frequent LSS where 30th percentile < length <= 45th percentile (small medium-size sentences)
+
##*B1 = 30th percentile < length <= 45th percentile (small medium-size sentences)
−
#*NC-B2 = NC-B1 + 1,000 most frequent LSS where 45th percentile < length <= 60th percentile (long medium-size sentences)
+
##*B2 = 45th percentile < length <= 60th percentile (long medium-size sentences)
−
#*NC-C1 = NC-C1 + 1,000 most frequent LSS where 60th percentile < length <= 80th percentile (long sentences)
+
##*C1 = 60th percentile < length <= 80th percentile (long sentences)
−
#*NC-C2 = NC-C2 + 1,000 most frequent LSS where length > 80th percentile (very long sentences)
+
##*C2 = length > 80th percentile (very long sentences)
−
#The NC templates are used to compile the NC corpora: the training corpora and the testing corpora. The training corpora consists of 1 exemplar of each LSS, and will be used to prepare the grammar. The testing corpora consists of 4 exemplars of each LSS randomly selected in the Base Corpus. The whole NC corpora (i.e., 5 exemplars for each LSS) is used to calculate the [[F-measure]], which is the parameter for assessing the precision and the recall of the grammars.
+
#Each subcorpus is used to compile the NC corpus: the training corpora and the testing corpora. They refer to the 1,000 most frequent LSS.
 +
#:The training corpora consists of 1 exemplar of the each LSS, and will be used to prepare the grammar. (1,000 sentences in total)
 +
#:The testing corpora consists of 4 exemplars of each LSS randomly selected in the Annotated Corpus. (4,000 sentences in total)
 +
#The whole NC corpus (i.e., 5 exemplars for each LSS) is used to calculate the [[F-measure]], which is the parameter for assessing the precision and the recall of the grammars.
  
 
== Files ==
 
== Files ==

Revision as of 18:33, 17 April 2014

The NL Reference Corpus (NC) is the corpus used to prepare and to assess grammars for sentence-based UNLization. It is divided in 6 different levels according to the Framework of Reference for UNL (FoR-UNL):

  • NC-A1: NL Reference Corpus A1
  • NC-A2: NL Reference Corpus A2
  • NC-B1: NL Reference Corpus B1
  • NC-B2: NL Reference Corpus B2
  • NC-C1: NL Reference Corpus C1
  • NC-C2: NL Reference Corpus C2

Methodology

As a natural language corpus, the NC varies for each language. It is derived from a base corpus to be compiled and processed according to the following criteria:

  1. The Base Corpus must have at least 5,000,000 tokens (strings isolated by blank space and other word boundary markers). It must be representative of the contemporary standard use of the written language, and should include documents from as many different genres and domains as possible.
  2. The Base Corpus must be segmented (in sentences).
  3. The Segmented Corpus must be tokenized (according to the natural language dictionary exported from the UNLarium).
  4. The Tokenized Corpus must be annotated for lexical category, in order to generate the linear sentence structures (LSS).
  5. The Annotated Corpus must be subdivided into 6 different subsets, according to the number of tokens:
      • A1 = length <= 15th percentile (very small sentences)
      • A2 = 15th percentile < length <= 30th percentile (small sentences)
      • B1 = 30th percentile < length <= 45th percentile (small medium-size sentences)
      • B2 = 45th percentile < length <= 60th percentile (long medium-size sentences)
      • C1 = 60th percentile < length <= 80th percentile (long sentences)
      • C2 = length > 80th percentile (very long sentences)
  6. Each subcorpus is used to compile the NC corpus: the training corpora and the testing corpora. They refer to the 1,000 most frequent LSS.
    The training corpora consists of 1 exemplar of the each LSS, and will be used to prepare the grammar. (1,000 sentences in total)
    The testing corpora consists of 4 exemplars of each LSS randomly selected in the Annotated Corpus. (4,000 sentences in total)
  7. The whole NC corpus (i.e., 5 exemplars for each LSS) is used to calculate the F-measure, which is the parameter for assessing the precision and the recall of the grammars.

Files

Retrieved from "http://www.unlweb.net/wiki/index.php?title=NL_Reference_Corpus&oldid=7643"
Software