English Disambiguation Grammar

The English Disambiguation Grammar is used to control the tokenization of the English sentences, i.e., to prevent wrong lexical choices and to induce the best matches. It follows the formalism described at UNL Grammar Specs and comprises two different types of rules:

• Negative (blocking) rules, where the probability is equal to 0, prevent lexical choices

  For instance, the rule (D)(BLK)(V)=0; informs that the sequence determiner+blank space+verb is not allowed, i.e., there cannot be a determiner before a verb.

• Positive rules, where the probability is more than 0, force lexical choices

  For instance, the rule (['s],V)(BLK)(GER)=1; informs that the entry ['s] as a verb is to be preferred before a gerund (there are three entries ['s] in the dictionary: the contracted form of "is", the particle used to form the genitive and a plural suffix; if this rule is not stated, the system would simply select the first one appearing in the dictionary with the highest frequency)

Examples of disambiguation rules

Negative rules

Preventing the hyper-segmentation of temporary entries

"asdfg" must be tokenized as [asdfg] (one single temporary entry) instead of [as][dfg], which would be the case, because [as] is in the dictionary

(^W,^" ",^PUT,^SBW,^DIGIT,^PFX,^SHEAD)(^W,^" ",^PUT,^SBW,^DIGIT,^SFX,^STAIL)=0;

This rule states that words must be isolated by blank spaces, punctuation sign or SHEAD and STAIL, except in the case of subwords, prefixes and suffixes.

Preventing the generation of two temporary words in sequence

"asdfg hijkl" must be represented as a single temporary word [asdfg hijkl] instead of [asdfg][ ][hijkl], which would be the case, because [ ] is in the dictionary

(TEMP,^" ",^DIGIT,^W)(^" ",^STAIL)=0;

(a temporary word, i.e., a word not found in the dictionary, must be followed by a blank space or the end of the sentence)

(^" ",^PUT,^SHEAD)(TEMP,^" ",^W)=0;

(a temporary word, i.e., a word not found in the dictionary, must be preceded by a blank space, a punctuation sign or the beginning of the sentence)

Determiners x Pronouns

There are many ambiguities in the dictionary between determiners and pronouns. The string "that", for instance, is represented in the dictionary as a demonstrative determiner ("that book") and as a demonstrative pronoun ("that is the book"). The following rules help differentiating them:

(D,^AFT)({PUT|STAIL})=0;

Determiners cannot come at the end of the sentence, except if their distribution is AFT (after) ("He said that", but "There are books enough")

(D)(BLK)(V)=0;

Determiners cannot come before a verb ("That is the book")

Auxiliary verbs x main verbs

Many auxiliary verbs may also play the role of main verbs: "He has done that" ("has" is an auxiliary) x "He has a car" ("has" is the main verb). The following rule helps differentiating them:

(AUX,^COP)(BLK)(^V,^[not])=0;

Auxiliary verbs which are not copula must be followed by a verb or the adverb [not]