F-measure: Difference between revisions
From UNLwiki
Jump to navigationJump to search
imported>Martins No edit summary |
imported>Martins No edit summary |
||
| Line 15: | Line 15: | ||
**The overall discrepancy is less than 0.5, WHERE | **The overall discrepancy is less than 0.5, WHERE | ||
***Discrepancy of relations is calculated by the formula: | ***Discrepancy of relations is calculated by the formula: | ||
(exceding_relations + missing_relations)/total_relations | |||
***Discrepancy of UW's is calculated by the formula: | ***Discrepancy of UW's is calculated by the formula: | ||
(exceding_UW + missing_UW)/total_UW | |||
***Overall discrepancy is calculated by the formula: | ***Overall discrepancy is calculated by the formula: | ||
((3*(exceding_relations+missing_relations))+(2*(exceding_UW+missing_UW)+(exceding_attribute+missing_attribute))/((3*total_relations)+(2*total_UW)+(total_attribute)) | |||
**WHERE | **WHERE | ||
***exceding_relations is the number of relations present in the actual output but absent from the expected output | ***exceding_relations is the number of relations present in the actual output but absent from the expected output | ||
| Line 30: | Line 30: | ||
***missing_attribute is the number of attributes<ref name="b"/> absent from the actual output but present in the expected output | ***missing_attribute is the number of attributes<ref name="b"/> absent from the actual output but present in the expected output | ||
***total_attribute is the sum of the total number of attributes<ref name="b"/> in the actual output and in the expected output | ***total_attribute is the sum of the total number of attributes<ref name="b"/> in the actual output and in the expected output | ||
*In [[NLizatioN]], when the difference between the actual result and the expected result is less than 30% | |||
**The difference between the actual and the expected result is calculated by the formula | |||
(exceding_words+missing_words)/(total_words) | |||
;**WHERE | |||
***exceding_words is the number of words present in the actual output but absent from the expected output | |||
***missing_words is the number of words absent from the actual output but present in the expected output | |||
***total_words is the sum of the total number of words in the actual output and in the expected output | |||
== References == | == References == | ||
<references /> | <references /> | ||
Revision as of 21:11, 1 April 2013
In the UNL System, the F-measure (or F1-score) is the measure of a grammar's accuracy. It considers both the precision and the recall of the grammar to compute the score, according to the formula
F-measure = 2 x ( (precision x recall) / (precision + recall) )
In the above:
- precision is the number of correct results divided by the number of all returned results
- recall is the number of correct results divided by the number of results that should have been returned
A result is considered "RETURNED" in the following cases:
- In UNLization, when the output is a graph (i.e., all the nodes are interlinked) made only of UW's (i.e., without natural language words)
- In NLization, when the output is a list of natural language words (i.e., without any UW).
A result is considered "CORRECT" in the following cases:
- In UNLization, when
- The discrepancy of relations between the actual and the expected output is less than 0.3; AND
- The discrepancy of UW's between the actual and the expected output is less than 0.3; AND
- The overall discrepancy is less than 0.5, WHERE
- Discrepancy of relations is calculated by the formula:
(exceding_relations + missing_relations)/total_relations
- Discrepancy of UW's is calculated by the formula:
(exceding_UW + missing_UW)/total_UW
- Overall discrepancy is calculated by the formula:
((3*(exceding_relations+missing_relations))+(2*(exceding_UW+missing_UW)+(exceding_attribute+missing_attribute))/((3*total_relations)+(2*total_UW)+(total_attribute))
- WHERE
- exceding_relations is the number of relations present in the actual output but absent from the expected output
- missing_relations is the number of relations absent from the actual output but present in the expected output
- total_relations is the sum of the total number of relations in the actual output and in the expected output
- exceding_UW is the number of UW's[1] present in the actual output but absent from the expected output
- missing_UW is the number of UW's[1] absent from the actual output but present in the expected output
- total_UW is the sum of the total number of UW's[1] in the actual output and in the expected output
- exceding_attribute is the number of attributes[2] present in the actual output but absent from the expected output
- missing_attribute is the number of attributes[2] absent from the actual output but present in the expected output
- total_attribute is the sum of the total number of attributes[2] in the actual output and in the expected output
- WHERE
- In NLizatioN, when the difference between the actual result and the expected result is less than 30%
- The difference between the actual and the expected result is calculated by the formula
(exceding_words+missing_words)/(total_words)
- WHERE
- exceding_words is the number of words present in the actual output but absent from the expected output
- missing_words is the number of words absent from the actual output but present in the expected output
- total_words is the sum of the total number of words in the actual output and in the expected output
References
- ↑ 1.0 1.1 1.2 For the sake of comparison, UW's appearing in the source position are considered to be different from UW's appearing in the target position of a relation. Scopes are ignored.
- ↑ 2.0 2.1 2.2 For the sake of comparison, attributes appearing in the source position are considered to be different from attributes appearing in the target position of a relation. Optional attributes (such as .@def and .@indef) are ignored.
