Download MaltEval 0.1

User Guide for MaltEval 0.1

This document is a concise desciption of MaltEval, an evaluator with a graphical user interface for comparing corpus data with tagged, parsed or chunked natural language texts, in MaltEval called "treebank" and parsed data, respectively. The evaluator takes two files as input, one containing some processed data, and one containing the correct data. Depending on the information available in the processed file and in the correct file, the applicable evaluators (i.e. for tagging, dependency parsing and chunking) are enabled. The following sections will present the input formats that MaltEnv understands, as well as its available features.

Input formats

The current version of MaltEval understands two file formats, a tab separated format, Malt-TAB, and a XML-based format, Malt-XML. Both formats contain the same kind of information, but the difference is that the XML-format includes meta-information such as tagsets. For the tab separated format, that kind of auxiliary data must therefore be provided in addition to the actual data. Malt-XML and Malt-TAB are both used for representing treebank material in a special kind of dependency format. It is based on the following simple principles of representation: For both Malt-XML and Malt-TAB, the only common compulsory attribute for the word-element is form. Malt-XML also requires that each word has the attribute id (where the first word in each sentence has id=1). However, in order to enable any kind of evaluation one or more of the other four needs to be available in the two input files. Obviously, if you want to evaluate the part-of-speech tags the postag-attribute must be set for all words in both files. For evaluation of unlabeled dependency parsing all head-attributes must be set, and for labeled dependency evaluation the deprel-attributes are needed in addition to the head-attributes. The representation is based on the assumption that each word has at most one head. By convention, root words are represented by the value "0" for the head-attribute and the value "ROOT" for the deprel-attribute. The head-attribute is an abolute pointer to a word in the same sentence. For chunking, the evaluator only needs the chunk-attributes. For word that are inside more than one chunk, the token "|" is used as delimiter. By convention, the letter "O" denotes that the word is not part of any chunk, and words beginning with a new chunk must end with the letter "B", whereas words inside a chunk must end with the letter "I" in order for the evaluation work properly. See the example below.

Malt-XML

A dependency tree for the Swedish sentence "Sjukdomen tuberkulos fortsätter att minska stadigt i Sverige - men bara bland dem som är svenskfödda." can be represented as follows: 
		<sentence id="3" user="" date="">
			<word id="1" form="Det" lemma="den" postag="pn.neu.sin.def.sub/obj" head="2" deprel="SUB" chunk="NPB"/>

			<word id="2" form="innebär" lemma="innebära" postag="vb.prs.akt" head="0" deprel="ROOT" chunk="VCB"/>
			<word id="3" form="bl." lemma="bland" postag="pp" head="2" deprel="ADV" chunk="ADVPB"/>
			<word id="4" form="a." lemma="annat" postag="nn.neu.sin.ind.nom" head="3" deprel="ID" chunk="ADVPI"/>
			<word id="5" form="att" lemma="att" postag="sn" head="2" deprel="OBJ" chunk="O"/>
			<word id="6" form="endast" lemma="endast" postag="ab" head="7" deprel="ADV" chunk="ADVPB"/>
			<word id="7" form="en" lemma="en" postag="dt.utr.sin.ind" head="8" deprel="DET" chunk="NPB"/>
			<word id="8" form="skatteskala" lemma="skatteskala" postag="nn.utr.sin.ind.nom" head="9" deprel="SUB" chunk="NPI"/>
			<word id="9" form="kommer" lemma="komma" postag="vb.prs.akt" head="5" deprel="UK" chunk="VCB"/>
			<word id="10" form="att" lemma="att" postag="ie" head="9" deprel="VC" chunk="VCI"/>

			<word id="11" form="finnas" lemma="finna" postag="vb.inf.sfo" head="10" deprel="IM" chunk="VCB"/>
			<word id="12" form="för" lemma="för" postag="pp" head="11" deprel="ADV" chunk="PPB"/>
			<word id="13" form="beräkning" lemma="beräkning" postag="nn.utr.sin.ind.nom" head="12" deprel="PR" chunk="NPB|PPI"/>
			<word id="14" form="av" lemma="av" postag="pp" head="13" deprel="ATT" chunk="NPI|PPB|PPI"/>
			<word id="15" form="statlig" lemma="statlig" postag="jj.pos.utr.sin.ind.nom" head="16" deprel="ATT" chunk="APB|NPI|PPI|PPI"/>
			<word id="16" form="inkomstskatt" lemma="inkomstskatt" postag="nn.utr.sin.ind.nom" head="14" deprel="PR" chunk="NPB|NPI|PPI|PPI"/>
			<word id="17" form="." lemma="." postag="mad" head="2" deprel="IP" chunk="O"/>
		</sentence>
The tagsets used for parts-of-speech, dependency relations and chunk relations must be specified in the header of the XML document. An example document can be found here. Worth mentioning is that in Malt-XML, more attributes can be incorporated in the word element (such as the lemma-attribute), but they will be ignored by MaltEval.

Malt-TAB

The corresponding sentence in Malt-TAB looks like this: 
Det	pn.neu.sin.def.sub/obj	2	SUB	NPB
innebär	vb.prs.akt	0	ROOT	VCB
bl.	pp	2	ADV	ADVPB
a.	nn.neu.sin.ind.nom	3	ID	ADVPI
att	sn	2	OBJ	O
endast	ab	7	ADV	ADVPB
en	dt.utr.sin.ind	8	DET	NPB
skatteskala	nn.utr.sin.ind.nom	9	SUB	NPI
kommer	vb.prs.akt	5	UK	VCB
att	ie	9	VC	VCI
finnas	vb.inf.sfo	10	IM	VCB
för	pp	11	ADV	PPB
beräkning	nn.utr.sin.ind.nom	12	PR	NPB|PPI
av	pp	13	ATT	NPI|PPB|PPI
statlig	jj.pos.utr.sin.ind.nom	16	ATT	APB|NPI|PPI|PPI
inkomstskatt	nn.utr.sin.ind.nom	14	PR	NPB|NPI|PPI|PPI
.	mad	2	IP	O
The example document can be found here. Sentence splits in Malt-TAB are represented by a blank line as can be seen in the example document. As mentioned, the tagsets used for parts-of-speech, dependency relations and chunk relations are not specified in the Malt-TAB format and must therefore be specified in auxiliary files, one for each tagset. For the example file, the parts-of-speech file can be found here, the dependency relation file here and the chunk relation file here. The only compulsory attribute for Malt-TAB is the first column, namely the word form. The other attributes are all optional, but the order among the included attributes cannot be altered. For example, if both the part-of-speech and the head-attributes are included, then the part-of-speech column must be to the left of the head column. Also, if the head attribute is present the the dependency relation attribute must also be present. This is the total order:

form (required) < postag (optional) < head (optional) < deprel (optional) < chunk (optional)

And since MaltEval does not try to "guess" which attributes are included in the file when you choose to read a Malt-TAB file, you are asked by the program to specify this in a dialog box.

The order of the tags in the tagset files is irrelevant, but it is crucial that all tags in the input files are enumerated in the tagset file. The same is true if the input format in Malt-XML, i.e. the order of the tags in the header is not important but all tags for all words must be enumerated in the header. The evaluator will otherwise complain and the evaluation will be disabled.

The menus

The menu system contains five menus. The most important menus are described below.

The File menu

From File menu it is possible to read the parsed data and the treebank data. From this menu, it is also possible to export the computed results to a file. The name of the export file is by default "output", but it can be changed from the Edit menu. From the File menu you can also exit MaltEval.

When two input files have been successfully imported and before any evaluation is enabled, they conform to each other in a number of respects. So before it is possible to perform any kind of evaluation, MaltEval ensures that the files contain the same text and have the same tagsets. The two files must have the same number of sentences, and each sentence in one file must have the same number of words as the corresponding sentence in the other file. Moreover, the attribute form for each word must be equivalent to the corresponding word in the other file. It is also important that the tagsets in for both files are equivalent, i.e. have the same size and contain the same tags (although the order of the tags in the input files may differ).

NB:Very large input files can cause OutOfMemoryException. If this happens, more memory can be allocated by using the flag -Xmx<N>m when starting MaltEval. (For example, java -jar -Xmx500m MaltEval.jar sets the maximum memory allocation to 500 MB.)

The Edit menu

From this menu you can edit four different settings which are common for tagging, parsing and chunking. It is possible to exclude one or more parts-of-speech and/or dependency relations from the evaluation. The exclusion of parts-of-speech is applicable to both the tagging evaluation and the parsing evaluation, whereas the exclusion of dependency relations only applies to the parsing evaluation. No parts-of-speech or dependency relations can be excluded for the chunking evaluation.

From the Edit menu it is possible to specify the minimal and maximal sentence length. All sentences below the lower limit and above the upper limit are discarded from the evaluation. In order to avoid being forced to specify an extremely high value for the upper limit if you want to include all sentences, the value "0" represents infinity. By default the values are set so that all sentences are included. This feature works for all three evaluation options.

The Edit menu also includes the possibility to change the file name of the export file.

The View menu

This menu contains tools for looking at the content of the input files in different aspects, as well as settings such as the excluded parts-of-speech and dependency relations. If an input file has been successfully read, several different viewing options are enabled. It is for example possible to view the imported treebank in a visualization tool for treebanks (e.g. View->Parer Data->Visualize trees). In addition to this feature, the view menu includes tools for viewing the input files as tab separated text similar to the Malt-TAB format.

Evaluation

This section briefly presents the options available when two files successfully have been imported and matched so that they contain the same text and the same tagsets.

Tagging

If all words in both input files have a postag-attribute, the radio button for tagging evaluation is enabled. By clicking on this button you will be provided with two options: If the first checkbox is checked, the evaluator will compute the overall precision and recall by first computing precision and recall for each individual sentence and the computing the mean value over all sentences. If is is not checked, precision and recall will be computed dividing the total number of correctly tagged words with the total number of words. The other option is connected to the parts-of-speech excluded in the dialog box accessible form the menu Edit->Exclude Postags... . If the checkbox is checked and one or more parts-of-speech are selected in that dialog box, then these will be excluded in the evaluation.

Parsing

The radio button for parsing evaluation includes the same checkboxes as the tagging evaluation. The only difference is that it is possible to exclude one or more dependency relations in addition to excluding parts-of-speech. The parsing evaluator also provides two groups of radio buttons. The first one lets the user select whether the result shall be presented per part-of-speech tag or per dependency relation, or if the attachment score computed per sentence shall be displayed. The second group makes it possible to base the evaluation on the head-attribute alone, on the deprel-attribute alone, or on both head and dependency relation. If the evaluation is based only on the head-attribute, this is the same as unlabeled evaluation, and if the evaluation is based on both head and dependency relation, this is the same as labeled evaluation.

Chunking

The chunker does not have any options. Evaluation can therefore only be computed using the default setting. The result is presented by merging chunks of the same kind, i.e. the last letter is discarded in the presentation of the evaluation. An exception is the chunk tag "O", which is assigned words not part of any chunk.