The MaltEval webpage

MaltEval is an evaluation tool for dependency parsers. Dependency parsing has been the shared task of CoNLL for two years, 2006 and 2007. MaltEval can, among other data formats, handle the data format used in these two CoNLL shared tasks. It is to a large extent adapted to the evaluation scripts eval.pl and eval07.pl provided by the organizers, but is more flexible and contains many addional features to facilitate evaluation. It is freely available for download and to use, but it comes with no guarantees. Here is a list of some of its features:

• Can not only evaluate single parsed dependency files, but also automatically evaluate multiple files as well as automatic evaluation of cross validation experiments.
• Supports three different dependency based file formats: CoNLL, MaltXML and MaltTab.
• It comes with default evaluation settings, which can be manipulated through flags or files containing the evaluation settings.
• A large number of strategies for grouping the data. Besides the default attachment score evaluation (correct tokens / number of tokens), there are currently 16 grouping strategies, such as ArcLength, ArcDepth, BranchingFactor, ArcProjectivity and Frame.
• Support for excluding sentences longer and/or shorter than specified lengths.
• Support for excluding various tokens from the evaluation based on e.g. word form or part-of-speech.
• Flexibility to format the evaluation output, such as disabling or enabling detailed output
• Possibility to automatically produce confusion matrices.
• Possibility to automatically test statistically significance.
• And more...

Download MaltEval

• Download MaltEval (compiled for JRE 1.5): MaltEval-20081119.zip (MaltEval 1.0 beta)
• Is MaltEval something you may find useful? Have a look at the latest user guide here.
• MaltEval was presented at LREC'08, see the paper here.
• Bug reports and suggestions of improvements can be sent to jens.nilsson-at-msi.vxu.se.