Subject: the evaluation of parsing systems

call for participation the evaluation of parsing systems granada , spain , 26 may 1998 this workshop is part of the first international conference on language resources and evaluation at the university of granada , may 26th - 30th 1998 ( see < http : / / ceres . ugr . es / ~ rubio / elra . html > for details and how to register ) . this workshop will provide a forum for researchers interested in the development and evaluation of natural language grammars and parsing systems , and in the creation of syntactically annotated reference corpora . organisers : john carroll , roberto basili , nicoletta calzolari , robert gaizauskas , gregory grefenstette accepted papers a survey of parser evaluation methods john carroll , ted briscoe university of sussex & university of cambridge , uk evaluating a robust parser for italian language roberto basili , maria teresa pazienza , fabio massimo zanzotto universita ' di roma tor vergata , rome , italy evaluation of the syntactic analysis component of an information extraction system for german thierry declerck , judith klein , guenter neumann dfki , saarbruecken , germany chunking italian . linguistic and task-oriented evaluation stefano federici , simonetta montemagni , vito pirrelli ilc-cnr pisa , italy modifying existing annotated corpora for general comparative evaluation of parsing rob gaizauskas , mark hepple , chris huyck university of sheffield , uk dependency - based evaluation of minipar dekang lin university of manitoba , canada evaluating parses for spoken language dialogue systems wolfgang minker , lin chase limsi , france corpus - based parse pruning sonja mueller - landmann ibm , heidelberg , germany the tosca parsing system reviewed nelleke oostdijk katholieke universiteit nijmegen , the netherlands grammar & parser evaluation in the xtag project srinivas bangalore , anoop sarkar , christine doran , beth ann hockey at&t labs - research & ircs , university of pennsylvania , usa workshop scope and aims the aim of this workshop is to provide a forum for discussion of evaluation methods for parsing systems , and proposals for the development of syntactically annotated language resources . with increased attention to evaluation of component technology in language engineering , evaluation of parsing systems is rapidly becoming a key issue . numerous methods have been proposed and while one , the parseval / penn treebank scheme , has gained wide usage , this has to some extent been due to the absence of workable alternatives rather than to whole-hearted support . parseval / ptb evaluation has several limitations and drawbacks , including a commitment to a particular style of grammatical analysis , and oversensitivity to certain innocuous types of misanalysis while failing to penalise other common types of more serious mistake . also , the original published description of the scheme - - and the evaluation software widely distributed as a follow-up to it - - is specific to the english language . it may be that there are currently no alternative more workable schemes or proposals , but this needs to be more fully discussed : this workshop will provide an opportunity for such a debate . this workshop is particularly timely given the large number of cec language engineering projects that involve parsing in one form or another and which need to evaluate and share the results of their efforts . parsing is an essential part of many larger applications , such as information extraction , which have gained in importance over the last few years . often in such systems , the strength of the parser and grammar has a direct effect on the desired results , and thus achieving good results rests on being able to determine and improve weaknesses in the parser / grammar . without a reliable parser evaluation method this cannot be done effectively . a parsing evaluation workshop is also appropriate at this time given the imminent creation of large-scale syntactically annotated resources for european languages . contributions from those involved in such activities are welcomed , so as to improve communication between the resource construction and the resource utilisation communities . this should ensure that the resources constructed are maximally useful to the general language engineering community . the workshop is jointly organised by the cec language engineering 1 projects sparkle and ecran programme committee roberto basili gregory grefenstette ted briscoe mark hepple nicoletta calzolari tony mcenery john carroll maria teresa pazienza roberta catizone paola velardi robert gaizauskas yorick wilks
