Subject: the evaluation of parsing systems

the evaluation of parsing systems a workshop jointly organised by the cec language engineering 1 projects sparkle and ecran to be held at the first international conference on language resources and evaluation granada , spain , 26 may 1998 this workshop will provide a forum for researchers interested in the development and evaluation of natural language grammars and parsing systems , and in the creation of syntactically annotated reference corpora . organisers : john carroll , roberto basili , nicoletta calzolari , robert gaizauskas , gregory grefenstette workshop scope and aims the aim of this workshop is to provide a forum for discussion of evaluation methods for parsing systems , and proposals for the development of syntactically annotated language resources . with increased attention to evaluation of component technology in language engineering , evaluation of parsing systems is rapidly becoming a key issue . numerous methods have been proposed and while one , the parseval / penn treebank scheme , has gained wide usage , this has to some extent been due to the absence of workable alternatives rather than to whole-hearted support . parseval / ptb evaluation has several limitations and drawbacks , including a commitment to a particular style of grammatical analysis , and oversensitivity to certain innocuous types of misanalysis while failing to penalise other common types of more serious mistake . also , the original published description of the scheme - - and the evaluation software widely distributed as a follow-up to it - - is specific to the english language . it may be that there are currently no alternative more workable schemes or proposals , but this needs to be more fully discussed : this workshop will provide an opportunity for such a debate . this workshop is particularly timely given the large number of cec language engineering projects that involve parsing in one form or another and which need to evaluate and share the results of their efforts . parsing is an essential part of many larger applications , such as information extraction , which have gained in importance over the last few years . often in such systems , the strength of the parser and grammar has a direct effect on the desired results , and thus achieving good results rests on being able to determine and improve weaknesses in the parser / grammar . without a reliable parser evaluation method this cannot be done effectively . a parsing evaluation workshop is also appropriate at this time given the imminent creation of large-scale syntactically annotated resources for european languages . contributions from those involved in such activities are welcomed , so as to improve communication between the resource construction and the resource utilisation communities . this should ensure that the resources constructed are maximally useful to the general language engineering community . the organisation of this workshop brings together two european language engineering projects which are closely related and whose partners share similar research interests : sparkle and ecran . the organisers solicit contributions from the general community on the following topics : - descriptions of generic syntactic annotation schemes - methodologies and metrics for parsing system evaluation - reports and analyses of the results of utilising particular parser evaluation schemes - description / analysis / experience of language-dependent ( especially for languages other than english ) and task-dependent syntactic annotation schemes programme committee roberto basili gregory grefenstette ted briscoe mark hepple nicoletta calzolari tony mcenery john carroll maria teresa pazienza roberta catizone paola velardi robert gaizauskas yorick wilks paper submission papers should not exceed 4000 words or 10 pages . submission may be in either hard copy or electronic form . the submission deadline is february 15th , 1998 . hard copy submission : three copies of the paper should be sent to : dr john carroll cognitive and computing sciences university of sussex brighton bn1 9qh uk electronic submission : electronic submission may be in either self-contained latex , postscript , or rtf formats , to john . carroll @ cogs . susx . ac . uk . for each submission - - whether hard copy or electronic - - a separate plain ascii text email message should be sent to john carroll , containing the following information : # name : name of first author # title : title of the paper # pages : number of pages # note : any relevant instructions # keys : keywords # email : email of the first author # abstr : abstract of the paper . . . . . . important dates paper submission deadline ( hard copy / electronic ) february 15th notification of acceptance march 10th camera - ready papers due april 10th workshop may 26th conference information general information about the conference is at : < http : / / www . icp . inpg . fr / elra / conflre . html > see < http : / / ceres . ugr . es / ~ rubio / elra . html > for information on local arrangements specific queries about the conference should be directed to : lrec secretariat facultad de traduccion e interpretacion dpto . de traduccion e interpretacion c / puentezuelas , 55 18002 granada , spain tel : + 34 58 24 41 00 - fax : + 34 58 24 41 04 reli98 @ goliat . ugr . es
