{Utrecht University}

ELSNET/ELSE Workshop at LREC 1998
Steven Krauwer


                        FINAL PROGRAMME

                       FOR NL AND SPEECH.

        A workshop jointly organised by the European Network of
            Excellence in Language and Speech ELSNET and
             the EC Language Engineering-4 project ELSE

                held on Wednesday May 27, 9:00-13:00
                             at the


                         GRANADA, SPAIN

NB: The workshop has taken place. For copies of the proceedings
please contact ELRA. Continuation of the discussions will take
place electronically at the EAGLES-2 web site:

Right now, a generic framework for semi-automatic quantitative black-box evaluation of Speech and NLP systems does not exist in Europe. When confronted to a choice, developers and users prefer to ask the opinion of local experts as any other way of processing is either unrealistic or too costly. The LE-4 project ELSE aims at providing developers with a generic strategy and definition of the primary building blocks needed to implement a semi-automatic quantitative black-box evaluation scheme. Prominent speakers from the field have been invited to present papers addressing motivation, advantages, but also problems in connection with the implementation of such an evaluation scheme at an international scale. PROGRAMME: 09:00-09:10 OPENING (Steven Krauwer / Joseph Mariani) 09:10-09:30 "The Darpa experience" (Charles Wayne) 09:30-09:40 "Ethnology and sociology of evaluation" (Lynette Hirschman) 09:40-09:50 "The Aupelf experience" (Joseph Mariani) 09:50-10:00 "Experience in Grace tagging evaluation" (Patrick Paroubek) 10:00-10:10 "Experiences in bilingual text alignment evaluation and word sense disambiguation" (Jean Veronis) 10:10-10:20 "Best practice and evaluation" (Ole Bernsen / Laila Dybkjaer) 10:20-10:40 BREAK 10:40-10:50 "Confidence measures and evaluation" (Lin Chase) 10:50-11:00 "Evaluation within Eagles" (Maghi King) 11:00-11:10 "Technology vs User-evaluation" (Marc Blasband) 11:10-11:20 "Evaluation for better products" (Christian Dugast) 11:20-11:30 "Resources for evaluation" (Mark Liberman) 11:30-11:40 "Organising Parser Evaluation" (Richard Sutcliffe) 11:40-11:50 "Specification and infrastructure for text summarization evaluation" (Beth Sundheim) 11:50-12:00 BREAK 12:00-12:10 "ELSE" (Patrick Paroubek) 12:10-13:00 PANEL SESSION (Chair: Rob Gaizauskas) 13:00 CLOSING The selected topics include the multilingual nature of evaluation, lessons from the past (in Europe and the US), and the need for language resources. At the workshop the first intermediate results of the ELSE project will be presented and discussed. The workshop is very timely as it takes place when the EC's 5th Framework Programme is taking shape. It is clear that the availability of a European evaluation infrastucture can be an important factor in European R&D activities, and that it can only be successful if it is organized and implemented on a European scale. PROGRAMME COMMITTEE The workshop is coorganized by ELSNET and ELSE. The Programme Committee are the participants in the ELSE LE-4 project: Niels Ole Bernsen Jean-Pierre Chanod Khalid Choukri Robert Gaizauskas Steven Krauwer Isabelle de Lamberterie Joseph Mariani Klaus Netter Patrick Paroubek Martin Rajman Antonio Zampolli CONTACT Steven Krauwer Trans 10, 3512 JK Utrecht, The Netherlands Phone: +31 30 253 6050 Fax: +31 30 253 6000 Email: steven.krauwer@let.ruu.nl

Steven Krauwer (s.krauwer@uu.nl) Utrecht Institute of Linguistics UiL OTS
Phone +31 30 253 6050 Faculty of Humanities, Utrecht University
[Page last modified: 02-01-2014] Drift 10, 3512 BS Utrecht, Netherlands