FINAL PROGRAMME
TOWARDS A EUROPEAN EVALUATION INFRASTRUCTURE
FOR NL AND SPEECH.
A workshop jointly organised by the European Network of
Excellence in Language and Speech ELSNET and
the EC Language Engineering-4 project ELSE
held on Wednesday May 27, 9:00-13:00
at the
FIRST INTERNATIONAL CONFERENCE
ON LANGUAGE RESOURCES AND EVALUATION
GRANADA, SPAIN
NB: The workshop has taken place. For copies of the proceedings
please contact ELRA. Continuation of the discussions will take
place electronically at the EAGLES-2 web site:
http://www.cst.ku.dk/projects/eagles2/mail/maillist.html
Right now, a generic framework for semi-automatic quantitative
black-box evaluation of Speech and NLP systems does not exist in
Europe. When confronted to a choice, developers and users prefer
to ask the opinion of local experts as any other way of
processing is either unrealistic or too costly. The LE-4 project
ELSE aims at providing developers with a generic strategy and
definition of the primary building blocks needed to implement a
semi-automatic quantitative black-box evaluation scheme.
Prominent speakers from the field have been invited to present
papers addressing motivation, advantages, but also problems in
connection with the implementation of such an evaluation scheme
at an international scale.
PROGRAMME:
09:00-09:10 OPENING
(Steven Krauwer / Joseph Mariani)
09:10-09:30 "The Darpa experience"
(Charles Wayne)
09:30-09:40 "Ethnology and sociology of evaluation"
(Lynette Hirschman)
09:40-09:50 "The Aupelf experience"
(Joseph Mariani)
09:50-10:00 "Experience in Grace tagging evaluation"
(Patrick Paroubek)
10:00-10:10 "Experiences in bilingual text alignment evaluation and
word sense disambiguation"
(Jean Veronis)
10:10-10:20 "Best practice and evaluation"
(Ole Bernsen / Laila Dybkjaer)
10:20-10:40 BREAK
10:40-10:50 "Confidence measures and evaluation"
(Lin Chase)
10:50-11:00 "Evaluation within Eagles"
(Maghi King)
11:00-11:10 "Technology vs User-evaluation"
(Marc Blasband)
11:10-11:20 "Evaluation for better products"
(Christian Dugast)
11:20-11:30 "Resources for evaluation"
(Mark Liberman)
11:30-11:40 "Organising Parser Evaluation"
(Richard Sutcliffe)
11:40-11:50 "Specification and infrastructure for text
summarization evaluation"
(Beth Sundheim)
11:50-12:00 BREAK
12:00-12:10 "ELSE"
(Patrick Paroubek)
12:10-13:00 PANEL SESSION
(Chair: Rob Gaizauskas)
13:00 CLOSING
The selected topics include the multilingual nature of
evaluation, lessons from the past (in Europe and the US), and the
need for language resources. At the workshop the first
intermediate results of the ELSE project will be presented and
discussed.
The workshop is very timely as it takes place when the EC's 5th
Framework Programme is taking shape. It is clear that the
availability of a European evaluation infrastucture can be an
important factor in European R&D activities, and that it can only
be successful if it is organized and implemented on a European
scale.
PROGRAMME COMMITTEE
The workshop is coorganized by ELSNET and ELSE.
The Programme Committee are the participants in the ELSE LE-4 project:
Niels Ole Bernsen
Jean-Pierre Chanod
Khalid Choukri
Robert Gaizauskas
Steven Krauwer
Isabelle de Lamberterie
Joseph Mariani
Klaus Netter
Patrick Paroubek
Martin Rajman
Antonio Zampolli
CONTACT
Steven Krauwer
Trans 10, 3512 JK Utrecht, The Netherlands
Phone: +31 30 253 6050
Fax: +31 30 253 6000
Email: steven.krauwer@let.ruu.nl
