TOWARDS A EUROPEAN EVALUATION INFRASTRUCTURE
FOR NL AND SPEECH.
A workshop jointly organised by the European Network of
Excellence in Language and Speech ELSNET and
the EC Language Engineering-4 project ELSE
held on Wednesday May 27, 9:00-13:00
FIRST INTERNATIONAL CONFERENCE
ON LANGUAGE RESOURCES AND EVALUATION
NB: The workshop has taken place. For copies of the proceedings
please contact ELRA. Continuation of the discussions will take
place electronically at the EAGLES-2 web site:
Right now, a generic framework for semi-automatic quantitative
black-box evaluation of Speech and NLP systems does not exist in
Europe. When confronted to a choice, developers and users prefer
to ask the opinion of local experts as any other way of
processing is either unrealistic or too costly. The LE-4 project
ELSE aims at providing developers with a generic strategy and
definition of the primary building blocks needed to implement a
semi-automatic quantitative black-box evaluation scheme.
Prominent speakers from the field have been invited to present
papers addressing motivation, advantages, but also problems in
connection with the implementation of such an evaluation scheme
at an international scale.
(Steven Krauwer / Joseph Mariani)
09:10-09:30 "The Darpa experience"
09:30-09:40 "Ethnology and sociology of evaluation"
09:40-09:50 "The Aupelf experience"
09:50-10:00 "Experience in Grace tagging evaluation"
10:00-10:10 "Experiences in bilingual text alignment evaluation and
word sense disambiguation"
10:10-10:20 "Best practice and evaluation"
(Ole Bernsen / Laila Dybkjaer)
10:40-10:50 "Confidence measures and evaluation"
10:50-11:00 "Evaluation within Eagles"
11:00-11:10 "Technology vs User-evaluation"
11:10-11:20 "Evaluation for better products"
11:20-11:30 "Resources for evaluation"
11:30-11:40 "Organising Parser Evaluation"
11:40-11:50 "Specification and infrastructure for text
12:10-13:00 PANEL SESSION
(Chair: Rob Gaizauskas)
The selected topics include the multilingual nature of
evaluation, lessons from the past (in Europe and the US), and the
need for language resources. At the workshop the first
intermediate results of the ELSE project will be presented and
The workshop is very timely as it takes place when the EC's 5th
Framework Programme is taking shape. It is clear that the
availability of a European evaluation infrastucture can be an
important factor in European R&D activities, and that it can only
be successful if it is organized and implemented on a European
The workshop is coorganized by ELSNET and ELSE.
The Programme Committee are the participants in the ELSE LE-4 project:
Niels Ole Bernsen
Isabelle de Lamberterie
Trans 10, 3512 JK Utrecht, The Netherlands
Phone: +31 30 253 6050
Fax: +31 30 253 6000