Workshop on VERIFICATION, VALIDATION, AND TESTING OF LEARNING SYSTEMS
In conjunction with the Neural Information Processing Systems, NIPS-2004,
Learning has the potential to provide several key features to adaptive,
autonomous, and large scale systems: adaptability to changing
environments, capability of processing different types of sensor data,
and addressing multiple objectives in parallel - to name just a few.
In the meantime, most of these systems require a reliable deployment and
operation.
In other words, for most applications, in order to be deployed,
learning components need to be proven as trustable to the users
(engineers, designers, quality control specialists).
Failures of these systems can occur and will occur, regardless of
whether they contain learned or learning components. Therefore,
questions such as "what are the tradeoffs for improving the quality of
the outputs of a learning system in a certain region of the space?",
or "what can be inferred (regarding future decisions) from observing
the operation of a learning system?" have deep ramifications and, and
if answered can result in learning technology having a deeper impact
on newly developed systems.
The workshop aims to explore the requirements of practical
applications that make use of, or could benefit from learning methods
- such as adaptive flight control systems, autonomous navigation,
health management systems, robotics, and security.
Topics of interest include, but are not limited to:
If you have an issue or contribution that is not covered by the topics above, please contact Dragos Margineantu
by e-mail to discuss your idea prior to submitting a position paper.
The organizers will review the submissions with the goal of assembling
a stimulating and exciting workshop. Attendence will be limited to 40
people, with preference given to people who are presenting position
papers.
Whistler, BC, December, 2004
Workshop Motivation and Description
Compared to the attention given to the development of new learning
methods, our community has devoted only very little effort to developing
princlpled approaches for (1) assessing the goodness of complex systems
that contain learning components, (2) estimating the quality of the outputs of
learned models in the context of the actual problem that needs to be addressed, (3)
assessing online learning methods, (4) evaluating learning methods
employed in safety-critical tasks, and for (5) understanding the
tradeoffs between robustness and risk in making complex decisions.
The purpose of the workshop is to bring together researchers and users
of learning and adaptive systems and to create a forum for discussing
recent advances in verification, validation, and testing of learning
systems, to understand better the practical requirements for developing
and deploying learning systems, and to inspire research on new methods
and techniques for verification, validation, and testing.
- formal specification of learning requirements and verification
- statistical testing and validation of learned models
- metrics for the performance of learning systems
- integration of learning components into adaptive control systems
- statistical and logical inference for validation purposes
- integration of online learning into large scale systems
- learning for safety-critical applications
- new approaches for machine learning software development
- algorithms and tools for monitoring learning and adaptive systems
- analysis of the robustness vs. risk tradeoff
- validation and testing for sequential decision making
- V&V for learning robots
- new problems and applications that require principled assessment of learning
Workshop Format
The workshop will have two sessions. Each session will start with an invited talk and will continue as a mix of
position paper presentations and discussions.
Participation and Submissions
To participate in the workshop, please send an e-mail message to Dragos Margineantu (dragos.d.margineantu@boeing.com) giving your name, affiliation, address, e-mail address,
and a brief description of your reasons for wanting to attend.
In addition, if you wish to present a position paper on one or more of
the topics listed above, please see the instructions on the
submissions page.
Important dates
Organizers