Invitation to Participate, Call for Papers and Contributions

Workshop on VERIFICATION, VALIDATION, AND TESTING OF LEARNING SYSTEMS

In conjunction with the Neural Information Processing Systems, NIPS-2004,
Whistler, BC, December, 2004


Workshop Motivation and Description

    Compared to the attention given to the development of new learning methods, our community has devoted only very little effort to developing princlpled approaches for (1) assessing the goodness of complex systems that contain learning components, (2) estimating the quality of the outputs of learned models in the context of the actual problem that needs to be addressed, (3) assessing online learning methods, (4) evaluating learning methods employed in safety-critical tasks, and for (5) understanding the tradeoffs between robustness and risk in making complex decisions.

    Learning has the potential to provide several key features to adaptive, autonomous, and large scale systems: adaptability to changing environments, capability of processing different types of sensor data, and addressing multiple objectives in parallel - to name just a few. In the meantime, most of these systems require a reliable deployment and operation. In other words, for most applications, in order to be deployed, learning components need to be proven as trustable to the users (engineers, designers, quality control specialists). Failures of these systems can occur and will occur, regardless of whether they contain learned or learning components. Therefore, questions such as "what are the tradeoffs for improving the quality of the outputs of a learning system in a certain region of the space?", or "what can be inferred (regarding future decisions) from observing the operation of a learning system?" have deep ramifications and, and if answered can result in learning technology having a deeper impact on newly developed systems.

    The workshop aims to explore the requirements of practical applications that make use of, or could benefit from learning methods - such as adaptive flight control systems, autonomous navigation, health management systems, robotics, and security.
    The purpose of the workshop is to bring together researchers and users of learning and adaptive systems and to create a forum for discussing recent advances in verification, validation, and testing of learning systems, to understand better the practical requirements for developing and deploying learning systems, and to inspire research on new methods and techniques for verification, validation, and testing.

    Topics of interest include, but are not limited to:

    - formal specification of learning requirements and verification
    - statistical testing and validation of learned models
    - metrics for the performance of learning systems
    - integration of learning components into adaptive control systems
    - statistical and logical inference for validation purposes
    - integration of online learning into large scale systems
    - learning for safety-critical applications
    - new approaches for machine learning software development
    - algorithms and tools for monitoring learning and adaptive systems
    - analysis of the robustness vs. risk tradeoff
    - validation and testing for sequential decision making
    - V&V for learning robots
    - new problems and applications that require principled assessment of learning


Workshop Format
    The workshop will have two sessions. Each session will start with an invited talk and will continue as a mix of position paper presentations and discussions.


Participation and Submissions
    To participate in the workshop, please send an e-mail message to Dragos Margineantu (dragos.d.margineantu@boeing.com) giving your name, affiliation, address, e-mail address, and a brief description of your reasons for wanting to attend.
    In addition, if you wish to present a position paper on one or more of the topics listed above, please see the instructions on the submissions page.

    If you have an issue or contribution that is not covered by the topics above, please contact Dragos Margineantu by e-mail to discuss your idea prior to submitting a position paper. The organizers will review the submissions with the goal of assembling a stimulating and exciting workshop. Attendence will be limited to 40 people, with preference given to people who are presenting position papers.


Important dates
  • Submission deadline: October 20, 2004
  • Notification of acceptance: November 1, 2004
  • Workshop to be held on December 17 or December 18, 2004


Organizers
  • Dragos Margineantu, Boeing, Math & Computing Technology
  • Johann Schumann, RIACS/NASA Ames
  • Pramod Gupta, QSS/NASA Ames
  • Michael Drumheller, Boeing, Math & Computing Technology
  • Roman Fresnedo, Boeing, Math & Computing Technology