TESTING OF DEPLOYABLE LEARNING AND DECISION SYSTEMS

A workshop in conjunction with NIPS-2006
Whistler, British Columbia, December 8, 2006


  • Workshop notes

  • Workshop schedule
  • Call for papers/contributions
  • Submission format


    Although evaluation and testing has been an important consideration for some time, our community has devoted little effort towards developing principled approaches for assessing the efficacy of deployed systems. We need to be able to estimate:
    • the quality of the learning model in the context of the actual problem,
    • how safe our learning methods are in safety-critical tasks,
    • the effectiveness of online learning methods over an extended period of time,
    • the "stability" of online learning tasks, and
    • the tradeoffs between robustness and risk needed in making complex decisions.
    Aside from providing several key capabilities to adaptive and autonomous systems (such as adaptability to changing environments), we believe that learning and statistical can be successfully applied to the meta-problem of evaluating the above list of properties of a learning or decision system, before it is deployed.
    The first goal of this workshop is to explore the requirements and risks of developing learning algorithms and tools with the purpose of deploying them for tasks such as autonomous navigation, health management, flight control, decision support (interacting with human experts), etc. Next, we would like to understand how (possibly novel) learning methods can contribute to the process of testing and evaluating learning and decision systems and what techniques are needed to evaluate the decisions computed by complex (even human- computer) systems. Finally, this workshop will bring together researchers and users of learning and adaptive systems and create a forum for discussing recent advances in validation and testing of learning systems, to better understand the practical requirements for developing and deploying learning systems, and to inspire research on new methods and techniques for the testing and evaluation of learning.

    The workshop will try to offer support for covering workshop registration fees and parts of the travel expenses to graduate students and junior researchers who will be presenting at the workshop.


    Workshop Organizers
    • Dragos Margineantu, Boeing, Mathematics and Computing Technology (dragos.d.margineantu=at=boeing.com)- workshop chair
    • Chris Drummond, National Research Council (christopher.drummond=at=nrc-cnrc.gc.ca)
    • Kiri Wagstaff, NASA Jet Propulsion Laboratory (kiri.wagstaff=at=jpl.nasa.gov)