Knowledge Base ValidationAs a member of the AI community, one of my long term goals is to develop software programs, or agents, that act intelligently--performing tasks faithfully on our behalf. As we ask these agents to operate in increasingly complicated environments, the task of correctly specifying the parameters for correct behavior becomes more and more difficult. As a result, agents that simulate human-level behavior within even modestly complicated domains are time-consuming and difficult to develop. One of the most significant bottlenecks in the development process is due to the fact that it is very difficult to ensure that an agent will behave appropriately in its environment without undergoing a lengthy and expensive proving process that involves manually observing the agent's behavior while it performs its task in a wide range of environmental circumstances. My research on KB Validation seeks to reduce the cost of this process by exploring automated methods to identify and report the similarities and differences between an agent's and a human's behavior. Once created, these reports can be assessed by human experts to isolate potential problem areas in the agent's knowledge. In this manner, we seek to identify semi-automated approaches that benefit from the speed of automation and the robustness of human-oversight. Related Papers Scott A. Wallace.
Identifying Incorrect Behavior: The Impact of Behavior Models on Detectable Error Manifestations. Fourteenth Conference on Behavior Representation in Modeling and Simulation (BRIMS-05). 2005.
Scott A. Wallace and John E. Laird.
Comparing Agents and Humans
Using Behavioral Bounding. International Joint Conference on
Artificial Intelligence (IJCAI-03). pp. 727-732. August 2003.
Scott A. Wallace.
Validating Complex Agent Behavior.
Doctoral Thesis. University of Michigan, Ann Arbor. August, 2003.
Scott A. Wallace and John E. Laird. Toward Automatic Knowledge Validation.
In Proceedings of the Eleventh Conference on Computer Generated Forces and Behavioral
Representation. pp. 447-456. May 2002.
Safe AgentsA natural extension to my work in Knowledge Base Validation, is to push validation capabilities into the agent itself. These agents, which I term self-assessment agents, are a particular type of meta-cognitive system. In contrast to standard agents which assign preferences to actions based solely on their own knowledge, self-assessment agents continuously monitor (or assess) their actions to ensure that external constraints are maintained. In essence, a self-assessment agent attempts to satisfy its own agenda while complying with a mandate given by its designer or some other external source. By performing real-time validation of the agent's behavior as it interacts with the environment, the self-assessment framework provides a natural mechanism for ensuring safe behavior at a relatively high-level of abstraction. However, the possibilities for the self-assessment framework extend far beyond validation and safety assurance. The same mechanisms used for ensuring safety could be used for tasks such as ensuring social compliance or producing variable degrees of autonomy. Related Papers
Scott A. Wallace. S-Assess: A Library for Self-Assessment. In Proceedings of the Fourth International Conference on Autonomous Agents and Multiagent Systems (AAMAS-05). pp. 256-263. Utrecht, The Netherlands. 2005.
Scott A. Wallace. Abstract Behavior
Representations for Self-Assessment. AAAI Spring Symposium on
Meta-Cognition in Computation (ASSMC 2005). AAAI
Technical Report SS-05-04. pp. 120-125. Palo Alto, CA. 2005.
Intelligent DebuggingIn this joint project with Orest Pilskalns, we are applying techniques from artificial intelligence to help create a better software debugger. This project targets hard to find errors; specifically those that occur only on a subset of the input. By using information about how the program flow changes in both successful and failure cases, we hope to identify the location of the problem, and thereby reduce debugging time. Students
Evaluation of AI ArchitecturesOne fascinating question within computer science is what makes a particular programming language more or less suitable for a particular task? Within the area of Artificial Intelligence, this question could well be rephrased as what type of software architecture is most appropriate for designing different types of intelligent agents? Some of my early work addressed this question with an comparative analysis between two rule-based agent architectures: Soar and CLIPS. Although both architectures have many commonalities (both in terms of the constructs defined by the language, and in run-time efficiency) we identified some salient differences between them. Most significantly, we observed a significant source of run-time overhead within the Soar architecture. Subsequently, this led to a new variant of the Soar architecture, Soar-Lite, designed specifically for speed. This variant of Soar is publicly available in Soar version 8.4 and above. Related PapersScott A. Wallace, John E. Laird, and Karen Coulter. Assessing the Run-Time Performance of Artificial Intelligence Architectures. NIST Workshop on Performance Metrics for Intelligent Systems (PerMIS 2000). Gaithersburg, MD. 2000.
Scott A. Wallace and John E. Laird. Toward a Methodology for AI Architecture Evaluation:
Comparing Soar And CLIPS. In Intelligent Agents VI--Proceedings of the Sixth International
Workshop on Agent Theories, Architectures and Languages (ATAL-99). N. R. Jennings and
Y. Lesperance, editors. Springer-Verlag, Berlin. pp. 117-131. 2000.
Scott A. Wallace, John E. Laird, and Karen Coulter. Examining the Resource Requirements of
Artificial Intelligence Architectures. In Proceedings of the Ninth Conference on Computer
Generated Forces and Behavioral Representation. pp. 73-82. May 2000.
|