| Abstract: | High-performance, expert-level computer systems require that the expert system prototype be continously evaluated during its development. Expert system validation—that is, testing systems to ascertain whether they achieve acceptable performance levels—has with few exceptions been ad hoc, informal, and of dubious value. This paper describes a variant of the Turing Test technique that may be used to formalize the validation process. A microcomputer-based prototype expert system, the Hazardous Location Analyst (HLA), in the accident location analysis domain was used as a case study of the technique. Turing tests provide a blind method for multiple experts to assess expert system performance qualitatively, and provide a means of determining reasonable performance levels for the particular domain. The HLA has been modified based upon findings from the case study work done previously. The improved prototype was implemented in the City of Greensboro, North Carolina, and a set of 10 case studies are analyzed using the HLA, and by a Greensboro traffic engineer. The results were summarized in identical form and mailed to four traffic engineers outside of Greensboro. Their ratings of case study findings were summarized, and the results used to find a reasonable performance level for this application and to assess the performance of the HLA. The consistency of the expert ratings was also assessed. It was concluded that there exists excellent consistency among experts with regard to human performance, but not with regard to the HLA. This paper demonstrates that the Turing Test as a validation methodology provides an objective, quantitative way to assess system performance. |
