expert problem solving in complex human-machine systems?
by
Asbjørn Følstad
Contents
Introduction
Expert problem solving
EID as support of expert problem solving
The operator as expert problem solver
Characterizing the expert problem solver
Case-based reasoning as an alternative framework for understanding expert problem solving
Consequences of adopting the framework of case-based reasoning
Consequences for the SRK taxonomy
An explanation of the results of "Supporting operator problem solving through ecological interface design"
Extended possibilities of supporting operator problem solving through interface design
The evolving of complex technology may be said to have put great strains on the users. Increasingly complex systems are seen to put increasingly complex demands on the people serving, or being served by them. There has also been a tendency to assume human users and operators to accommodate themselves to technology, rather than to accommodate technology to the abilities of the users. However, over the last couple of decades there has been an increasing awareness of the possible gains of suiting complex systems to the psychological resources of their human operators. The rationale of this being a simple one with strong intuitive appeal: Less strain on the operators, increase in productivity, decline in errors. Man-machine studies, as a field of research, has developed as a consequence of this recognition of the need to tailor technology to its users. In the field of man-machine research, the relationship between human operators and complex machine systems has been of central interest. This particular relationship has been treated thoroughly by Jens Rasmussen and Kim Vicente (Vicente & Rasmussen, -88/-90/-92; Vicente, -95/-96; Rasmussen, -90/-95). The present paper is written in response to their work.
Vicente and Rasmussen address the field of complex human-machine systems as a problem of developing adequate interfaces between the human operator and the work domain (Vicente & Rasmussen, -88). Their prime example of a complex work domain being nuclear power plants, and their prototypical human operator being the expert operators of the plant. Focusing in on the interface as the main area of interest is to be taken as an expression of understanding operator and system as mutually dependent. The performance of the one depending on the performance of the other. When it comes to complex work domains, the main area of interest is naturally the performance of the system. The approach of Vicente and Rasmussen, however, accentuates the necessity of being sensitive to the abilities of the operator. This in order to ensure safe and effective system control. Information about system status is a predicament for operator system control. This information is to be brought to the operator by the system interface. But what constitutes adequate information is not just dependent on the constitution of the system to be controlled, it is also dependent on the constitution of the controlling operator. This is one of the basic issues in developing adequate interfaces.
In complex human-machine systems, at least as presented in this paper, the operator of the system is expert. The operator is understood as being expert in the system total, or in a system sub-domain. The operator assignment is the routine running of the system on a daily basis, but it may also extend to extraordinary operating in particular circumstances or during system breakdown. Information of system status is provided through the system interface, and the operator also controls the system by its interface. The assignment of an operator may be seen as a continuous process of expert problem solving based on the information provided in the system interface. Following this, operator performance is seen as depending on what information presented in the interface, and how this information is presented. The issue of interface design may be rephrased as ‘how to optimally support expert problem solving in complex human-machine systems?’
The work of Vicente and Rasmussen can well be read as an answer to this question. Their concept of ecological interface design (EID) is to be thought of as a possible way of increasing interface support of expert problem solving. What will be argued in the present paper is that their answer may not be a complete one. This, of course, is no radical argument, given that any process of making things better hardly will reach its final conclusion. What is of a more radical nature is the proposal that what may lead to an improvement in Vicente and Rasmussen’s understanding of the system-operator relationship, is a fundamental change in their theoretical assumptions. What is proposed is that the basic theoretical framework for understanding expert problem solving in Vicente and Rasmussen is less than optimal, and that discussing this framework will make visible the need for it’s replacement with an alternative framework. Through adopting an alternative framework for understanding expert problem solving, it may be shown that some changes and extension of the conclusions of Vicente and Rasmussen are needed to further optimize the support for expert problem solving as provided through the theory of ecological interface design. A change of framework for understanding expert problem solving may also provide a vehicle for a different understanding of findings like those reported by Vicente et. al. from experimental work on DURESS (Dual Reservoir System Simulation) (Vicente, -95). Findings that otherwise seem ‘surprising’ given the present framework.
In the present paper it will be argued that the framework assumed by Vicente and Rasmussen for understanding expert problem solving is grounded in problem-space theory as evolved from the work of Newell and Simon (Newell & Simon, -72). This being particularly evident in Vicente and Rasmussen’s use of the concepts of rules, constraints, and mental models. The framework of problem-space theory will be held as representing an inadequate understanding of expert problem solving in humans, following which basic assumptions held by Vicente and Rasmussen may need rephrasing. The framework proposed as better suited for understanding expert problem solving than problem-space theory, will be that of case-based reasoning as presented by Roger Schank in ‘Dynamic memory’ (Schank, -82) and ‘Inside case-based reasoning’ (Riesbeck & Schank, -89).
The discussions of the present paper will naturally take a starting point in, and include perspectives on, the central question of interface design and expert problem solving:
With a basis in this most basic interface design issue of complex work domains, the central themes of discussion in the present paper will be as follows:
The discussion is meant as an elaboration on the work of Vicente and Rasmussen. The starting point is the support of expert problem solving provided in EID. The conclusion will be that of an extension and modification of this theoretical position.
For a thorough treatment of the above questions this paper will proceed in a stepwise manner. First a section will be devoted to a general presentation of expert problem solving, based on works of Green and Gilhooly (Green & Gilhooly, -92/Gilhooly, -88), Chi et. al. (Chi, -81), Larkin (Larkin, -80), and others. In the second section a presentation of EID as supporting expert problem solving is given. This section will include a general presentation of the theory of EID, assumptions held about the cognitive preferences and abilities of the operator, and the results of an empirical study on operator problem solving executed by Vicente et. al. (Vicente, -95). The third and fourth sections are presentations of the frameworks of problem-space theory and case-based reasoning respectively. The concept of mental models (Eberts, -94; Rasmussen, -90; Norman, -83/-90) is included in the third section as one possible expression of problem-space theory. In the fourth section it will be argued that general results on expert problem solving are well taken care of in the framework of case-based reasoning, and that this framework may substitute the framework of problem-space theory on expert problem solving in humans. In the final section the consequences for the theory of EID by the suggested change of framework will be discussed. One possible consequence is proposed to be a replacement of the SRK taxonomy of Vicente and Rasmussen as theoretical justification for interface design principles. The SRK taxonomy may be replaced with a framework supported in the theory of case-based reasoning. Other consequences may be the explanation of ‘surprising’ empirical results in a study on operator problem solving by Vicente et. al. (Vicente, -95), and the suggestion of novel ideas for interface designs supporting operator problem solving. One of these being the proposal of an external library of cases in support of a case-based reasoning expert. This proposal is reminiscent of a ‘data base of experiences’ suggested in the work on the BISP project by Gunnar Wille at the department of industrial economics and technology management, NTNU (presented at NTNU October, -98).
In this first section, the aim is to give a general introduction to the research area of expert problem solving. This introduction is hoped to give an understanding of what may said to be known in general about expert problem solving, and also present some empirical studies supporting this knowledge. At this point in the paper, no presentation of theories of expert problem solving will be given. The theoretical issues will be amply exposed in following sections.
The reason for this general presentation is that (1) it will be of help in understanding expert problem solving in the particular context of complex human-machine systems, and (2) it will serve as a point of departure for discussing the different frameworks for conceiving expert problem solving as outlined in problem-space theory and case-based reasoning. Principles for expert problem solving presented in this section will be utilized in both the presentation of the work of Vicente and Rasmussen on EID, in the discussion of the adequacy of frameworks, and in the general discussion of consequences for EID on adopting the framework of case-based reasoning. The core of the presentation relies on the review on expertise research presented by Alison Green and Kenneth Gilhooly in ‘Empirical advances in expertise research’ (Green & Gilhooly, -92).
In the words of Green and Gilhooly "an ‘expert’ is typically defined as an individual who possesses a large body of knowledge and procedural skill" (Green & Gilhooly, -92, p.46). It is important to recognize that this body of knowledge and skill is tied up to a particular work domain. An expert in the domain of solving physics problems is unlikely to display a similar level of proficiency in solving economics problems (unless, of course, this person happens to be an expert in both domains). In other words, expertise is to be thought of as domain specific. Expertise is traditionally studied in accordance with a basic expert-novice distinction, where the concepts of expert and novice are to be seen as the end points on a continuum, but there is also work, like that of deGroot on the expertise in chess, where the basic distinction is one between expert and less expert performance.
The domain specificity of expertise may be taken as self-evident. Even so, an experimental study by Gilhooly et. al. clearly demonstrates the degree of specificity expertise may have (Gilhooly, -88). The theme of the study was the effect of expertise in map reading on recall of presented map material. Experts were supposed to be superior in recall to non-experts. Subjects were classified as expert or non-expert on the basis of self-reported earlier experience with topographical contour maps, and a simple test in the use of this kind of maps. The subjects were then assigned to learn an unfamiliar map in a specified amount of time, after which they were presented with a recall task. The map to be learnt was either a planimetric map (like city maps, displaying houses, parks, roads, etc.) or a contour map (like the map of a mountain area, displaying contour lines as a presentation of topographical features). The results of the recall task were quite extraordinary from an ‘expertise specificity’ point of view. The expert subjects demonstrated superior recall in their area of expertise, topographical contour maps. There was, however, no significant difference between the groups in recall of planimetric map features. One conclusion of the Gilhooly study is that expertise may not, at least not always, readily be transferred to seemingly related work domains. Expertise is domain specific.
Green and Gilhooly present in their article five ‘maxims’, which are said to summarize the general thrust of many detailed results in the area of expertise research. These maxims will be presented below, including presentation of relevant empirical research.
1. Experts remember better
Green and Gilhooly hold that this principle owes much to the early work of deGroot on chess skill (Green & Gilhooly, -92, p.46). DeGroot demonstrated the difference in board position recall for players of different skill levels, the grand masters being superior in recall to masters, masters superior to experts, and experts superior to your regular strong player. This difference in recall existed only for board positions from real games, and did not extend to random board positions. The suggestion is made that meaningful chess patterns, or any domain specific configuration, are stored in the long term memory of experts facilitating their recognition. As will be evident, this suggestion points forward to dynamic memory and the theory of case-based reasoning as developed by Schank.
4. Experts are superior in knowledge, not in basic capacities
Green and Gilhooly make the claim that experts do not differ from non-experts in their basic (cognitive) capacities. Rather, experts differ from novices in being superior in domain knowledge. This maxim finds, like the first, support in the work of deGroot on skill in chess playing. DeGroot found in his study very little difference in number of first moves and number of different moves considered, and also little difference in depth of search between grand masters and experts (‘experts’ being less expert than ‘grand masters’). Expertise does not seem to be dependent on individual cognitive abilities. More likely the determining factor is the amount of domain knowledge possessed, the amount of knowledge, as it will be argued at a later point in the paper, a consequence of experience.
5. Experts become experts through extensive practice
The last maxim of Green and Gilhooly is that one way, possibly the way, to expertise is practice. This is particularly evident in the acquisition of expert skills, where the degree of expertise may be said to come as a direct consequence of the amount of practice invested in the work domain. Green and Gilhooly refers to Newell and Rosenbloom’s ‘Power law of practice’, where the number of trials is seen as standing in a logarithmic relationship to the achieved level of expertise. In the present paper it will be argued that the relationship of practice and skill may be said to parallel the relationship of experience and knowledge. This theme will be continued in the presentation of the framework of case-based reasoning.
It is of importance, when reading this paper, to realize the similarity of expert problem solving and operator control tasks in complex human-machine systems. As has already been taken for granted, the operator of a complex system is to be understood as an expert. Expertise seems imperative for doing the job as operator. Further, the control tasks the operator is assigned may be seen as examples of problem solving. Both routine control tasks as routine problem solving, and novel control tasks as the solving of novel problems. What is known about expert problem solving in general may thus be extended to comprise the operator effectuating of control tasks in complex human-machine systems.
From the five maxims of Green and Gilhooly some points of particular importance for this paper may be extracted. These points are, like the maxims they are derived from, thought of as fairly representative for experts and expert problem solving (as opposed to non-experts and non-expert problem solving).
These general observations on expert problem solving will be utilized in the later discussion of what general framework is to prefer for understanding execution of operator control tasks and how to support operator problem solving in complex human-machine systems.
EID as support of expert problem solving
In this section there will first be given a general presentation of the theory of EID. After this presentation follows a focusing in on the views of Vicente and Rasmussen on the nature of the operator as expert problem solver. The section terminates in the presentation of an empirical study conducted by Vicente et. al. on what it takes to become an expert in a complex system.
The theory of ecological interface design has been developed and presented by Vicente and Rasmussen in numerous articles (Vicente & Rasmussen, -88/-90/-92; Rasmussen, -90). The different articles focuses in on different aspects of EID, elaborating each other. The article ‘Ecological interface design: Theoretical foundations’ (Vicente & Rasmussen, -92) is a general presentation, and will serve as a starting point for the following presentation.
EID is developed in response to the challenges meeting the operator of complex technological systems. Vicente and Rasmussen describes this project as an extension of principles developed in the work of Hutchins, Hollan, and Norman on direct manipulation interfaces (DMI) (Hutchins, -86) to comprehend interfaces for complex work domains (Vicente & Rasmussen, -92, p. 600 ff.). The theory of EID is primarily developed for highly structured and tightly coupled systems, as a prime example of which Vicente and Rasmussen use the nuclear power plant (but see also Rasmussen, -95 for a more differentiated view). In this paper EID will only be discussed in relation to such highly structured systems.
The prime focus of EID naturally is the system interface, by which is meant the meeting points between operator and system. Through the system interface (1) the operator may be informed of system status, history, and possible future, and (2) the operator in turn controls system behavior through means provided by the system. The main assumption of EID is that the interface is there for supporting operator system control, and to optimize this function it is necessary to take in regard both system needs and operator abilities. Optimizing system interface function requires scrutinizing the characteristics of the actual system as well as its operator. There is a need to understand how the system is structured, but there is equally a need to understand how the operator works. The reason for describing this theory of interface design as ‘ecological’ is the heed paid by Vicente and Rasmussen to the understanding of operator and system as mutually dependent in the task of running the system, this understanding reminiscent of the organism-environment mutuality assumed in the ecological approach of J. J. Gibson (Gibson, -79). The human operator is to be regarded as part of the system loop, and this whole is described as a human-machine system.
The structure of the design problem, as it is conceived of in the theory of EID, is that of the relationship between a complex work domain and its operator as communicated through the system interface. Vicente and Rasmussen reduces the problem of interface design to two basic questions, the first related to the presentation of information about system structure, the second related to the psychological characteristics of the operator (Vicente & Rasmussen, -92, p. 591).
As is evident, answers for both questions require assumptions about expert problem solving. In answering the first question Vicente and Rasmussen proposes ‘the abstraction hierarchy’ as a psychological relevant formalism for domain representation. In answering the second, they propose ‘the SRK taxonomy’ as a conceptual tool for describing human information processing ability.
The abstraction hierarchy
The abstraction hierarchy is a framework for representing a work domain in a way that is relevant for the operator working on the system interface. It is a stratified hierarchy where the different levels deal with the same system at different physical and functional levels of abstraction, utilizing different conceptual systems. Each stratum has its own unique set of terms and principles, and the selection of strata depends on the system knowledge and interest of the operator. The structure of the abstraction hierarchy is further specified by a means-end relationship between the levels, the purpose of the properties of one level being the functions of the next higher level. Thus, the requirements for proper system function at any level appear as constrains on the meaningful operation of the lower levels. Operator control tasks are thus specified in the constraints of the relevant level, given through the functional demands of the higher levels. Vicente and Rasmussen hold that five levels of constraints have been found useful for describing process control systems: (1) functional purpose, the purposes for which the system was designed; (2) abstract function, the superordinate causal structure of the process; (3) generalized function, the basic functions the system is designed to achieve; (4) physical function, the characteristics and connections of the components; and (5) physical form, the appearance and location of the components. (For an exemplification of the five levels see the description of alternative DURESS interfaces below.) Presenting information at all levels of the abstraction hierarchy enables the operator to engage in process control at the appropriate level of abstraction. In cases of unfamiliar system behavior, the operator may access adequate information at lower levels of abstraction through a ‘zooming in’ process. Structuring the interface by the principles of the abstraction hierarchy is thought, and also empirically demonstrated, to make operator control more effective than interfaces structured after more traditional principles, like what Goodstein has termed 'single-sensor-single-indicator' (Effken, -97).
The abstraction hierarchy is thought to embody a representation of complex system structures
It may already be noted that what will be argued in the present paper is that the SRK taxonomy of Vicente and Rasmussen, and the assumptions it demands, represents an inadequate understanding of human cognition in general, and expert problem solving in particular. It will, however, not be argued that the abstraction hierarchy, which in Vicente and Rasmussen is founded in the SRK framework, is unsuitable. However, it is argued that it may be equally well be supported in an understanding of the human problem solver as a case-based reasoner. This alternative understanding may also provide ideas for new supportive interface devices for an operator of complex systems, in addition to the organization of information in an abstraction hierarchy.
The operator as expert problem solver
When Vicente and Rasmussen are to describe the problem solving abilities of the operator, they present these as divided in two fundamental categories: (1) Perceptual processing and (2) analytical problem solving. ‘Perceptual processing’ is thought to comprise the cognitive control modes of SBB and RBB. ‘Analytical problem solving’ encompasses KBB. In general, perceptual processing is described as fast, effortless, and proceeding in parallel, while analytical problem solving is slow, laborious, and proceeding in a serial fashion. Perceptual processing can only be activated in familiar situations, because they require the operator to be attuned to the perceptual features of the environment. Analytical reasoning, on the other hand, allows the operator to cope with novelty.
The main thrust of the argument of Vicente and Rasmussen is people’s propensity for perceptual processing, as opposed to analytical problem solving. A study conducted by G. A. Klein is presented (Vicente & Rasmussen, -92, p. 595), providing a good example of this. Klein et. al. conducted a series of naturalistic studies of expert decision making in the domains of fire-fighting, military operations, and engineering design. Non-routine events requiring skilled decision making were identified, and interviews were conducted to examine the nature of the decision making process. Since the incidents examined were non-routine, one would, in the words of Vicente and Rasmussen, expect the decision making to be analytical rather than recognitional. However, the results indicated that, even in such critical incidents, experts often relied on the recognitional mode of decision making, or problem solving through perceptual processing. A somewhat similar study, reported by Erik Hollnagel on the subject of mode of cognitive control in operator problem solving, is also referred to (Vicente & Rasmussen, -92, p. 596). Hollnagel makes a distinction between ‘surface control’ (utilizing perceptual processing) and ‘deep control’ (equaling analytical problem solving through the use of mental models of the domain). His results clearly indicate a preference for surface control supported by perceptually available information, rather than deep control founded in mental models of the domain. This was so even in situations where reliance on surface control lead to less than optimal system control. When given the possibility of not thinking analytically, it seems like the operator happily gives analytical problem solving a miss.
This preference for perceptual processing in operator problem solving, Vicente and Rasmussen argues, ought to be utilized in interface design. Complex work domains are highly specialized and tightly structured, the domain hardly changing across time, and operators are to be highly skilled with extensive practice on the system. Thus, the task of routine control of the domain seems ideal for utilizing operator perceptual processing, rather than the more demanding and less preferred level of KBB. The enabling of this puts certain demands on the system interface. There is a need for (1) a consistent one-to-one mapping of system components and functions on the interface, (2) an interface display of relevant system properties, and (3) a display, in the interface, of system functional relations. In other words, the utilization of operator propensity for perceptual processing may be said to demand an interface designed on the principles of the abstraction hierarchy. By structuring the interface, and interface information, by the principles of the abstraction hierarchy, one is ensured to have made perceptual processing an adequate operator option during routine system control.
Perceptual processing, as understood by Vicente and Rasmussen, is not always enough to ensure safe and effective control of system behavior. In cases of unusual or abnormal system behavior, the operator may no longer rely on a recognitional mode of problem solving. Rather, system control demands a different mode of operator cognition, the mode of analytical problem solving. The rationale behind this is as follows: Events in a complex, highly structured system may be subdivided in three groups; familiar (routine tasks), unfamiliar (rare, but anticipated by the designer),and unanticipated (unfamiliar for the operator, not anticipated by the system designer) (Vicente and Rasmussen, -92, p. 589). Unfamiliar events, according to Vicente and Rasmussen, require analytical problem solving. But they may also be prepared for through simulations and practice, thus avoiding the problem of unfamiliarity. There is, however, no safeguarding against unanticipated events (the Three Mile Island accident mentioned as the usual example of this). Unanticipated events demand analytical problem solving on the basis of domain knowledge and first principles. Because of this, operator functioning may always require this highest level of cognitive control. Uncritical reliance on a level of perceptual processing may result in so called ‘procedural traps’, where the operator follows the routines of normal system functioning, while these routines during abnormal system states may no longer be adequate (Vicente & Rasmussen, -88, p. 36).
Following this line of reasoning, Vicente and Rasmussen concludes that the system interface, in addition to support perceptual processing, also ought to support analytical problem solving. Analytical problem solving is thought of as a process where the operator incorporates knowledge of the current system state in his mental model of the system, and on the basis of general and domain rules and principles reasons his way to the appropriate control action. No wonder, the level of KBB is thought of as prone to error even under the best of circumstances (Vicente and Rasmussen, -92, p. 595). Interface design based on the principles of the abstraction hierarchy is thought to alliviate operator problem solving, also on the level of analytical reasoning. Because this kind of interface, where the abstraction hierarchy is made visible in the interface, provides the operator with a "normative externalized mental model of the process that can support thought experiments and other planning activities" (Vicente & Rasmussen, -92, p. 599). This argument follows the principle of Donald Norman of supporting ‘knowledge in the head’ with ‘knowledge in the world’ (Norman, -88). Analytical problem solving requires processing of relevant domain information at the relevant level of abstraction. If this information is provided in the system interface, and in an EID interface it is, controller problem solving is alleviated because of the reduced demand on controller cognitive capacity.
The description of operator problem solving as relying either on perceptual processing or analytical problem solving seem nicely to support interface design along the lines of the abstraction hierarchy. And interfaces based on principles of the abstraction hierarchy is clearly demonstrated to support operator problem solving (Vicente, -95/-96). Following which, the description of expert problem solving as either recognitional or analytical must be equally adequate? Or, maybe not. In the next section it will be argued that the SRK taxonomy, and also the division of expert problem solving in the categories of perceptual processing and analytical problem solving, is founded in assumptions of problem-space theory on the nature of human reasoning. Assumptions that reasoning requires knowledge of system constraints, knowledge that may be thought of as organised in a mental model, and that problem analysis requires mapping the necessary information on the mental model for cognitive processing. By adopting the assumptions of problem-space theory, familiarity is reduced to an intimate knowledge of domain constrains. Experience is represented as the expert having developed a mature mental model. But it may be argued that this is not how expert problem solving seem to work at all. Expert problem solving seems to rely mainly on the experience of earlier situations, and the comparison of the present situation with the past ones. This observation is at the core of the discussion of this paper. The subject will be pursued after presenting an empirical investigation of Vicente on operator problem solving and the nature of expertise (Vicente, -95).
Characterizing the expert problem solver
In the article ‘Supporting operator problem solving through ecological interface design’ (Vicente, -95), Vicente et. al. presents an empirical investigation of the implementation of the abstraction hierarchy in the interface of the complex system simulation DURESS. DUal REservoir System Simulation is a thermal-hydraulic process simulation designed to be representative of complex human-machine systems. The system is described as consisting of two redundant feedwater streams, each consisting of a pump and three valves, which can be configured to supply water to two reservoirs. System goals are specified in terms of reservoir water temperature and continually changing, externally demanded water flow rates. As control condition, a traditional interface is included displaying goal variables (demand and temperature), the state of the components (pumps, valves, and heaters), and the topographical layout of the system components. The variables of the traditional interface are respectively at the levels of ‘functional purpose’, ‘physical function’, and ‘physical form’ of the abstraction hierarchy. The EID interface utilized in the experimental condition displays the variables of the traditional interface in addition to variables of the system mass and energy balance (level of ‘abstract function’), and variables of feedwater flowrates and reservoir heating rates (level of ‘generalized function’). In other words, the EID interface displays information at all the levels of the abstraction hierarchy, while the traditional interface displays information only at three of these levels.
In the article, two experiments conducted on the Duress research vehicle are presented. In the first experiment subjects classified as novices or experts on thermal hydraulics were presented with system simulations on either the traditional or the EID interface. Experts were defined as graduate students in either mechanical or nuclear engineering. (Those are theoretical experts, not experts at controlling the system.) The results showed the EID interface to be the superior tool in diagnosing system condition for experts, but that novises performed equally poor in both conditions. This is interesting in itself, and in clear support of the principles of the abstraction hierarchy. What is of main interest for the present purposes, however, is the surprisingly great overlap in the performance of some of the experts with the performance of some of the novices. This made Vicente et. al. question their original definition of expertise, and conduct a second experiment to investigate what definition of expertise actually correlates with performance on DURESS.
One of the research question of this second experiment may be rephrased: ‘Who is an expert of a complex system?’, and through the course of the study several possible answers were investigated. ‘Would-be’ experts were chosen on the basis of the following categories: (1) Academic expertise, judged on the basis of amount of relevant formal education; (2) analytical work experience, defined as the amount of related professional experience in analyzing thermal and/or fluid systems; (3) operator work experience in process control systems, (4) experience with the DURESS system. The subjects were rated individually according to the four measures above, given a shorthand presentation of DURESS, and presented with ten scenarios of DURESS behavior on the EID interface. Each presentation lasting about one minute. The subjects were told that the simulation represented a ‘real life’ system, and that their task was to diagnose the system behavior.
It may be useful to note that in the experimental situation of this second experiment, recognitional problem solving, or problem solving based on perceptual processing, was supposedly barred for most of the subjects due to the unfamiliarity of the situation. The principles governing the simulation and the relevant domain constrains ought to be known by, and familiar to, would-be experts of all the categories above. Even so, when subject performance was correlated with the level of expertise in the different categories, significant correlation of subject system diagnosis with level of expertise were found in only one. Only the ratings on ‘experience with the DURESS system’ correlated positively with the level of performance in diagnosing system behavior. In other words, of the four categories investigated only experience with the DURESS system may be said to be an adequate measure of DURESS system expertise. General knowledge of domain principles and constraints, or seemingly similar controller experience may not be regarded as measures of expertise in this particular context. These results are somewhat outstanding in regard to the importance of analytical reasoning in expert problem solving earlier assumed by Vicente and Rasmussen. If analytical problem solving based on domain knowledge were the hallmark of expert handling of novel problems, expertise ought not to be a quality depending on former DURESS experience. Rather, expertise ought to be a result of theoretical domain knowledge, and general experience in problem solving requiring this theoretical knowledge. Expert problems, in accordance with the view of Vicente and Rasmussen presented above, ought to be solved through the processing of adequate information on an adequate mental model of system principles and constraints. When this is not the case the question has to be raised weather expert problem solving actually is analytical in nature.
Vicente et. al. nearly rises this question in admitting that the subjects with prior knowledge of DURESS sometimes seemed to diagnose the system in a ‘recognition-based’ manner. This is attributed to the fact that these subjects had previous exposure to the typical behavior of the system. Thus, previous experience may be the real cause of their expertise. Maybe their expert problem solving was not analytical at all, but founded in earlier experience. This conclusion, however, is not reached by Vicente et. al. They write: "The results indicate that system-specific knowledge, and not generic analytical or control experience, is the best predictor of performance in a task requiring KBB." (Vicente, -95, p. 541). The assumption of expertise as a consequence of analytical ability is kept. This assumption, it will be argued, has its roots in problem-space theory. And this assumption is what will be discussed in regard to the adoption of case-based reasoning as a more adequate framework for explanation. As will be made clear, the change of general framework for understanding expert problem solving will provide a more reasonable explanation of the results of Vicente et. al. Experts may not be analytical problem solvers. Experts may just be experienced.
The framework of problem-space theory
In their textbook ‘Cognitive psychology’, M. W. Eysenck and M. T. Keane presents the problem solving research of Allen Newell and Herb Simon as "the bedrock of the information-processing framework" (Eysenck & Keane, -95, p. 361). Newell and Simon’s problem-space theory of problem solving, as recounted in ‘Human problem solving’, is said to remain at the center of current problem solving research. In the present section, a short presentation of problem space theory, and its implications for theory on expert problem solving, will be made. After this, it will be argued that Vicente and Rasmussen, in their work on operator problem solving, assumes the framework of problem-space theory. Problems with problem-space theory in regard to expert problem solving will be alluded to, as a prelude to the presentation of case-based reasoning in the following sections as a more adequate framework for understanding expert problem solving.
According to Eysenck and Keane, Newell and Simon suggested that the objective structure of a problem can be characterized as a set of states. Beginning with an initial state, involving many intermediate states, and ending with a goal state. Thus, there is a whole space of possible states and paths, beginning at an initial position, ending at a previously defined goal situation, where the intermediary states to be understood as the total set of possible legal moves. This space is thought to describe a possible abstract structure of a problem, and is designated ‘problem space’. Problem space is thus considered a conceptual framework for a possible problem solver. If a problem can be expressed within the framework of a problem-space it may be solved by moving through the path of legal intermediary states that leads from the initial state to the goal state. In its simplest fashion, problem space may be said to express the possible proceedings of a possible problem solver. But Newell and Simon take the concept of problem space one step further (and this is the controversial part of the story). The abstract structure of problem-space is proposed to be a representative model of the actual framework for problem solving in humans. Human problem solving abilities are represented as the construction of a problem-space, and the moving through this space on a path of legal moves, beginning at the initial state, ending at the goal state.
Initially, problem-space theory may be considered only to model a general problem solver (GPS). In solving problems the GPS way, all the constraints of the problem space must be given prior to the actual solving of the problem. This requires a characterization of the problem as ‘closed’, where all the relevant parameters for solving the problem are specified and unchanging (or changing in the predicted manner). Examples of GPS-problems are the ‘Tower of Hanoi’ and the ‘missionaries and cannibals problem’. But problem-space theory is easily extended to embrace domain specific problem solving as well. What is critical for the expression of the process of problem solving in the terms of problem-space theory is the possible definition of an initial state, a goal state, and the intermediary states of the problem space. And these states may be defined in any domain specific problem, as long as the constraints of the domain are known and unchanging. In the classic problem solving situation of the ‘Tower of Hanoi’ and the like, the constraints of the domain are given explicitly in the description of the problem. In so called domain specific problem solving the constraints may partially be taken as known by the problem solver. In this way, domain specific problem solving may be said to require domain knowledge, that is knowledge of the domain not given in the problem specification. Still, the domain knowledge tacitly presumed may not differ in principle from the constraints embodied in the expressed problem specification. Both may be considered as unchanging, or predictably changing, constraints.
An example may be in order to make visible the in-principle similarity of GPS problem solving and expert problem solving in regard to problem-space theory. The solving of physics problems, like those utilized in the study on expertise by Chi et. al. (Chi, -81), may be seen as an activity of expert-, and therefore also domain specific problem solving. The problems given are underspecified, in that their solution requires not explicitly given knowledge of domain principles and constraints. Like in the problem ‘A man of mass M1 lowers himself to the ground from a height X by holding onto a rope passed over a massless frictionless pulley and attached to another block of mass M2. The mass of the man is greater than the mass of the block. What is the tension of the rope?’ (Chi, -81). The problem is held to be solvable only by applying the appropriate principles and constraints from the physics domain. The important point, however, is that the principles and constraints required to reach a solution are held to be known and unchanging, and therefore could have been specified in the problem specification. From a problem-space theory point of view it may be argued that all problems are domain specific, and that a GPS may solve them as long as the actual problem may be described in terms of problem-space theory.
The concept of ‘mental models’ may be seen as an extension of problem-space theory. Ray Eberts in ‘User interface design’ presents mental models as a term describing expert understanding of a target system (Eberts, -94), where mental models develop as an increasing knowledge of system constrains through extended interaction with the system. A similar understanding is expressed by Vicente and Rasmussen in their allusion to ‘the laws of control theory’ (Vicente & Rasmussen, -92, p. 590). In the laws of control theory it is expressed that: (1) complex systems require complex controllers, the complexity inherent in the system cannot be displaced; (2) physical systems can be described by a set of constraints; (3) every good controller must be, or possess, a model of the system it is controlling. The first two points are included in the assumptions of the abstraction hierarchy. The third point, however, says something about the nature of operator problem solving. Operator problem solving is thought to proceed analytically within the constraints of the adequate mental model. The mental model, through its embodiment of system constraints, is ideally held to constitute the possible intermediate states, or legal moves, in the problem-space of the domain.
The use of ‘mental models’ in Vicente and Rasmussen can be held to lean on the framework of problem-space theory, and it will be argued that the same holds true for the SRK taxonomy as an understanding of human information processing. As is remembered, the categories of cognitive control suggested in the SRK taxonomy are divided between two modes of expert problem solving: perceptual processing and analytical problem solving. Within the mode of perceptual processing are the categories of skill-based behavior, corresponding closely to what is normally meant by ‘motor- or behavior skill’, and rule-based behavior, thought of as behavior controlled through the perceptual recognition of a situation. These modes of cognitive control are assumed to be preferred in the execution of familiar tasks. However, when the assigned task is no longer familiar, and therefore no longer recognizable, the problem solver ought to enter knowledge based behavior as mode of cognitive control. Or, put differently, the problem solver should engage in analytical problem solving. The EID interface is thought of as a tool for supporting this analytical problem solving, in that the required constraints for a full problem specification ideally is presented in the interface. Later in this paper it will be argued that the RBB and KBB categories of the SRK taxonomy are of dubious value in describing modes of human problem solving. For now it is sufficient to show that the proposed level of KBB, or expert problem solving based on analytical reasoning, demands the assumptions of problem-space theory.
In the theory of Vicente and Rasmussen, KBB requires an accurate mental model of the target system. Problem solvers engaging in KBB are thought to map relevant system information on their mental model of the system, and then ‘run’ the mental model in order to learn the possible outcomes of current system behavior. The information provided by the interface is assumed to be processed through the model in an analytical fashion, the processing consisting of legal moves determined through domain constraints. The similarity to problem-space theory as presented above is striking, and it may be inferred that the claim that Vicente and Rasmussen are making assumptions equaling those of problem-space theory hardly is controversial.
It may be stated that one of the problems with the SRK taxonomy is the assumed demarcation of problem solving as either recognitional (as perceptual processing), or analytical. This division seems to require the problem situations to be described as either familiar and recognizable, or unfamiliar and not recognizable at all. Following this, unfamiliar problem situations may be said to require analytical reasoning, because nothing like them has never been encountered. The question of whether there are such things as totally unfamiliar situations, however, is not as straight forward as it may seem. It may be argued that all situations represents a certain amount of familiarity, and, as an extension of this claim, there is no need to assume an analytical problem solver in explaining operator tasks. This line of thought differs radically from that presented in problem-space theory, and as one manifestation of this differing perspective the theory of case-based reasoning will be presented.
Case-based reasoning as an alternative framework for understanding expert problem solving
The theory of case-based reasoning may be thought of as developed in reaction against problem-space theory and the assumption that human reasoning follows the principles laid down for a GPS. The historical roots of the theory can at least be traced to schema theory, in particular the work of Schank and Abelson on script theory in the 1970’s, and the theory is fleshed out in it’s present form in ‘Dynamic memory’ (Schank, -82) and the more implementationally oriented ‘Inside case-based memory’ (Riesbeck & Schank, -89).
It may be held that much of Schank’s work is addressed to the computer sciences, rather than psychology. But his research has attracted the interest of psychologists because of its specificity and scope, and his later work has been inspired by, and addressed to, the field of psychology (Eysenck & Keane, -95). In his work on expert problem solving and reasoning, Schank has been aiming within the departments of both psychology and computer science, the object of his research being (1) the investigation of the problem solving- and reasoning abilities in human experts, and (2) the implementation these abilities in computer programs. In many ways these research goals are complementary to those of Newell and Simon, but there is one fundamental difference: When Newell and Simon take the computer program as their point of departure, and then transfers this to comprise problem solving in humans, Schank takes human memory and reasoning as his starting point, after which he tries to develop computer programs working in a similar fashion. Stated bluntly, Newell and Simon may be held to mold human reasoning in the image of the machine, while Schank tries to mold the machine in the image of human reasoning.
In ‘Inside case-based reasoning’ (Riesbeck & Schank, -89), Schank presents a description of human reasoning and expert problem solving as it meets us in the every-day world, the first assumption being made is that "the conception of the nature of human reasoning embodied in ... the GPS ...is wrong" (Riesbeck & Schank, -89, p. 2). Human reasoning depends on understanding, and understanding depends on explanation. In order to understand a story, or a scene witnessed, it is necessary to explain why people seen or referred to act the way they do. An understander of the world is an explainer of the world, and, according to Schank, the world is explained through earlier experiences, or cases. The process of explanation may seem to be a difficult one, since it involves understanding new ideas and new situations. But Schank claims that the explaining of new situations does not require reasoning from first principles. Rather, explanation is a process of adaptation, not creation, since it is possible to explain something new by adapting a standard explanation from other situations. "Maybe we don’t ever generate new explanations. Maybe we just adapt old ones to new situations" (Riesbeck & Schank, -89, p. 5).
This line of thought is elaborated in what Schank, somewhat ironically, calls ‘the basic planning algorithm’. When facing a novel situation people do not construct plans from first principles. Rather, they try to find the best plan they have heard of, or previously used, that is closest to the problem at hand, and attempt to adapt that plan to the current situation. "Note that the question here is not whether reasoning from prior cases is what you ought to do. The question is whether it is what people do do" (Riesbeck & Schank, -89, p. 6). Planning in novel situations is thought to rely on earlier experiences, not a perfect mental model of the situation, and may be said to require (1) recognition of present situation as reminiscent of earlier known situation, and (2) adaptation of the earlier situation to the present. The passing through these two stages is the execution of the basic planning algorithm.
The idea expressed in the basic planning algorithm is at the heart of case-based reasoning. People reason from experience. They use their own experiences if they have a relevant one, or they make use of the experience of others to the extent this is possible. An individual’s knowledge, according to Schank, is the collection of experiences that he has had or that he has heard about. But, what if there are no earlier similar experiences to be found? What if the current setting is a totally unfamiliar one? The proposed answer to these questions is: There is no such thing as a totally unfamiliar situation.
Schank makes a distinction between reasoning by rules, by cases, and by stories, claiming the distinction between these three constructs essentially to be a memory distinction. The three constructs are named ‘ossified cases’, ‘paradigmatic cases’, and ‘stories’. (1) Ossified cases are rules, because they have been abstracted from their original cases. The case from which they were originally derived is unknown, and the ossified cases themselves are shared by large numbers of people. When the rule fails, the only alternative for its user is to create a case that captures this failure. A case stands alone as an exception until numerous other cases just like it is encountered, and a new rule is created. (2) Paradigmatic cases exist for situations where only one previous experience is found, or there are many similar experiences, but with such differing results that no path is obvious. The task for the reasoning system is to find where the current situation differs from the paradigmatic case, and to adopt it to help problem solving in the new situation. (3) Stories are the third category of cases presented by Schank. By stories is meant cases that are unique and full of detail, like paradigmatic cases, but with points, like proverbs. Schank believes that the basis for creativity in a cognitive system comes from the stories it knows, rather than the ossified or pragmatic cases it has. Stories, by virtue of their complexity, may relate to a large variety of possible circumstances. And, more importantly, they can be indexed in multiple ways.
According to Schank, human experts are not systems of rules, they are libraries of experiences. Experiences that may be categorized as ossified cases, paradigmatic cases, and stories. The level of expertise may be said to depend on the volume of cases relevant to the target domain, and the ability to retrieve an adequate case for the given situation. The problem of retrieval is addressed through the term ‘reminding’. Encountering a novel situation will remind the actor of similar experiences he or she has had earlier. An expert is someone who gets reminded of just the right prior experience to help processing the current experiences. Awareness of the reminding phenomenon may alter our conception of what is meant by the term ‘understanding’. Understanding does not mean the mapping of information on the adequate mental model of the domain. Understanding means being reminded of the most similar previously experienced situation. Equally, expert problem solving is taken to rely on the getting reminded of cases, not the ‘running’ of a mental model. Schank tries to explain the phenomenon of reminding by the assumption that memory structures for storage and processing structures for analysis of input are the same structure. This, however, is not important for the present paper. What is important is that expert problem solving is described as a process of case-based reasoning, where recognition of former cases through the process of reminding is of imperative importance for the problem solving process.
It is interesting that virtually all the empirical studies of expert problem solving presented in the first section, ‘Expert problem solving’, may be explained equally well, or maybe better, within the framework of case-based reasoning with reminding, recognition, and library of cases as core concepts, than within the framework of problem-space theory. It may be said that the work of deGroot on experts in chess is an early hint at a theory of case-based problem solving. It will be recalled that grand masters did not seem to differ in basic cognitive capacities from less expert players in that they processed the same amount of possible moves, and had equally poor recall of random board positions. Still, grand masters seemed to have a larger library of known board positions, and also had greater recall of ‘real’ board positions. These abilities seem better explained through the concepts of reminding and recall, rather than analytical problem solving. This point is excellently illustrated in the comparison of the reasoning routine of a human expert, as the one presented above, with that of the expert computer Deep Blue, which maps the current board position on a model including the constraints of the game, and then ‘runs’ this model at the speed of 9 billion moves a second (Eysenck & Keane, -95). By the way, the problem solving capacities of Deep Blue may best be understood within the framework of problem-space theory.
Equally, the results of Chi et. al. on expert analyzing of physics problems, and the results of Larkin et. al. on expert work on physics problems (both presented in the section ‘Expert problem solving’), may be well taken care of within the framework of case-based reasoning. Experts, having superior case-based memory structures as a consequence of greater domain relevant experience, will be supplied with a superior tool for effective classification, and will also be able to proceed in their problem solving in an adequate manner. The study of Gilhooly et. al. on expertise in map-reading (presented in the section ‘Expert problem solving’) is also of interest in an evaluation of the framework of case-based reasoning as opposed to the framework of problem-space theory. The demarcation of map-reading expertise as extending only to the domain of contour maps, but not the area of planimetric maps, seems to support a theory where reasoning from first principles and knowledge of constraints is not assumed. Rather, the results of Gilhooly et. al. seem compatible with an understanding of memory and reasoning as founded in the experience of cases. The theory of case-based reasoning should also be able to throw some additional light on results in studies like that of Klein referred to by Vicente and Rasmussen (Vicente & Rasmussen, -92), and presented above in ‘EID as support of expert problem solving; The operator as expert problem solver’. The study of Klein shows that experts seem to utilize a recognitional mode of problem solving, even in unfamiliar situation. Vicente and Rasmussen understand these results as a propensity for perceptual processing, at the expense of the more strenuous analytical problem solving. But, according to the theory of case-based reasoning, the results may not just represent a propensity for what Vicente and Rasmussen terms perceptual processing. Rather, they may display the very nature of expert problem solving abilities in humans. Following this, the proposed categories of perceptual processing and analytical problem solving may turn out to be inadequate. Maybe the categories ought to be those of ‘skill’, as founded in practice, and ‘knowledge’, as founded in experience.
In this section case-base reasoning has been presented as an alternative framework to problem-space theory for understanding human reasoning in general, and expert problem solving in particular. It may be held that the theory of case-based reasoning has far greater intuitive appeal as an understanding of human reasoning than that of problem-space theory. The work of Schank may appear as a reasonable explication of an every-day, man in the street, understanding of the way people actually seem to think. Still, there are difficulties. One of the main problems of the theory of case-based reasoning is the problems that have been in computationally implementing case-based reasoning expert problem solvers. Schank may have been giving a good description of the reasoning abilities of humans. Still, this knowledge has been shown to be difficult to utilize for implementational purposes, even though quite a few simple expert problem solving programs are presented in ‘Inside case-based reasoning’ (Riesbeck & Schank, -89). Implementational difficulties, however, cannot be said to undermine the usefulness of the theory as a psychological theory of human cognition.
One difficult point with case-based reasoning as a theory of cognitive psychology is that it is underspecified (Eysenck & Keane, -95). Following this, the theory may be said to be good at accounting for results in an ad hoc fashion, but it may not be very predictive. Put differently, the theory of case based reasoning can be regarded as a description of human reasoning problem solving, but not as an explanation of those phenomena. ‘Recognition’ and ‘reminding’ are introduced as crucial concepts, but apart from denoting known phenomena these concepts hardly carry any explanatory value. Then, what is the use of the theory of case-based reasoning?
The problem of underspecification may be a serious problem, given that the theory is to be taken to explain human reasoning. This, however, is not what the theory is needed for in the context of supporting expert problem solving through interface design. On the contrary, what is needed is a description of expert problem solving abilities, in order to optimize operator system control. To get to know how expert problem solving best may be supported, what is needed is an accurate description of how expert problem solving actually work. Such a description is given in the theory of case-based reasoning. And, just as important, this description does not seem to be given through problem-space theory. To be sure, the theory of case-based reasoning truly is underspecified, but this does not present any objections for it’s utilization in theory of interface design. It seems reasonable to conclude that a theory of interface design for supporting operator problem solving may be better off relying on the theory of case-based reasoning as an understanding of human problem solving, rather than on the assumptions of problem-space theory.
Consequences of adopting the framework of case-based reasoning
In the sections above, two different theories of human reasoning and expert problem solving has been presented. These two theories has been discussed against each other as possible frameworks for understanding expert problem solving (1) as it appears in psychological research in general, and (2) for use in a theory of interface design in particular. It has been suggested that the theory of case-based reasoning represents a more adequate understanding of human expert problem solving than problem-space theory. Furthermore, it has been shown that Vicente and Rasmussen, in the elaboration of their theory of EID, assumes the principles of problem-space theory in their SRK taxonomy. Following this, it will be suggested a change of framework for understanding expert problem solving in the theory of EID. In this final section, possible consequences for the theory of EID in adopting the theory of case-based reasoning as framework for understanding operator problem solving will be discussed. An alternative explanation of the results in "Supporting operator problem solving through ecological interface design" (Vicente, -95) will be included, and so will possible new suggestions for interface design brought on by the theory of case-based reasoning.
The theory of EID may be said to consist of two conceptual entities: (1) the abstraction hierarchy, and (2) the SRK taxonomy. As is remembered from the second section, the abstraction hierarchy is the answer put forward to the question of a psychologically adequate structure for presenting information about a complex system. The SRK taxonomy serves as an explication of human information processing abilities thought to justify the adequacy of the abstraction hierarchy. As is learnt from experimental evaluation, interfaces structured by the principles of the abstraction hierarchy actually do seem to outperform traditional interfaces (Vicente, -95/-96). Following which, it may be held that the abstraction hierarchy is one substantial improvement in the area of interface design brought on by the theory of EID. On the other hand, the SRK framework does not seem to be underpinned by the same amount of experimental evidence as the abstraction hierarchy. Rather, the empirical studies referred to by Vicente and Rasmussen (among those the study by Klein mentioned above, but see also Vicente & Rasmussen, -92, p. 595 ff.) seem to conclude with the propensity for problem solving in a recognitional manner. Also, the study of Vicente et. al. (Vicente, -95) hardly provides evidence for analytical problem solving in experts (or KBB as the mode of cognitive control). It may be argued that, as a contrast to the abstraction hierarchy, the SRK taxonomy provides little, in the vein of practical implications, to the theory of interface design apart from an apparent theoretical justification of the abstraction hierarchy.
Consequences for the abstraction hierarchy
The basic theoretical foundations for EID are the abstraction hierarchy and the SRK taxonomy, the latter serving the function of justifying the former. It is also of importance that the proposed change of framework will have direct implications for the SRK taxonomy only. The abstraction hierarchy will be left untouched by a change in framework, provided that it is found to be compatible with, and supported by the new framework.
The compatibility of the abstraction hierarchy as a psychological valid structuring of information and the theory of case-based reasoning is clearly demonstrated in the empirical evidence for operator problem solving on EID interfaces through situational recognition of problems. This was shown to be the case in the study of Vicente et. al. presented above, where experts on the DURESS interface were found to be those familiar with DURESS, and, further, that these experts made system diagnoses in what is described as a ‘recognition-based manner’ (Vicente, -95, p. 540). Results like these seem to be perfectly corresponding to an understanding of expert problem solving as based on (1) recognition through reminding and (2) the utilization of a library of cases. Also, EID design is thought to make ‘recognition-based’ operation as easy as possible, through the structuring of the interface by the principles of the abstraction hierarchy. This deemed necessary by Vicente and Rasmussen because of the proposed operator propensity for perceptual processing (Vicente & Rasmussen, -92). But this may equally well, or better, be understood by proposing a propensity for case-based reasoning. Or, rather, that case-based reasoning is the norm in human problem solvers, and that the demarcation of problem solving as either recognitional or analytical may be considered faulty. Conceptualizing operator problem solving on EID interfaces in terms of the theory of case-based reasoning, it may be held that an interfaced structured by the principles of the abstraction hierarchy supports operator system control in that the situational cases of the system are specified at the adequate level of abstraction to facilitate operator reminding and recognition. The principles of the abstraction hierarchy seems compatible with, and supported by, the theory of case-based reasoning as a framework for understanding operator problem solving.
Consequences for the SRK taxonomy
As has been argued above, the SRK taxonomy seems to be founded on assumptions of problem-space theory. Following this, a change in framework for understanding expert problem solving will demand the discarding of the SRK taxonomy as a conceptualization of human information processing. In particular, the division of problem solving as either recognitional (based on RBB) or analytical (based on KBB) has been deemed difficult. The categorizing of situations as either familiar, and demanding RBB, or unfamiliar, and demanding KBB, is not acceptable within the framework of case-based reasoning. Rather, within domain specific reasoning situations are held to be more or less familiar, more or less similar to already known cases. Following this, problem solving within a domain may be said to proceed through the utilization and adaptation of the closest known case. The SRK taxonomy supports a three-division of cognitive control: skill-based behavior (SBB), rule-based behavior (RBB), and knowledge-based behavior (KBB). Within the framework of case-based reasoning there will rather be a division in two: ‘Skill’ and ‘knowledge’. Skill is to be understood as ‘behavioral- and motor skill’, and is the result of practice. The level of ‘skill’ in expert problem solving may be said to equal the level of SBB of Vicente and Rasmussen. The departure from the theory of EID as fleshed out by Vicente and Rasmussen may be thought of as the proposal of the level of ‘knowledge’ instead of the levels of RBB and KBB. Knowledge is to be understood as ‘case-knowledge’, and may thus take on the form of Schank’s suggested categories of ossified cases, paradigmatic cases, and stories. As skill is the result of practice, knowledge is the result of experience. The amount and variety of domain experience determining the quality of the case-library of the expert problem solver. Expert problem solving through the level of knowledge is thought to follow the principles of the theory of case-based reasoning. Knowledge implies the ability to be reminded of known cases, and the ability to recognize the current situation as similar to the closest known case. The knowledge of an expert has to include an as comprehensive library of cases as necessary to make the appropriate problem solving actions. The SRK taxonomy is held to be inadequate, but it may be substituted by a conceptual understanding in line with the theory of case-based reasoning.
An explanation of the results of "Supporting operator problem solving through ecological interface design"
In the end of the second section a study of Vicente et. al. was presented where one of the research goals was to work out a better definition of DURESS expertise. Surprisingly, at least in the light of problem-space theory, subject’s level of general knowledge of thermal-hydraulic systems did not correlate significantly with task performance on the DURESS interface. This in spite of the proposed importance of this knowledge for the execution of the task. On the other hand, former DURESS knowledge correlated strongly with task performance. The subjects scoring high in DURESS knowledge even seemed to base their diagnoses on recognition of situations and system patterns of behavior, even though the system situations were thought to be reasonably unfamiliar as to only being solvable through KBB. After the discussion of problem-space theory vs. the theory of case-based reasoning, it ought to be fairly obvious that the seemingly surprising results of the experiment on expert characteristics presented in "Supporting operator ...", may be better understood within the framework of case-based reasoning. The lack of expertise displayed by the subjects scoring high in general knowledge of thermal-hydraulics may be explained in the lack of knowledge of DURESS-cases, and also by the lack of reminders to their store of relevant knowledge in the DURESS interface. The lack of exposure to the particular interface disabled the utilization of relevant knowledge, probably because of lack of recognition. This is not to say that the subjects scoring high in general thermal-hydraulic knowledge would not be fast learners of the DURESS interface. What is said is that without any exposure, a person cannot be an expert. This is explained in the framework of case-based reasoning, but not in the framework of problem-space theory.
Extended possibilities of supporting operator problem solving through interface design
It has been suggested that the one substantial contribution of the theory of EID to the area of interface design is the abstraction hierarchy. In the present paper it has been held that the theory of case-based reasoning represents a more adequate understanding of expert problem solving than those founded in the framework of problem-space theory. It has also been held that the adoption of case-based reasoning as a general framework for understanding human problem solving abilities may be compatible with, and supportive of interface design by the principles of the abstraction hierarchy, and, in addition that this framework provides an understanding for empirical results on operator problem solving not easily resolved within the concepts of the SRK taxonomy. In this concluding part of the paper, there will be presented two additional suggestions for supporting operator problem solving. Both springing from the alternative understanding of the human problem solver as a case-based reasoner. The suggestions are not at all thoroughly worked out, but are intended to show the possibility of new ideas springing from a shift in the theoretical understanding of human problem solving.
As is explicated above, problem solving in particular, and also reasoning at large, may bee seen as a matter of utilizing formerly known cases. Successful problem solving depends on the getting reminded on an adequate case in a library of cases, the library of cases being thought of as internal to the expert problem solver. But may not also an additional library of cases be stored in the target system, accessible through the system interface? Would it not seem reasonable to support the problem soving abilities of a human case-based reasoner through the supplementing of an external case library? Something like this sort of case library is already proposed in the work of Gunnar Wille at the department of industrial economics and technology management NTNU on a system called BISP (Beslutning, Informasjon, Styring, Prosesskontroll; or, in translation, decision, information, administration, process-control) (Presented at a meeting at NTNU, the fall of -98). Wille proposes the usefulness of a ‘data base of experiences’ as one of several components of a program for operator education and updating. Possibly as part of a system interface. Such a database is thought to contain information of real and simulated cases, and it does not take a whole lot of imagination to represent the usefulness of such a device. Given that the operator can be considered a case-based reasoner, it may be possible to support operator problem soving through this kind of external case library. One challenge for this kind of enterprise is the problem of navigation in the database, which accentuates the importance of labeling of information. It may also be possible to conceive of the system parameters working as reminders for such an external library of cases.
Vicente and Rasmussen point to the problem of procedural traps as one to be taken seriously in interface design. A procedural trap is when the operator assumes system state to be different from it really is, and proceed with procedures suitable for the situation which the present state is mistaken to be. This is thought of as particularly prevalent when unfamiliar situations on the basis of interface information look similar to routine scenarios (Vicente & Rasmussen, -88). The issue of procedural traps is well made sense of within the framework of case-based reasoning. The operator is reminded, but the reminder is faulty. The result of such ‘false reminding’ may be that the operator recognizes the system state as something it is not. It is recognized as something demanding routine operations, while what is needed is extraordinary precautions. This kind of failure is probably to be characterized as ‘human’, all the time it comes as a consequence of the way human reasoning seem to work. But it may also be held that, at least in theory, this problem may be avoided through interface design. One possible suggestion, maybe approaching the field of science fiction, is the implementing of a ‘procedural trap detector’, an external data base keeping track of what really is routine scenarios through the continuous evaluation of system parameters. Routine scenarios may be defined as scenarios preprogrammed in the database, and probably, in a highly structured complex system (like a nuclear power plant), most scenarios are possible to define as routine. Then, when non-routine scenarios are detected, the operator may be given notice. This to make sure potential procedural traps through false reminding can be avoided. Considering the operator as a case-based reasoner paves the way for predicting possible failures, and may also provide hints of solutions to problems caused by this psychological make-up of the operator.
In this final section there has been a presentation of the possible consequences for the theory of EID when trading assumption based in the framework of problem-space theory for those of the theory of case-based reasoning. It has been suggested that the principles of the abstraction hierarchy may be kept, and also supported within this novel framework. It is further suggested that the SRK will have to be discarded. Maybe to be replaced with an understanding of expert problem solving abilities as a function of ‘skill’ and ‘knowledge’. The possible gains of a change of framework for understanding human problem solving have been presented, in addition to the adoption of more appropriate conceptual tools for describing operator problem solving, as (1) the provision of an explanation for empirical results not easily conceived of within the SRK taxonomy (Vicente, -95) and (2) the suggestion of new ideas for interface design springing from considering the operator to be a case-based reasoner. The purpose of the present paper, however, has not been to present any full blown design of possible interface devices. Rather, what is aimed at is to present ideas that may flow from the adoption of a new framework for understanding expert problem solving in EID. In conclusion it may be held that the theory of case-based reasoning provides a more adequate understanding, however underspecified, of human expert problem solving than do problem space theory. This seem to hold, both for expert problem solving as studied generally in psychological research, and for operator problem solving in complex human-machine systems. Following this, it is concluded that the theory of EID is better founded in the framework of case-based reasoning as an understanding of human problem solving abilities. The framework of the SRK taxonomy, founded in the assumptions of problem-space theory, is held to represent a less than optimal understanding. Consequences of a possible change of framework has been presented, and it has been alluded that a change of framework will bring a change in perspective, following which novel ideas may be generated.
Chi, M. T. & Feltovich, P. J. & Glaser, R. (1981). ‘Categorization and representation of physics problems by experts and novices’. Cognitive science, 5, 121-152.
Eberts, R. E. (1994). ‘User interface design’. Englewood Cliffs, NJ: Prentice Hall.
Effken, J. A. & Kim, N. G. & Shaw, R. E. (1997). 'Making the constraints visible: Testing the ecological approach to interface design'. Ergonomics, 40 (1), 1-27.
Eysenck, M. W. & Keane, M. T. (1995). ‘Cognitive psychology’. Exeter: Erlbaum (UK) Taylor & Mifflin.
Gibson, J. J. (1979). ‘The ecological approach to visual perception’. Boston: Houghton Mifflin.
Gilhooly, K. J. & Wood, M. & Kinnear, P. R. & Green, C. (1988). ‘Skill in map reading and memory for maps’. The quarterly journal of experimental psychology, 40 A, 87-107.
Green, A. J. & Gilhooly, K. J. (1992). ‘Empirical advances in expertise research’. In: Keane, M. T. & K. J. Gilhooly (eds.). ‘Advances in the psychology of thinking (Vol. I)’. London: Harvester Wheatsheaf.
Hutchins, E. L. & Hollan, J. D. & Norman, D. A. (1986). ‘Direct manipulation interfaces’. In: Norman, D. A. & Draper, S. W. (eds.). ‘User centered systems design. New perspectives on human-computer interaction’. Hillsdale, NJ: Lawrence Erlbaum Associates.
Larkin, J. & McDermott, J. & Simon, D. P. & Simon, H. A. (1980). ‘Models of competence in solving physics problems’. Cognitive science, 4, 317-345.
Norman, D. A. (1983). ‘Some observations on mental models’. In: Gentner, D. & Stevens, A. L. (eds.). ‘Mental models’. Hillsdale, NJ: Lawrence Erlbaum Associates.
Norman, D. A. (1990). ‘The design of everyday things’. New York: Currency Doubleday.
Rasmussen, J. (1990). ‘Mental models and the control of action in complex environments’. In: Ackermann, D. & Tauber, M. J. (eds.). ‘Mental models and human-computer interaction I’. Ansterdam: North-Holland.
Rasmussen, J. & Pejtersen, A. M. (1995). ‘Virtual ecology of work’. In: Flach, J. & Hancock, P. & Caird, J. & Vicente, K. J. (eds.). 'Global perspectives on the ecology of human-machine systems'. Hillsdale, NJ: Lawrence Erlbaum Associates.
Riesbeck, C. K. & Schank, R. C. (1989). ‘Inside case-based reasoning’. Hillsdale, NJ: Lawrence Erlbaum Associates.
Schank, R. C. (1982). ‘Dynamic memory’. Cambridge: Cambridge university press.
Vicente, K. J. & Christophersen, K. & Pereklita, A. (1995). ‘Supporting operator problem solving through ecological interface design’. IEEE transactions on systems, man, and cybernetics, vol. 25, no. 4, 529-545.
Vicente, K. J. & Christophersen, K. & Hunter, C. N. (1996). ‘A longitudinal study of the effects of ecological interface design on skill acquisition’. Human factors, 38(3), 523-541.
Vicente, K. J. & Rasmussen, J. (1988). ‘A theoretical framework for ecological interface design’. Risø-M-2736. Risø national laboratory.
Vicente, K. J. & Rasmussen, J. (1990). ‘The ecology of human-machine systems II: Mediating direct perception in complex work domains’. Ecological psychology, 2(3), 207-249.
Vicente, K. J. & Rasmussen, J. (1992). ‘Ecological interface design: Theoretical foundations’. IEEE transactions on systems, man, and cybernetics, vol. 22, no. 4, 589-606.