Johns, R --- "The logic doctor is in: using structure training and metacognitive monitoring to cultivate the ability to self-diagnose legal analysis skills" [2010] LegEdDig 25; (2010) 18(2) Legal Education Digest 30

• The logic doctor is in: using structure training and metacognitive monitoring to cultivate the ability to self-diagnose legal analysis skills

---

The logic doctor is in: using structure training and metacognitive monitoring to cultivate the ability to self-diagnose legal analysis skills

R Johns

26(4) J of Legal Stud Ed, 2009, pp357-297

The call to teach critical thinking skills has a long history among educators – and for good reason. The various rationales advanced for cultivating these skills seem to converge on the clearly important need to prepare students to effectively, and independently meet life’s challenges. It must also be recognised that any useful response to the call for teaching critical thinking must be based on some level of agreement as to what constitutes critical thinking, accompanied by effective pedagogical methods. With respect to teaching methods, there is evidence to support the effectiveness of methods that engage students in educational experiences that are focused, interactive, and provide timely corrective feedback. But there is also the recognised concern that, because of time constraints, the inclusion of skill-development methodologies might come at the expense of content coverage. Thus, a method’s technical effectiveness should be augmented by its temporal effectiveness.

From a pedagogical perspective, the Logic Doctor – the teaching method described and demonstrated here – is a discipline-specific and skill-specific method developed and used by the author. From a learning perspective, it is a way for students to learn to approach and analyse legal problems in an orderly way, and for them to learn how to teach themselves. Because the acquisition of any new skill can be laborious and time consuming, and because instructor time is typically quite limited, this self-teaching aspect is an important feature of the method. The self-teaching aspect relies on student access to an instructor-conducted demonstration of the method, followed by access to identically constructed problems that students work through, on their own, using a highly constrained, stepwise process. Each step in the process systematically produces corrective feedback that is simple to record, readily interpretable, and diagnostic of the skill needed to perform that step. Because the sequence of steps in the method is the same across all problems, feedback generated from one problem is directly comparable to feedback generated from other problems. A method is provided for recording, in one place, the feedback generated from a series of problems, so that patterns of skill mastery and skill deficit become visible.

The pedagogical foundations of the Logic Doctor method rest upon metacognitive monitoring and structure training. Metacognitive monitoring involves teaching students to use metacognition – ‘what we know about what we know’ – in order to monitor their thinking process, checking whether progress is being made toward an appropriate goal, ensuring accuracy, and making decisions about the use of time and mental effort.

Metacognitive monitoring is a combination of metacognition and monitoring. Metacognition, itself, is comprised of two functional components – knowledge of cognition and regulation of cognition. Metacognitive monitoring has an empirically demonstrated positive relationship to the regulation of study and the regulation of study has an empirically demonstrated positive relationship to test performance.

A common and very simple example of a system that cultivates metacognitive monitoring is the electronic form containing fields into which a user is required to insert information in a required format, such as online job applications, online event/organisation registration forms, and online product ordering forms. Typically, when a user submits an incomplete or improperly completed form, the system rejects the submission. If, in response to a defective submission, the system simply rejects the submission, the system is providing feedback of limited usefulness – it is indicating that the user has made a mistake, but gives no indication of where the mistake has occurred or how to remedy the mistake. If, in addition to rejection, the system highlights the incomplete or improperly completed fields, the feedback is more useful, since it is informing the user of where mistakes have occurred. As a user repeatedly encounters the rejection-only systems that provide the least useful feedback, the user learns only that he is mistake prone. Limited learning is promoted, and frustration is likely to be high. However, as a user repeatedly encounters systems that provide corrective feedback, learning is enhanced because the system promotes awareness of the types of mistakes to which the user is prone and how to prevent or remedy them, and frustration is minimised. Such corrective feedback systems prompt the user to monitor his behaviour in order to become aware of what he is doing, how well he is doing it, and how to improve what he is doing. The Logic Doctor is modelled after this latter type of system.

Structure training involves teaching students to ‘recognise or notice that a particular thinking skill may be needed’ and to ‘actively focus on the structure of problems or arguments so the underlying characteristics become salient instead of the domain-specific surface characteristics’. It is a method for cultivating, in students, the ability to recognise when an already-learned thinking skill is needed in a new situation. Consequently, the transferability of thinking skills from the contexts in which they were originally learned to novel contexts depends upon the ability of the problem-solver to recognise the structural similarities between the original problem and the new one, and to rely on such structural similarities as the cues that prompt selection of the correct problem-solving skill.

In simplest terms, structure training helps students learn how to solve problems, in general, and metacognitive monitoring helps them understand whether they are properly solving a particular problem.

Structure training and metacognitive monitoring are embedded in the features of the Logic Doctor. The analysis is broken into a sequence of described, demonstrated, repeatable steps. The nature and order of the steps in the sequence is identical across all problems, and the particular thinking skill needed to perform each step is brought to the students’ attention. Consequently, repeated experience with the method allows students to know, ahead of time, the nature and order of the analytical steps they will face, and what thinking skills they will need to accomplish each step. This foreknowledge will allow them to consciously monitor the congruence of their thought processes with those established by the Logic Doctor.

Within the confines of a given problem, the method is designed to discourage a student from moving past a misstep in her analysis. This encourages the student to ‘[make] decisions about the use of time and mental effort’, by forcing the student to focus time and energy on recognising and correcting mistakes as they are made, instead of looking for them later. And, finally, requiring students to interact with the method by providing specified information in response to written cues makes the monitoring skills conscious and public, exposing them to feedback and criticism, and making it possible for students to develop a more or less real time understanding of the functionality of each component of the analytical skill they are presently using. That being said, the ultimate goal of these features must be to habituate the student toward a systematic approach to thinking through problems, so that over time, the metacognitive monitoring promoted by repeated use of the Logic Doctor will lead the student to the stage where the method ‘like any other procedural skill that has become automatic, ... ceases to demand self-conscious attention [so that a]ttention is freed from the “method” to “the matter at hand”’.

The Logic Doctor method also provides the student with a mechanism for tracking the results of efforts across numerous problems so that patterns in skill functionality can be identified, giving student and instructor a rather precise insight into the exact skill or skills upon which improvement effort needs to focus.

The Logic Doctor, as a method of self-regulated learning, falls into a category known as discrepancy-reduction models. In discrepancy-reduction models, a recursive loop consisting of the establishment of a learning goal, followed by study, followed by the monitoring of learning progress and comparison of the results of studying with the established learning goal is repeated until the discrepancy between current knowledge and the original goal becomes zero.

The Logic Doctor has four operational components: (1) a five-step systematic analysis for problem solving; (2) a real time, feedback-based method for evaluating solutions at each step of the analysis; (3) a set of charts for students to use to track and analyse the accuracy of their step-by-step performance within a problem and across multiple problems; and (4) a rubric for interpreting the accumulated information generated by students’ performance to self-diagnose skill deficits and identify exercises designed to address those specific deficits.

The five steps in the problem-solving analysis are phrased as a series of questions. (1) What is the problem? (2) What rules govern the solution of this problem? (3) What information do the rules require me to know in order to solve this problem? (4) What information do I have? and (5) What is my answer/conclusion to this problem? These questions represent a refinement of the traditional Issue, Rule, Application, and Conclusion (IRAC) system of problem solving. The IRAC system prompts students to solve problems by identifying the issue or problem under consideration and the rules to be used to solve the problem, then applying the rule to the issue or problem to reach a conclusion. Questions 3 and 4 in the Logic Doctor problem-solving analysis offer a needed clarification of the analysis step of the IRAC system, breaking it into the explicit functional thought processes of using the rule as a criterion or set of criteria to determine what information is necessary, turning those criteria into straightforward questions, then searching the facts on hand for information needed to satisfy those questions and comparing the facts available to the requirements of the rule.

After reading the problem, the student begins the five-step analysis. At ‘Step 1: What is the Problem?’, space is provided for the student to state what she believes the problem to be, after which she will have an opportunity to view the instructor’s answer, and to evaluate whether her answer matches the instructor’s answer, by answering the self-evaluation question for this step: ‘Did Your Answer Match the Instructor’s Answer?’ She will then record her answer to this evaluation question in a tracking chart. The tracking chart and its uses are shown after the demonstration of the five-step process.

Recognising the existence and nature of a problem is obviously the threshold step in solving every problem. Consequently, Step 1 and the skills necessary to perform it are critical.

Given the importance of getting Step 1 correct, it is critical for students to appreciate that problems phrased in terms of the rules are ‘better’ than problems phrased in terms of everyday language. It should be part of the basic instructional repertoire for the instructor to help the student learn to translate vernacular phrases like ‘hear Jack’s case’ into more solution-friendly, rule-indicative language such as ‘subject matter jurisdiction over Jack’s case’. The importance of this approach flows from the fact that the existence and nature of problems encountered in business are often defined by explicitly stated rules. The significance or insignificance of facts, and how they are used, is determined by the precise words of the rule, so approaching problems with imprecise phraseologies increases the risk of incorrect solutions.

The next step, ‘Step 2: What Rules Govern the Solution to this Problem?’ requires the student to state the rule or rules that will be used to perform the analysis.

Access to the correct information, at each step, allows students to make midcourse corrections, reducing wasted time and effort and frustration. After attempting an answer, and accessing the instructor’s answer, as shown above, the student should take away the notion that fully stated rules are the goal. This need for fully stated rules should prompt instructors to produce complete, coherent statements of the rules they want their students to master.

Step 3 has two sub steps. First, the rule must be broken into its component parts, and then those components must be turned into yes/no questions. As with the earlier steps, the student can attempt an answer, obtain immediate feedback by comparing her answer to her instructor’s answer, perform an immediate self-evaluation by comparing her answer to her instructor’s, and then record the outcome of the evaluation in the tracking chart.

The next sub step in Step 3 requires the student to turn the rule components into yes/no questions. While this sub step is fairly straightforward, it is very important since the yes/no questions will direct the student’s examination of the facts.

The student now moves on to Step 4, which will require her to seek answers to the yes/no questions from the information given in the problem. Students should be encouraged to quote directly from the facts.

Thought processes that should be explicitly encouraged at this step are: (1) comparison of given facts to the questions; (2) recognition that answered questions indicate required information that is known; (3) recognition that unanswerable questions indicate required but unknown information – an information deficit; and (4) the decision to cure an information deficit with either external research or an assumption should be a conscious one.

After performing the self evaluation, and recording the outcome, the student is now in a position to use the contents of the ‘Information I Have’ column to answer the yes/no questions in the last column.

At this point, the student is ready to proceed to ‘Step 5: What is My Conclusion’ – which requires the student to interpret the pattern of answers to the yes/no questions. For this part, students should be reminded of the significance of the disjunctive and the conjunctive – that if the criteria specified by the law are disjunctive, then a ‘yes’ to any of the criteria would produce a ‘yes’ conclusion, and that if the criteria are linked conjunctively, then a ‘yes’ to all criteria would be required to produce a ‘yes’ conclusion.

While the insights gathered from working just one problem are useful, the insights gathered over a series of problems of similar difficulty are much more significant. Hence, the third operational component requires the student to keep track of her successes at each step, across a series of problems of similar difficulty. She is now ready to use the fourth operational component, which deals with diagnosis and remediation.

Because these tracking charts facilitate self-diagnosis, they need not be handed in to the instructor; however, once a student is armed with the knowledge the tracking chart provides, the instructor should be approached for additional problems that focus the student’s efforts on just those skill areas in which improvement is indicated. The instructor can encourage students to request exercises targeting these skills, or the instructor can make appropriately labelled and targeted problems available online. Once the method is introduced and demonstrated in class, the online availability of the exercises is usually sufficient.

The value to students of being able to determine, rather precisely, which skills need work, would be lost, unless skill-specific exercises, focused on their specific skill deficits, were available. There are two basic approaches to providing students with skill specific-exercises: (1) assigning problems dedicated to only the skill deficiencies revealed by the information in the tracking chart and (2) assigning comprehensive problems, but directing the student to begin at the step, or steps involving the skills on which they need to work.

There are a number of ways to implement the Logic Doctor. The first and simplest way is to do so in hard copy. With this method, students are provided with hard copy workbooks containing the problems, empty screens and blank tables for working through the five-step problem solving method, blank tables for tracking results across multiple problems, indicators of problem difficulty, and the step-by-step solutions to the problems. A second, more labour-intensive, but very student-friendly method would be to create interactive versions – a variety of software is available for this. Regardless of the implementation method, the Logic Doctor provides students with the opportunity to engage in repetitive practice, guided by instantaneous corrective feedback.

The value of the Logic Doctor can be further enhanced by the use of problem difficulty ratings. If all of the problems in a set are of the same difficulty level, students gain insight into not only the mechanics of their analysis skills, but also into the level of their skills. Instructors can facilitate the development of these additional insights by assigning a series of problems of the same difficulty, followed by assignments of series of increasingly difficult problems, where the problems within each series are of equal difficulty, but the difficulty of each successive series is greater than that of the preceding series.

The initial investment of class time needed to demonstrate the method is approximately 30 – 40 minutes. The subsequent investment of time by the student varies, of course, according to the quantity and quality of study time. Assignment of increasingly difficult examples/problems to be worked by students, on their own, prior to the class period in which the subject matter of the problems is to be discussed provides an incentive for students to continue with the method. And, while the investment of time and effort can be substantial, as one exponent of the value of metacognitive monitoring has observed.

The goal of the Logic Doctor is to provide students with a chance to develop transferable problem-solving skills using a repeatable technique based on structure training and metacognitive monitoring – approaches designed to foster the development and recognition of specific thinking skills. The method serves the structure training function by exposing the conceptual infrastructure common to many legal problems, thereby cuing the student to call into play the appropriate sequence of thought steps, in response to that structure. It serves the ‘thinking about what you’re thinking about’ function of metacognitive monitoring by externalising and documenting the thought steps needed to perform legal analysis and by providing corrective feedback, at each step, thereby isolating any point of error and permitting the student to engage in midcourse corrections. A central benefit of monitoring – the ability to distinguish what needs to be studied from that which does not, which leads to better studying and better learning – is realised by the ability to track performance, and identify explicit skill deficits, across a series of problems.