Adcroft, A --- "Speaking the same language? Perceptions of feedback amongst academic staff and students in a school of law" [2011] LegEdDig 4; (2011) 19(1) Legal Education Digest 12

• Speaking the same language? Perceptions of feedback amongst academic staff and students in a school of law

---

Speaking the same language? Perceptions of feedback amongst academic staff and students in a school of law

A Adcroft

The Law Teacher, Vol. 44, No. 3, 2010, pp250-266

Within both the generalist and the specialist legal education literature, there is a growing body of evidence which suggests that there is growing concern about the quality of feedback to students and how it impacts on learning.

The review begins, therefore, with the assumption that feedback is central to learning and a crucial part of the whole student learning experience. This assumption is accepted by practically all the literature from both a generalist and specialist legal education perspective.

Quality feedback, according to Miller, should clarify what is expected of students. Gibbs and Simpson elaborate on this by suggesting that quality feedback will have a number of characteristics based around its specificity, factors under the control of students, its timeliness and that it is in a form which allows students to both receive and act upon it. The logical outcome of this is that the diverse student body in higher education will require feedback in a variety of forms and through a variety of processes if it is to have a positive impact on performance. This approach mirrors the work of Nicol and MacFarlane-Dick who offer a number of principles which should underpin quality feedback such as how it clarifies what good performance is, promotes independent learning, motivates and encourages students and identifies gaps in knowledge and understanding. Broadly speaking, there is little in this issue which distinguishes feedback in legal education from feedback more generally across higher education. The real test of feedback is more in the implementation of these principles than in the principles themselves.

The key question is, therefore, why there is a gap between the principles which underpin the practice of feedback and the outcomes which result. Vardi argues that this gap is the result of poor practice such as feedback being too brief, not specific enough, involving arbitrary judgements about standards and using terms which may be vague, cryptic, sarcastic and lacking in praise. Burke also suggests that poor practice is the primary cause of poor outcomes: ‘it is too brief, too negative, too difficult to decipher or to understand’. For many, however, this explanation is too one-dimensional because the cause of the problem is a gap in expectations between academics and students. The main manifestation of this gap is the body of evidence which suggests that students do not understand the content and purpose of feedback: Crisp points out that what may be ‘self evident’ to academics is often not to students; Orrell raises issues of the ‘expert language of academic disciplines’; Hyatt argues that the ‘rhetorical conventions’ of feedback are frequently confusing to students; Williams suggests that academic language is ‘opaque to many university students’; and Weaver’s evidence suggests that students do not have a significant understanding of academic discourse to make best use of feedback.

The sample for this study was drawn from undergraduate students in the School of Law (SoL) at a pre-1992 university in the UK. SoL is predominantly an undergraduate school; just over 95 per cent of all full-time students study on SoL’s undergraduate programmes in Law and, of these students, over 90 per cent study on the LLB degree. All degree programs are three years in duration and students have the option of undertaking a professional training year between the second and final year although very few students take this opportunity. The School has 24 members of academic staff in a range of posts from professor through to reader, senior lecturer, lecturer and tutor.

The method of data collection chosen for this study was primarily determined by the ethical regulations of the university in which SoL is located. These regulations insist that students are made aware of a clear separation between their degree studies and surveys they may participate in. In practical terms this meant that a paper based survey distributed in, for example, large lectures was not possible even though ‘response rates for web surveys are lower than those for paper and pencil surveys’. The instrument for the study was, therefore, administered online and all undergraduate students and academic staff in SoL were invited to participate.

In total, 240 fully completed responses were received from students giving an overall response rate of 44 per cent. There is variation in response rate across levels of study. These response rates were, on the whole, not surprising although the response rate amongst second year students was a little disappointing. This is, however, consistent with Sax et al’s study which shows that university students are ‘responding at lower rates than in previous decades’. The study does offer significantly higher response rates and sample size than similar studies. No demographic data such as age, gender or ethnicity were collected on advice from the University’s Ethics Committee. The implications of this are two-fold. First, any conclusions drawn from the data must be tentative as generalisability may be an issue and, second, findings of this study need to be examined in relation to findings of previous studies in this area.

The study was conducted in two parts with a similar questionnaire used in each. The first part of the study was for academic staff and the second part of the study was for students. In both cases, invitations to participate were emailed, reminder emails were sent weekly and the survey was kept open for four weeks. The questionnaire for the survey was in three parts. The first part contained 15 questions, modified slightly between the student and staff parts of the study, which asked about expectations of feedback, outcomes of feedback and experiences of feedback. For each of these questions, staff and students were offered a statement and asked to respond on a five point Likert scale which ranged from ‘Strongly disagree’ to ‘Strongly agree’. All the questions were derived from the generalist education and legal education literature on feedback and were pre-tested in a focus group of five academic staff and a focus group of eight undergraduate students. One of the most significant outcomes of the focus groups was the discrepancy in understanding of feedback between academics and students which manifested itself in a number of ways. Academics, for example, were keen to stress the importance of the diversity of feedback given to students in terms of both content and process whereas students were much more focused on feedback as something which was written on assignments. In assessing these discrepancies, therefore, the instrument was designed to assess the diversity of feedback offered and also to ask specific questions about written feedback

The second part of the questionnaire concerned the types and frequency of feedback given by staff and received by students. From the staff focus group, a list of different mechanisms through which feedback is given to undergraduate students was produced. In order to ensure validity, this list was cross-referenced with the literature on types of feedback to produce a final list of nine items which were then used in the study; staff were asked about the frequency with which they used each type of feedback and students were asked about the frequency of receiving each type of feedback and how useful they found it. A four point Likert scale was used to measure frequency (ranging from ‘Frequently’ to ‘Never’) and a five point Likert scale ranging from ‘Not at all useful’ to ‘Very useful’ was used to measure usefulness. The final part of the instrument concerned the use and understanding of words and phrases used in feedback. Staff were asked in the focus group to produce a list of frequently used words and phrases in feedback to undergraduate students which was, again, cross-referenced with the literature. From this, a final list of 12 items was created and staff were asked how often they used these phrases using a four point Likert scale ranging from ‘Frequently’ to ‘Never’. An open comment box was also used in the staff survey for staff to list additional comments but no additional suggestions were provided. In the student version of the survey they were first asked about how frequently they received these words and phrases in their feedback and they were also asked to comment on how confident they were in understanding what these comments meant using a four point Likert scale which ranged from ‘Very unsure’ to ‘Very confident’.

Within SoL there are a number of key differences in the perceptions of staff and students across all of the different dimensions of feedback investigated.

The most significant difference in perception between academics and students is that concerning the role that feedback plays in improving performance; almost twice as many students as academics felt that feedback is the crucial mechanism through which a student’s performance is improved. The extent to which feedback helps students self-diagnose their performance also shows marked differences whereby a much higher proportion of staff felt that this was a key role of feedback compared to students. There is much less of a gap between staff and students as far as establishing what good performance is and identifying gaps in knowledge and understanding. This contributes to the mixed signals about the expectations of feedback generated by the data. There is also an interesting question about how important feedback is to the whole student experience; practically all staff surveyed placed feedback as a central component of the wider student experience compared to just three in four students who felt the same way. If there are gaps in the expectations of what feedback should deliver, there are also significant gaps between what academics believe feedback does deliver and the beliefs of students.

Students do not believe that feedback delivers on many of its key purposes. For example, just under half of the students surveyed felt motivated and encouraged by feedback or that feedback helped them improve their study habits and only slightly more than one third of students felt that feedback helps them progress towards independent learning through self-assessment and self-correction. More positively, a significant majority of students did feel that feedback was successful in bridging the gap between how they did perform and how they wanted to perform. Student perceptions of outcomes, however, stand in stark contrast to those of academics, who have much higher perceptions of what feedback delivers. For example, two-thirds of staff believe that feedback motivates their students yet less than 50 per cent of students feel the same way. These differences in perception are, in many cases, at their most extreme when we consider the experiences of students as far as feedback is concerned.

There is clearly an issue between what academic staff in SoL are doing and what students believe they are receiving. For example, 88 per cent of staff say they give frequent feedback which goes beyond assessed work but only 12 per cent of students say they receive this kind of feedback. Almost all academics say they give a variety of written and non-written feedback but only one in four students claim to receive both written and non-written feedback and whilst just one academic felt that a grade or mark is the most important element of feedback, more than one in four students have the same belief. Overall, therefore, the gap between academics and students appears to be very wide and this is the case even when they meet directly; over three-quarters of academics believe that students get feedback whenever they meet to discuss an academic issue compared to just 12 per cent of students

There are five different types of feedback which more than 80 per cent of academics suggest are delivered frequently or often to students compared to just three which students say they receive frequently or often. The most popular form of feedback used by academic staff, individual written comments on assessed work, is received by only 61 per cent of students even though 100 per cent of staff use this type of feedback. The converse of this is feedback in the form of just grades on a piece of assessed work which a quarter of staff use but 61 per cent of students receive. In terms of variety, therefore, there is a gap between the types of feedback that are given and the types of feedback that are received. Perhaps a more important issue is not what types of feedback are used but rather how useful students find different types of feedback.

On a positive note, the most useful form of feedback that students receive is individual written comments on work which all staff give on a regular basis (although only 61 per cent of students receive it). The relationship between usefulness and frequency, however, does raise a number of areas for concern. For example, 80 per cent of students believe that model answers are a useful feedback mechanism but only 37 per cent of students feel they regularly receive this kind of feedback (compared to 69 per cent of staff who say they give this kind of feedback). Conversely, 94 per cent of staff give feedback on a group bases whereas only 41 per cent of students find this feedback useful. Similarly, the two most frequent forms of feedback students say they receive, general feedback and grades only feedback, are seen by students as being less useful than four other forms of feedback.

There is a common vocabulary of feedback used by staff. For example, all staff comment on the structure of a student’s written work and 94 per cent of staff comment on how an assignment is referenced, its descriptiveness and the quality of critical reflection it contains. The key issue is, however, not necessarily what is written but what is understood by students. Almost one-third of students have only a limited understanding of what the most commonly used phrase in feedback means and, of the 12, seven are not fully understood by more than 30 per cent of students. The effort put into feedback by academic staff does not, for a substantial minority of students, result in understanding. What these results suggest is that there is a clear breakdown between the work that academic staff are doing in giving feedback to students and what students are receiving in terms of feedback and the outcome of this is that staff and students are working with different levels and types of expectation, they experience feedback in fundamentally different ways and do not share much of a common understanding of what feedback is and what it is supposed to do.

The first implication of this study is that there is little which makes feedback in legal education stand out from the rest of the higher education sector; it faces the same pressures, is built on the same assumptions and has many of the same problems. In the more specific issues raised by the data, there are many similarities between SoL and other elements of higher education. For example, Parikh et al identified a gap between what is given and what is received in terms of feedback, Williams noted the difficulty many students have in understanding the language used by academics and Crisp’s work pointed out that many students are more interested in grades than feedback, all of which are conclusions drawn from the data in this study. There is also evidence which suggests that SoL is not exceptional compared to other schools or departments of law in relation to feedback issues. As well as the psychological comfort that may come from knowing that you are not alone is dealing with difficult issues, the positive from all this is that SoL (and legal education in general) should not be deterred from looking outside the world of legal education for ways to deal with these problems.

The most important of these issues for SoL is the perceptions gap between academics and students; there seems to be little common understanding of what feedback is and means. Staff think it more important than students in the wider learning experience, staff see it as an ongoing iterative process compared to students who see it mainly in terms of assessment; staff focus on a wide variety of feedback mechanisms whereas students feel they receive only a few and staff give feedback in forms that students cannot engage with. It is unlikely that there is any quick fix for this kind of problem, especially when the growing emphasis on independent learning in higher education is increasingly at odds with the ethos in pre-university education in the United Kingdom. Studies from elsewhere suggest that this gap can only be addressed and closed with a systematic and coherent set of interventions which go beyond individual effort in the classroom. Mutch, for example, sees this as an issue of ‘program design’ which begins with how students are inducted and oriented into higher education and continues throughout their studies. With this kind of effort it could be possible to soften the contrast in SoL between National Student Survey (NSS) scores which rate feedback as poor and external examiner reports which are much more positive.

The main manifestation of all this is in the lack of understanding of the key terms used in feedback. This is a problem for two reasons. First, as Bone has pointed out, for law students feedback only has value if it is acted on and students in SoL cannot act on what they do not understand. Second, much of the lack of understanding is in the higher level academic skills like critical analysis and application of theory. From a staff perspective there is also the issue of workload and, in a lot of cases, the pointless exercise of writing feedback that is not understood. There is more to dealing with this issue than, for example, handing out dictionaries to all new undergraduates as this is probably a symptom of the wider systemic problem identified earlier. Nevertheless, the basic point that feedback must be understood to be acted on should not be lost especially in an increasingly competitive sector where the NSS and the promotion of independent learning are likely to become more, not less, important.

There are two main limitations of this study. The first is that it focuses on just one school from one university and so questions about generalisability are inevitable. Given that, demographically, there is little that makes SoL’s staff or students particularly different from other schools or departments of law in the same part of the sector and that the evidence presented is consistent with evidence from elsewhere, it is probably realistic to make the claim that SoL is pretty typical of the legal and higher education sector overall. The second limitation is that the paper offers a snapshot of perceptions of feedback rather than a running commentary which tracks how these perceptions change over time. The data gathered do allow for analysis of different levels of study, analysis which is beyond a paper of this length, and which is perhaps the best suggestion for future research in the field.

Oscar Wilde suggested that truth was rarely pure and certainly never simple and so too is it with feedback to law students. What this paper illustrates more than anything else is that there is probably no natural or automatic process through which something is transmitted from an academic and arrives at a student unencumbered and unchanged by experience, expectation and interpretation. What the paper does not really illustrate is that it is often easier to identify a problem than it is to solve a problem and in this specific case the problem and solution is multi-dimensional and complex. Dealing with this is most likely one of those rare and simple pleasures that can come from more research in the area.