Burns, K; Hutchinson, T --- "The impact of 'empirical facts' on legal scholarship and legal research training" [2010] LegEdDig 13; (2010) 18(1) Legal Education Digest 46

• The impact of ‘empirical facts’ on legal scholarship and legal research training

---

The impact of ‘empirical facts’ on legal scholarship and legal research training

K Burns & T Hutchinson

The Law Teacher, Vol 43, No. 2, 2009, pp153-178

There is a vast array of valid social science research available to the modern lawyer. Research of current Australian, United States and United Kingdom judicial decisions demonstrates that judges do refer to empirical facts and sometimes refer to social science research as part of judicial reasoning. There are existing rules of evidence in all jurisdictions allowing for a formal use of empirical data within the doctrinal framework. However these existing rules of evidence do not appear to adequately cater for the wide variety of ways in which empirical facts are utilised in judicial decisions. Increasingly, empirical fact assumptions and sometimes social science material are being subsumed within judgments. The way this material finds its way into judges’ decisions appears to rest primarily upon judicial discretion. Social science material relevant to empirical fact assumptions is not always (or even often) adequately acknowledged by judges. The recognition of the judicial use of empirical facts as part of judicial reasoning raises the need for new approaches to legal research and legal research training based in the social sciences. It suggests that lawyers need better training in non-doctrinal methodologies.

In 1942, Kenneth Culp Davis argued that there were two types of facts used by judges – ‘legislative facts’ and ‘adjudicative facts’. Adjudicative facts are not included within the definition of empirical facts in this article. Adjudicative facts are facts found by judges as part of litigation. They tend to be limited to the litigants in the specific dispute and are normally subject to the usual rules of evidence.

Where a ‘court or an agency develops law or policy it is acting legislatively’ and Kenneth Davis called the use of facts in this context ‘legislative facts’. Legislative facts aim to define legal contexts and relationships in society as a whole. Monahan and Walker have also noted that courts may use social science to ‘construct a frame of reference or background context for deciding a factual issue crucial to the resolution of a specific case’. They refer to this as ‘social framework’.

Justice Graham Mullane, in a study of 1990 Australian Family Court cases, discussed the use of assumptions by judges ‘concerning human behaviour’, which he called ‘social facts’. Kylie Burns has defined the term ‘social facts’ more widely as including the ‘continuum of assumptions judges make about society, the world and human behaviour’ in their reasoning. It is apparent from both Burns’ and Mullane’s studies that judges may sometimes refer to empirical evidence in support of these kinds of assumptions, but far more commonly there is no evidence provided or referred to in the judgment.

We define ‘empirical facts’ in this article as assertions of facts about society, the world and human behaviour which are hypothetically able to be proved by social science or empirical methodologies. This category includes Davis’ legislative facts, Burns’ and Mullane’s social facts and Monahan and Walker’s social authority and social framework. Similar to these categories, empirical facts are not statements of legal principle or adjudicative facts. They are assertions used as part of the judicial reasoning process. They may be used in a wide variety of ways by judges in their reasoning. They may be used to set background context, in a rhetorical way to support arguments of legal principle, to assist in the determination or interpretation of adjudicative facts, or as arguments of policy or consequence used in the development of law.

Justice Mullane, in a study of 302 final custody judgments from the Family Court of Australia in 1992, found 82 social fact statements. Sixty-five per cent of these had no source stated or the source was stated as undefined research.

The Burns study considered 11 negligence cases handled by the High Court of Australia in 2003. Burns found 325 statements of social facts in the relevant judgments. The vast majority of social fact statements made by judges were unsourced and only three social fact statements were sourced to a form of social science or empirical evidence. The social fact statements made in the cases were made by both judges considered ‘activist’ in judicial approach and judges considered ‘conservative’ in judicial approach. Similarly, Bradley Selway, in a 2001 study, identified many examples of the use of history and other facts in the judicial reasoning within High Court of Australia judgments.

The use of this form of empirical fact material in judgments in the United States has also been well documented. Many empirical fact statements are made by judges implicitly and without any empirical support. However, judges do sometimes explicitly reference empirical or social science material in their judgments. It is less common in the United Kingdom and Australia; nevertheless there are examples of the use of social science material in judgments.

Doctrinal research has been the dominant influence in legal scholarship during the 19th and 20th centuries. However, limiting legal scholarship and research training to traditional doctrinal analysis has obvious limitations when lawyers (and judges) are being confronted with the need for and the relevance of results of empirical and interdisciplinary scholarship.

Traditionally law has been viewed as a closed system. In terms of legal research scholarship and research methodologies it has meant that lawyers have looked at the law in isolation. The sources of law have been the primary materials, the doctrine of the law – the case law and legislation. The research carried out has largely been confined to an analysis of legal doctrine. Thus doctrinal research is the established traditional territory of the lawyer-researcher. As a result, where legal research has been taught in the law schools the methodology taught has been doctrinal research.

What is evident from this study on the use of empirical facts in the courts is that lawyers need to look at the law from a much broader angle than has been done previously. This is a quite concrete example of how the law does not work within a vacuum. Therefore, as researchers, lawyers need to be totally cognisant of the parameters of empirically based knowledge and research methodologies.

More extensive training needs to be offered in fundamental research. Fundamental research, which can include empirical and social science models, needs to be part of the graduating lawyers’ research skills and attributes.

There is a growing empirical law movement in the United States at present. As Tracey George has noted, the Association of American Law Schools (AALS) has had a section devoted to ‘social science technique’ since 1982, when it established the Law and the Social Sciences Section. The terms of reference for this interest group were ‘to promote communication among those persons who are interested in using the empirical techniques of the social sciences to study legal problems and institutions’.

Robert Ellickson, in a recent citation analysis of trends in US legal scholarship, found that ‘number crunching’ is also rising in law journals.

The New Realists are aware that the movement has ‘jurisprudential implications’, but this is certainly not the focus of their work. Much of their research focuses on links between judicial behaviour, gender and politics.

The New Legal Realism project jointly sponsored by the Institute for Legal Studies and the American Bar Foundation is an example of the new movement. It is a network of scholars who are developing an interdisciplinary paradigm for empirical research on law. In Canada, Roderick Macdonald writing in 2003 states that ‘Published research by law teachers is still overwhelmingly doctrinal and oriented to the professional tasks of planning, dispute avoidance and dispute resolution’. However, Shanahan’s 2006 survey of legal academics demonstrates that legal academic researchers are using non-doctrinal methodologies to some extent.

Anthony Bradney stated in 1998: ‘The academic doctrinal project which has dominated United Kingdom university law schools for most of their history ... is now entering its final death throes’. The 2006 Nuffield Inquiry on Empirical Legal Research concluded that there was an unmet need for empirical research stemming from a lack of capacity to undertake this type of research in the research institutes. The Report notes that ‘Empirical legal research is increasingly important to and valued by policy makers, law reformers, the judiciary, academics and practitioners’ but also that there is ‘clear evidence of a developing crisis in the capacity of UK universities to undertake empirical legal research’.

The empirical studies movement is not as strong in Australia. The Australasian Law Teachers Association (ALTA) has no empirical legal studies interest group. There is a Law and Social Justice Interest Group and a Legal Research Communications Group which has a focus on research methodologies including the promotion of empirical approaches. However, the Law and Society Association of Australia and New Zealand which aims ‘to promote and foster scholarship broadly focusing on the interactions and intersections between law and society’ has a growing profile and maintains links with national and international socio-legal associations.

It is timely to consider how we as legal educators might inculcate these skills in our graduates – and especially in our academic track higher degree research students.

Over the past decade there has been recognition that the law cannot be confined to a ‘black letter’ box. There has also been a move towards some relatively ‘safe’ research methods extensions including research into the philosophy underlying legal rules (theoretical research), research into the reform of legal rules (law reform research), and research into the policy behind legal rules (policy research).

Internationalisation and the advent of transnational legal contexts (especially in the number of international students entering the Australian law faculties) has popularised the comparative law methodology. Much of the empirical work has involved the ‘systematic and quantitative analysis of judicial decision making’.

Content analysis has been used to reveal the role of empirical facts in judicial reasoning. Cases are read and particular features (for example categories of comment) are coded.

Content analysis is used to evaluate ‘the influence of other disciplines (such as economics) on legal scholarship, the sources which influence judges when they draft judgments and the influence of particular articles, scholars and legal journals’. Content analysis is a rapidly developing methodology in the United States to study the content of judicial decisions. However, the methodology has been rarely utilised in Australia.

When we look at the law more widely and when we look at how the law actually works, it is obvious that law teachers need to revise and widen their views on what they are teaching their students in terms of research methods. Training in traditional doctrinal analysis methodologies does not equip students to deal well with empirical facts. In 1992, Twining noted that the ‘use of statistical arguments in court and in other contexts is developing fast in the United States and is likely to spread to other parts of the common law world well before the year 2000’. He termed this the ‘new evidence scholarship’. Twining also made the point that ‘in my experience most lawyers are innumerate and most law students are terrified of figures’. It is worthwhile noting too that Twining thought it ‘extremely unlikely’ that competence in empirical research could be developed by ‘quick fixes of CLE’.

Empirical research enhances lawyers’ ability to understand the implications and effects of the law on society. Legal researchers can use social science methodologies themselves to investigate issues, or they can collaborate with skilled researchers from other disciplines. They are able to use statistics freely available and gathered by governmental organisations to enhance their views on the law’s operational aspects. This strategy has very definite advantages for unskilled lawyers as it saves time and ensures accuracy and that public verification of the data has already occurred.

It is easy to speculate on the reasons for reluctance to move beyond the familiar doctrinal methods. Often lawyers do not have the skills to use publicly available datasets effectively. There is a lack of training in the undergraduate degree for non-doctrinal methods of research. Lawyers perceive they have insufficient expertise in order to judge empirical studies. It requires more time to undertake empirical work than doctrinal work. It costs more.

Empirical research is more inconvenient. The results are often uncertain and certainly not predetermined. Elementary errors can be fatal to the outcomes. Even a simple survey entails precision in sampling, wording of the questions, coding of the questionnaire for easy entry of returned data, conduct of speedy ethical consent processes, provision for privacy with returned forms and follow-up communication with those being surveyed. In addition, there is often a requirement to work as part of a group – and often an interdisciplinary team. This requires extra time and commitment. And once the research is completed and the reports written, there can be uncertainty in regard to where to publish – whether in a legal journal or an interdisciplinary one. The method and citation style for writing up the research will be different for each. The level and depth of analysis will be different. In all, therefore, using non-doctrinal methodologies equates to less control over the process and outcomes than doctrinal work.

There is a wealth of general social data that is used to some extent by the legislature but that also impinges on legal decision-making. There is a need to introduce students to the existence and nature of interdisciplinary research – the extensive work of anthropologists, criminologists, economists and sociologists that impinge on the law. Law schools need to introduce a wider range of research methodologies into their research training particularly those based in the social sciences. Students must be aware of the basic principles of social investigation, where to source publicly available information, and how to critique empirical research from the perspective of validity and reliability. They must be able to distinguish valid empirical research from anecdotal evidence. This means that empirical methodologies must be introduced into the law curriculum so that law students can deal with empirical facts in a knowledgeable fashion.

In doing this, legal academics have a role in ensuring that students are aware that there are various components in the judicial reasoning process – including the evidence and legal principle, but also facts based in the judge’s views and information based in the social sciences.

A number of Australian law faculties conducted curriculum reviews during 2007 and 2008, so there is constant flux in the degree offerings nationally. Legal education has embraced skills in the last decade under the rubric of graduate attributes. However, each Australian law school curriculum must include the subject areas identified by the Priestley Committee in 1992. There are strong views from many legal educators that the Priestley 11, which is skewed towards substantive rather than skills-based instruction, is ‘a significant constraint on re-formulating Australian legal education in ways that are modern and relevant’. However, the answer to this issue is not simply to take substantive material out of the degree and replace it with additional methodologies training. This is more a matter of practical exposure and appropriate treatment of empirical methodologies and evidence within the degree.

Educational theory suggests two approaches that are relevant to any attempt to enhance non-doctrinal research training. The Cognitive Apprenticeship approach to teaching advocates embedding ‘learning in activity’ and making ‘deliberate use of the social and physical context’. Secondly, current educational theory suggests the embedding of generic skills (which includes research methods) into the law curricula as a better framework than simply adding elective units at the end of the degree. It would involve a whole of curriculum approach but especially centring skills training in the legal research units.

At present there are a variety of opportunities for a law student to participate in empirical methodology training. This includes prior training whether at school or in a prior degree to law, combined degree offerings, electives offered within the law degree, components of core units, or the opportunity to participate in an elective from another discipline as part of the law degree.

In a March 2008 survey of the curricula from 29 law schools’ websites in Australia, it was evident that very few courses explicitly included empirical training in their law degrees.

In the United Kingdom, Caroline Hunter at York Law School and United Kingdom Centre for Legal Education are currently carrying out similar research into the use of empirical research in the undergraduate law curriculum. This project is being funded by the Nuffield Foundation, and is seeking data on (1) Whether undergraduates are being taught skills that would enable them to either carry out or critique empirical work; (2) Whether they are actually carrying out empirical projects of their own; and (3) Whether empirical work figures in other ways in teaching and assessment.

However, in terms of the overall law curriculum, research training units are now competing for space with other skills training as well as traditional substantive law content. Additional compulsory methodologies modules are therefore unlikely to find favour with administrators. At the very least, existing research modules may need to be remodelled to integrate some coverage of empirical methodologies.

The most efficient method would seem to be to expose students to the methodologies within compulsory undergraduate units, include further basic training within compulsory research units, and in addition to provide elective units for those seeking to augment the initial training.

At a very preliminary level, the challenge is to highlight empirical fact assumptions from first year. This can be achieved through the discussion of simple student surveys, and the thoughtful use of statistics and relevant empirical material in course content. Interdisciplinarity can be introduced for example through a discussion of policy considerations in tort law presentations or empirical evidence highlighted in criminal law contexts. Supervisors and legal academics can model the use of empirical methodologies for students. Students might be encouraged to undertake short informal surveys for group exercises, undertake interviews or access interdisciplinary material as part of the criteria of their assessment. Social science evidence can be highlighted within substantive areas, but enhanced treatment in evidence law units is of course warranted.

Critique is an important accompanying skill. Law students above all need to be able to critique arguments that include empirical research effectively. Lawyers need to be able to critique research that others have carried out in order to judge the reliability of empirical data whether it is discussed as evidence – or reported in the newspapers as fact.

Postgraduate students need exposure to the range of research methodologies possible for their projects. This requires an introduction to methodologies to augment the doctrinal work with which they are familiar. Some students might embark on extensive empirical methodologies. Others may consider a simple survey. Postgraduates also need information on research ethics and the process of requesting ethics approval from the relevant university committees.

To effectively introduce empirical facts recognition and awareness the material has to be introduced as part of assessment in units. This is more difficult to accomplish. One reason is that even for those units where students are at liberty to choose their own topics and their own research methods, there are time limitations involved. In Australian universities, there are often only 13 weeks in a semester. Even providing the students have their topic clearly defined at the beginning of the semester, there is still a lag time required for the ethics approval procedure and a simple survey can take time to set up.

Central to this discussion is the cost of teaching research to large student bodies. In the current context in Australia, the overall numbers of students entering law schools have increased dramatically. Legal research requires academics with specific expertise. It is time consuming to teach. The levels of marking tend to be higher than a normal substantive unit. In this context, it would seem that the ability to include additional non-doctrinal research training is less likely without a positive recognition of need.

For a very long time, the Nuffield Foundation and the Economic and Social Research Council (ESRC) have been concerned with the lack of capacity to undertake rigorous empirical research on the law. However as with the Nuffield Inquiry’s Final Report, Adler concludes that the problem is ‘a structural one which reflects the relatively weak position of socio-legal researchers and, in particular of those who conduct empirical research in law schools, and the absence of any real incentives that would encourage law schools to take postgraduate training in socio-legal studies seriously.’ The situation in Australian law schools is by and large similar.

This article argues that empirical facts are an established part of the judicial reasoning process. However, lawyers have not been trained sufficiently well to deal with this information or to use it effectively. In addition, the evidential rules and legal process are not sufficiently open to the effective use of this data in the courts. It is time that we as lawyers recognise all the aspects of the process necessary to deal with the modern factual context. Having done so, this will have quite far-ranging effects on the way law and indeed legal reasoning is taught.