Evaluating the Quality of Legal Scholarship in Italy: Two Investigations on Peer Review


Download presentation

Aim: Research evaluation in legal science is a delicate and complex process due to the fact that legal disciplines are not monolithic: there are profound differences between the various branches of law, for which the tools of communication are very different. Legal scholarship is both the science of law and one of the authoritative and influencing sources of that law (1).This is why there is a strict correlation between legal science and legal practice (2). As a result,legal science has to pass two “exams”: a quality test within legal academia, which first evaluates its robustness as scientific research and secondly, assesses the pertinence and relevance to legal practice. These overlapping dimensions produce legally relevant knowledge, which should both be considered in the process of evaluating legal science.

Methods: In this context two research studies have been carried out, both of which provided significant results concerning the evaluation of legal scholarship in Italy. The first study was conducted by the Institute of Legal Information Theory and Techniques of the National Research Council (ITTIG) on the assessment of legal research monographs in Italy (3). TheProject’s objective was to check both the possibility and opportunity to identify bibliometric indicators that can support the peer review process. A structured questionnaire was sent to 4,729 professors and researchers of Italian universities, identified using the public database of the Ministry of Education, Universities and Research (MIUR). Scholars belong to the 21 subject areas which make up the area of legal sciences. 4,645 invitations to participate have been sent via email. The invitations have arrived to 4,501 respondents (target population) representing95.2 % of the workforce resulting in the Ministry database. The questionnaire was made available online for three months and a total of 1,241 answers were received, representing about 26 % of the target population.The second study, a survey on evaluation process of legal journals identified as A rated journals,fits precisely in this panorama. This survey, always conducted by ITTIG in the second semester of the 2017 until March 2018, was focused on the peer review process and evaluation criteria indicated in the most important Italian legal journals.The list of legal journals taken into consideration for our survey is updated to March 2018 and includes 96 journals. The aim of the survey is to provide an overview of the situation of the review process of the A-rated Italian legal journals. In particular, the intent is to verify if and which procedures have been adopted by the various legal journals for the evaluation of the individual submissions, if, among them, can be identified some prevailing ones and what are, if existing, the specific qualitative criteria to be followed in order to accept and publish the contribution. Identifying the indications on the evaluation of the papers has not always been immediate and simple. In most cases the indications were found at the beginning or at the end of the contribution, however, in some cases, the journal did not show any indication except for referring to the journal’s website. Greatest difficulties were encountered when the journals provided an evaluation procedure referring to each single paper, so that the evaluation indications were to be found at the beginning of each contribution. It is to be noted that 6 % of the journals considered did not present any reference to an evaluation process and that certain contributions were not subject to evaluation, as they are written by “scholars of prestige and clear fame”. In this case, it is the Editor, the unique responsible for the publication. Object of the analysis were therefore those class A journals that present indications on the evaluation procedures.

Results and Discussion: The first research study on books should start from the assumption that the solution to address the first survey on books should be found in a mixed approach including primarily the result of a qualitative assessment, integrated by a quantitative assessment. The core of the questionnaire concerned the opinion of respondents on a number of indicators to support the evaluation of the quality of legal monographs. Respondents have been asked to indicate the usefulness of some specific indicators (15) so to define the quality, the impact on scientific community and the societal impact of a monograph. Four possible rating (high, fair,low, none) have been proposed to respondents. On the basis of respondents’ answers a ranking of indicators has been obtained. The Fig. 1 shows the ranking of the indicators with regard to each quality dimension. Furthermore, if we refer to the first five positions – summarized in Table 1 – it emerges that for the quality and the impact on scientific community, there is consensus on indicators considered as more reliable. On the other hand, it is not surprising that, as regards the societal impact,different indicators are ranked in the fourth and fifth position. However, this ranking should be read as an overall skeptical view on the concept of societal impact referred to legal monographs, which is very difficult to define and also very hard to assess (as clearly evidenced in the comments to the question related to such dimension of quality). In general terms and with regard to the most commonly used indicator in STEM, neither indexing in existing citation databases or specifically created for the legal domain nor the number of citations received are considered robust indicators. In conclusion, from the analysis of the survey and literature a number of factors have been emerged. In the field of law, monographs are definitely works with greater scope and they are used as a paramount criterion for promotion decisions by universities. The central role (if not exclusive) of the peer review is undeniable. However, measures should be taken to avoid the risk of subjectivity of judgment. Indicators are certainly useful to orient the evaluation, but cannot be strictly applied as the result of mere mathematical and statistical analysis, easily exposed to erroneous and arbitrary use. There is an implicit availability of publisher ranking but no consensus to the formalization of such ranking. Nevertheless, the weight of the container(editorial series, publishers) should not be emphasized, rather the debate should focus on the transparency of procedures in the selection and review of research outputs.The use of different languages and the value of international cooperation must not be held as an indicator of quality, but must hold a very important place in certain areas of law. On the other hand, it should be noted that in legal research matters the exasperation of this trend could lead to paradoxical and pernicious effects.The societal impact of legal monographs remains a very hard issue. According to the Italian respondents, there is no real societal impact of legal monographs. However, some monographs may indeed influence judges or lawmakers’ opinion, mainly by virtue of author’s prestige and their argumentative capacity.On the basis of the second survey on peer review process of legal periodicals, the final results show that 10.2 % of the journals taken into account do not refer to the modalities of the evaluation activity and, nevertheless, to the criteria for the revision of the contributions. 23 journals out of 96 give importance to a preliminary phase which is preparatory to the actual evaluation activity by one or more referees. This phase consists in an initial selection of the papers in which the Director, assisted by the Editorial Board, checks whether the content of the contribution reflects the disciplinary orientation and scientific approach of the journal and whether the paper meets the formal characteristics required for the publication: for example if it respects a specific electronic format and a predetermined length. After being accepted the article can follow the next step of the proper evaluation process, otherwise the contribution is sent back to the author with the indications of the reasons why it has been rejected or with the suggestions of a series of changes to be made in order for the paper to be accepted.Even where the evaluation process is provided, 37.5 % of legal journals do not specify whether the evaluation process is up to internal or external referees; 22. 9% of the journals do not show any indication of the number of referees; 12.5 % of the journals do not provide any information about the anonymity or not of the referees; finally, 37.5 % of the journals, while mentioning the anonymity of the reviewers, do not specify the type (single blind, double blind peer review or open peer review). Only 10 journals (10.4 %) expressly provide the existence and conservation of the evaluation report of the contribution to the management committee, the Editorial staff or the publisher. The reporting systems, while having strategic importance both in relation to the traceability of the refereeing process, and with regard to the responsibility of the referees themselves, are not widely used in the Italian scenario.Furthermore, only 16.6 % of the journals refer to qualitative evaluation criteria such as relevance of the subject, correctness of the methodological approach, adequacy of the essential bibliography, order and clarity, originality of the contribution, practical utility of the ideas expressed, adequacy of the documentation.

Conclusions: The described Italian surveys on legal monographs and periodicals provide a framework which empirically confirms the state of the Italian debate on the evaluation of scientific legal outputs. Italian legal scholars evaluate the quality of their scientific production using qualitative methods rather than bibliometric. Under the influence of the exact sciences,they are now encouraged to follow the evaluation model used for such sciences. However, this process is not obvious. It is possible to affirm, based upon the answers to the two surveys and the Italian and foreign literature, that the assessment of the quality of research in legal sciences must take into account a number of important factors.Quality indicators should not be imposed upon legal scholars from a top down perspective,Moreover it is necessary to think of a quality assessment of the results more transparent and comparable at the international level. In such a direction harmonisation of legal research assessment exercises at European level is desirable and legal scholars should maintain a leading role in order to avoid negative effects (restrictions on academic freedom, approval …) and to avoid that the market imposes restrictions on academic freedom under the guise of managing the quality of research results.This approach is confirmed by the work of some foreign legal scholars (4,5) who have been working for years in the field of legal sciences methodology. A European approach to harmonize the evaluation process of legal science could be perhaps the only one which can save countries from academic habits that are stratified in the past, and which gave rise to questionable and degenerative phenomena, especially in the areas of recruitment. The establishment of harmonized standards or quality indicators generally accepted is a challenge to be conquered despite the differences among national evaluation methods, different publishing legal cultures and the many academic traditions.