Evaluating and scoring research results – a case of the Slovenian Research Agency

Stojan Pečlin (Slovenian Research Agency, SI)

Abstract: This paper discusses the role of quantitative indicators in evaluation procedures, partly as a case study of Slovenia. Slovenia establishedacentralised system of researchers’ bibliographies, introduced in 1994 and became an obligatory part of the evaluation system in 1997. It is designed as a part of the COBISS (Co-operative Online Bibliographic System and Services) that connects libraries and other information institutions. Publications of all researchers in Slovenia are presented in a standardised manner, for every input (which is done by librarians) agreed typology of documents is used. COBISS is linked with Thomson Reuters Web of Science data and since last year also with SCOPUS, both mainly for citation data.

The introduction of COBISS first as a transparency tool increased the objectivity of the evaluation process and gave a significant momentum to the change of how Slovenian scientists presented their research results. It caused a shift toward publishing in international journals with high impact and reputation. Since the introduction of COBISS, the reputation of researchers is no longer based on abstract criteria but is grounded on solid quantitative data. The criteria for evaluating publications are available and anyone is easily able to determine the significance of publications according to pre-set criteria.

Journals indexed in bibliographic bases SCI and SSCI (and from last year also from SCOPUS) are evaluated according to their impact factor and divided in 4 quarters or quartiles. How many points one gets for his or her published article depends on the journal impact. Another important data base for evaluation procedures is Slovenian Current Research Information System (Sicris). Sicris is the information system on research in Slovenia that includes databases on the research organisations, research teams, researchers, research programmes and research projects. It was introduced in 1998. It is, therefore, an integral system which in a significant way supplements COBISS with more substantial information.

Quantitative, bibliometric and scientometric, indicators are used mainly in the stage of eligibility screening, deciding about possible research team leaders and applicants for funding, based on a minimum level of points. The minimum score is set as an entry condition. Those indicators were also used in determination of researchers who can enter automatically the second round of assessment of projects to decrease the burden of administration for researchers who already have a rich bibliographic record to prove their excellence. The basis of evaluation and assessment procedures is peer review system, but with minimizing conflict of interests. Although for searching peer reviewers also bibliometric indicators are used.

Integral system Sicris/COBISS is very useful also for bibliometricians in Slovenia as they can perform applicative analyses which in turn help SRA by its decisions. Research was done on the evaluation of different peer review systems, the influence of international co-authorship on papers impact, differences between WoS and Scopus.



Short Biography: Stojan Pečlin graduated from Faculty of Social Sciences, University of Ljubljana. He worked as an adviser at the Institute of Macroeconomic Analysis and Development and then at the Research Community of Slovenia. In the nineties of the last century he headed Department of Human Resources Development and afterwards Analysis Office at the Ministry of Science and Technology. In 2003 he obtained his master degree at the Faculty of Arts, University of Ljubljana in the research field of information science. Since 2004 he has worked in a new established Slovenian Research Agency. In the period 2010 – 2012 he was a representative of Slovenia in ESF Member Organisation Forum on Peer Review and was involved in preparation of European Peer Review Guide. In 2012 he obtained his PhD degree at the Faculty of Arts, University of Ljubljana with thesis “Scientific production of research groups in Slovenia and impact of their publications”.

As a researcher he has cooperated with research groups from Faculty of Arts and Medical Faculty, both University of Ljubljana and with research group from private non-profit organisation Biomedical Research and Innovative Society. In the period from 2009 to now he has been involved in the implementation of the following research projects: “Expert assessment and bibliometric methods: Two sides of the same coin?”, “Peer review, quantitative indicators and the use of IT tools for the evaluation of scientific activity” and “Bibliometric methods of evaluation research results and their application in medical science”. During this time he published 5 articles, four of them (as the first author in 3 articles) in journals with impact factor.

(Inter)national Evaluation Practices Metrics and Assessment
Location: Old Campus Date: September 19, 2014 Time: 10:00 am - 10:30 am Stojan Pečlin Stojan Pečlin