The measurement and evaluation of universities’ third mission activities in Italy.

Sandra Romagnosi

Over the past 30 year, universities have been engaged in the processes of exploitation of research and technology transfer, especially trough patenting and licensing and academic entrepreneurship. Commercialization is considered a way for researchers to contribute to economy and to generate academic impact on society (Perkmann et al., 2103). Nonetheless, in the recent past universities’ roles and impact on society and economy is not limited to commercialization. Academics are engaged in a variety of activities, often spontaneous and not always managed at the institutional level (Perkmann et al., 2105). So, the technology transfer is just a part of a wide process of knowledge exchange, also informal, with public or private partners (Abreu et al, 2009).
The social growing demand for accountability in the use of public funding also contributes to the openness of Universities. In many countries procedures for assessing the impact of research as well as its transfer to society are in place. While studies the scientific impact of research are several, there is less evidence on the impact that universities have on society and in innovative processes (Donovan, 2008).
In Italy, since 2012 the Italian National Agency for the Evaluation of Universities and Research Institutes (ANVUR) has based the evaluation of the third mission of universities and research institutes on a holistic approach, defining it as the openness of the university towards the socio-economic context through the valorization and transfer of knowledge.
In this paper we illustrate the development of an evaluation method and associated measurement tools conducted by ANVUR within the framework of the two Italian research assessment exercises, VQR 2004-2010 and 2011-2014.
It has been a gradual process based on continuous consultation of the main actors and the starting point has been the first round of VQR (VQR 2004-2010) where the evaluation of third mission was based on a set of indicators referred not only to technology transfer but also to management of cultural goods. Those indicators were designed to contribute with a specific weight to the construction of the university unique score measuring third mission activities.
Even though very informative on the state of art within Italian universities, that experience has discouraged the production of synthetic scores, in order to avoid a risk of major methodological and substantive mistakes and distortion in behaviors of universities and research institutes. As a matter of fact, third mission evaluation has not entered into the performance-based formula funding for universities.
The Agency’s position towards third mission evaluation was to implement a gradual process to ensure shared definitions, certified data and reliable indicators. With this aim, in preparation of the VQR 2011-2014, a “maturity model” was elaborated. The assessment method chosen was the informed peer review approach, in which indicators should be used to inform the evaluation of the reviewers. According to this model, experts in each of the areas of third mission should adopt the metrics considered most appropriate given the advancement of the international methodological debate and the quality of indicators.
Moreover, a further work has been done on the demarcation of the perimeter of the third mission concept and the creation of a map of all the activities involved in the processes of interaction between academic institutions and socio-economic context. In a dedicated Manual (ANVUR, 2015), for each of the activities comprised a set of criteria, indicators and evaluation questions has been identified and the evaluation methodology has been defined. In parallel, ANVUR has set up an information system together with the Italian Ministry of Education, University and Research (MIUR), to collect standardized and comparable data on all the universities involved in the evaluation.
In the second VQR, the evaluation of third mission has been kept separated from research assessment indicators and conducted with different methodology and quali-quantitative criteria. Again, the results of the evaluation have not entered into the performance-based formula funding for universities. Nevertheless, evaluated institutions are asking for forms of incentives and support to better enhance their activities on this side.