Phase 6 – Monitoring and evaluation

Keeping the track of RIS3 implementation and its result and reassessing the RIS3 with new evidence.

The monitoring and evaluation phase is related to watching the progress of the RIS3 implementation, tracing its effects and adapting the strategy to the changing context of the region. The process of forming Smart Specialisation strategies should be adaptable to new knowledge of the region and its surroundings as well as to the effectivity of RIS3 actions.

Monitoring activities yield answers to the question of are we doing the RIS3 implementation right? while evaluation gives insight to are we doing the right things?

Our 6 tools for the monitoring and evaluation phase provide concrete means to monitor the implementation of RIS3 and gathering data for monitoring and evaluation purposes.

It’s important to understand that there are no universal, or even European, standards on how to undertake monitoring and evaluation of the RIS3 implementation. The optimal way to organise them depends vastly on the region and its RIS3. Thus, monitoring and evaluation should always be tailored to the region in question.

Nonetheless, there are some common aspects of monitoring and evaluation that regions should assess.

Monitoring

Monitoring refers to monitoring the RIS3 implementation and its effects on the region both short and medium/long term. Thus, monitoring should yield knowledge on how the implementation is carried out and if it has succeeded in the way it was originally thought. Reflecting monitoring indicators to the baselines show whether the RIS3 implementation has had hoped results.

Who should be responsible for monitoring?

The RIS3 guide recommends that the monitoring should be carried out by the actors that are responsible for implementing the RIS3. Monitoring can also be a beneficial part of involving key stakeholders in the RIS3 process.

Monitoring relates closely to evaluation as monitoring mechanisms provide evaluators with empirical data about how the RIS3 process is advancing.

Did the RIS3 implementation actions yield outcomes that we wanted?

Did we get any unintended results?

Why things we did worked? Or why didn’t they?

Evaluation

The idea of evaluation is to assess if right things were done to achieve desired results, so it’s not only about the region following the implementation plan as outlined.

The foremost idea behind evaluation is to assess if the current RIS3 strategy is still feasible or if it should be changed somehow so that it would relate better to the regional situation, context or opportunities. This assessment should be done according to the new evidence provided by active monitoring.

The challenge is to maintain stable priorities and a stable specialisation strategy despite adapting to new opportunities and evidence.

So evaluation should refine the RIS3 strategy as new information comes onboard. However, it’s also necessary to think about how to improve the monitoring activities. Are there indicators that don’t really give useful information? Should there be some novel indicators?

Who should do the evaluation?

Evaluation should be done by individual experts along the policy-makers. The evaluation is more likely to be candid when those who implement RIS3 are not the ones making the evaluations.

Indicators

In order to determine how the region is doing in each of the priority areas, it’s important to plan clear indicators for each of them. It’s also necessary to determine baselines and target values for all of the indicators.

This way it’s easier to see how the region has advanced in each of the priority areas and whether the implementation is going according to plan. It’s also necessary to assess the indicators and targets in both short term and long term.

Types of indicators

There are several types of indicators that are recommended by the RIS3 Guide and Implementing Smart Specialisation Strategies Handbook. These include context -, output -, result – and implementation indicators, and indicators for structural change and specialisation.

Context indicators

Context indicators convey how the regional economy compares to other regions or nations. The domains of comparison should come from the overall RIS3 strategy and its pressure points. The Smart Specialisation Implementation handbook also recommends comparisons in overall research and innovation and the development of production systems.

Examples of context indicators:

Labour force participation rate
Indicators of innovation and R&D activities

Output indicators

Output indicators convey the progress of RIS3 implementation activities. More concretely, output indicators might also present the outputs of funded projects.

Examples of output indicators:

Number of filed patents
Number of new innovations adopted
New collaboration networks resulting from funded projects

Result indicators

Result indicators present the results of selected activities that should contribute to the strategic goals. Thus, they should indicate if the strategic goals were pursued with appropriate means.

These indicators vary greatly depending on the type of strategic goals and they should be articulated well when initially forming the strategic goals and means to achieve them.

Examples of result indicators:

Export performance
Private R&D expenditure
Energy consumption

Implementation indicators

Implementation indicators show how well the implementation process has advanced compared to the initial plans.

Examples of implementation indicators:

Number of funds allocated (regional, national, EU)
Number of accepted investments

Structural change and specialisation indicators

These indicators present the extent of structural change in the region related to the target of RIS3.

Examples of structural change and specialisation indicators:

Changes in business ownership structures
Specialisation in strategic priorities (e.g. local companies)
Changes in demographic dynamics of companies

Online S3 monitoring and evaluation tools

Our monitoring and evaluation tools help with creating RIS3 indicators and creating novel ways to gather monitoring data.

6.1 Monitoring

The Monitoring tool supports the development and adoption of monitoring mechanisms for the RIS3 implementation. This web application also allows to track and monitor the implementation process.

6.2 Definition of output and result indicators

The Definition of output and result indicators tool supports setting up output and result indicators for the RIS3 implementation. It also creates visualisations of the indicators and performs statistical analyses on them.

6.3 Balanced scorecard

The Balanced Scorecard (BSC) is a planning and management system that is widely used in a variety of areas in the private and public sector, with the aim of ensuring that the activities of an organisation are in line with its vision and strategy, improving internal and external communications, and monitoring organization performance against strategic goals.

6.4 End user satisfaction survey

The End user satisfaction survey enables policy-makers to understand how well-suited the strategy and the related activities are to the needs of RIS3 beneficiaries (firms, research institutes, universities, public sector and civil society organisations). This tool helps the collection of micro data directly from the beneficiaries and ‘end-users’ of public policies implemented through S3.

6.5 Social media analysis

This online tool offers simple indicators on the RIS3 process by using data coming from social media sites, such as Twitter or Google trends with regards the overall process, or also allow for specific searches with keywords of key policies or programmes implemented in the region.