CORE QUALITY CRITERIA:  PROGRESS AND SUSTAINABILITY

Main aspects important for quality assurance in the case of progress and sustainability. The partnership / network meets its mission and progresses towards their Vision through planning and achieving a balanced set of results that meet both the short and long term needs of their stakeholders and, where relevant, exceed them.

QUALITY INDICATORS
(CORE AND ADDITIONAL/DESCRIPTORS)

What might indicate quality in the partnership?

EVIDENCE TO SUPPORT INDICATORS

What evidence is available to support the inclusion of the indicators?

EXAMPLES OF GOOD PRACTICE

Cases from ROMANIA

The capacity to continuously understand results, reasons; and the capacity to use this understanding to influence ongoing planning

Definition:

Monitoring is about checking - checking whether inputs match outputs,whether income balances expenditure, whether actual activity matches planned activity. It is also about recording the gaps between them. Monitoring is not the same as evaluation because it is descriptive rather than interpretive and is not intrinsically directed toward learning but the two are often confused. However, evaluation is almost impossible if there is no monitoring system in place.

Definition:

There are probably as many definitions of evaluation are there are books written about it. However, the shared understanding is that

• evaluation is purposeful, it is a means to an end not an end in itself,
• evaluation of things, which have happened, helps people make decisions about the future,
• it is based on asking specific questions about a project and finding the answers,
• it is an investigative process, evaluation is systematic and scientific,
• it involves collecting evidence,making comparisons, measuring things against criteria, 
• evaluation means that someone, ultimately, has to make judgements about the value or worth of something so its outputs must be interpretive not simply descriptive.

Find below the handbook "Evaluate Europe", which has been developed by the Capitalization & Evaluation Research Network (CERN) in order to support project managers at the development of evaluation schemes for a variety of purposes.

Evaluate_Europe_Volume1.pdf  

There is a formal process agreed around evaluation, progress soft and hard indicators are determined and agreed; evaluation and review addresses product (outcomes) as well as process (lessons),

Monitoring mechanisms and evaluation procedures are in place and scheduled, and resources are allocated both, finance and human responsibility,

Clarity of monitoring, evaluation and review methods and instruments

All partner organisations have to fill-in a questionnaire on evaluation which is designed for all partners and possibly additionally used for primary stakeholders to collect the opinions and results and to benachmark them in a way

Research and planning skills are available, in order to effectively conduct monitoring, evaluation and review

Documentation of formal process,  signed by all stakeholders of the network

Documentation of monitoring instruments, implementation schemes, and responsibilities

Number of inquiries made and requests for support to effectively use monitoring, evaluation and review methods and instruments

PLAOIS Suceava County (Action for Occupation and Social Inclusion)

http://www.prefecturasuceava.ro/finantare/PLAOIS_final.pdf

A comprehensive need analysis, based on quantitative and qualitative data is not only the foundation for planning activities, monitoring and evaluation the progress and impact but also a powerful tool for fostering mutual understanding of expected results and reasons for specific interventions. Therefore initiatives such as PLAOIS in Suceava County demonstrates that an in-depth analysis of baseline indicators relevant for the areas of intervention included in the agenda of the learning region partnership is very important.

Monitoring and evaluation informs strategic direction and both policy and practice

Definition:

Capitalisation means building on the achievements of an activity, project (or programme) and using the results in future activities.

Key stakeholders ensure personnel time committed to review

There is a clear understanding of the capitalization process, allowing policymakers and practitioners to effectively capitalize on evaluation results

Results of evaluation and review are fed into planning of LR actions and approaches

Hours / days spent by stakeholders for review of monitoring and evaluation results within a certain period

Easy to use documentation of capitalization process, agreed by all stakeholders

Number of inquiries, concerned with capitalization issues

Changes applied during planning stage of actions and approaches 

Evaluation and review (quality) are prioritised and internalized – seen as a core activity and not just an add-on

Evaluation culture is established and there is a common understanding of evaluation as a stimulus for development

It is ensured that evaluation and review are part of day-to-day activities of the network

Self-evaluation is promoted within the network

Evaluation and review are acknowledged a prior task in strategic documents

Evaluation and review are acknowledged a prior task in operational plannings

Number of inquiries, concerned with difficulties in understanding the task

Number of self-evaluations implemented

CRBI – Institutional Development of Regional Consortium Bucharest Ilfov

http://media.unibuc.ro/comunicate-presa/conferinta-de-incheiere-a-proiectului-cresterea-capacitatii-institutionale-a-consortiului-regional-de-dezvoltare-bucuresti-ilfov-crbi

Self-evaluation of the partnership achieved results leads to an early identification of institutional development needs. The project CRBI – Institutional Development of Regional Consortium Bucharest Ilfov funded by Human Resources Development Operational Program indicated that specific training and exchange of experience could be effective if the self-evaluation is comprehensive and includes all actors of the learning region partnership. Moreover, the training providers were selected with a close look at the areas of expertise corresponding to the needs of the learning region activities. 

Means of measuring must be appropriate in context of the learning region

There is a macro as well as micro picture (product as well as process)

It is ensured that outcomes also are determined in longer term, since effective LR strategies may take generations for positive results – not immediate

Possible outcomes include effects on broader community (not just the individual) and positive benefits are identified in different contexts

Achievements in the product / process / professional practice-field / policy level

Experience of measuring good practice and contribution to public good

Measuring sticks are defined for both levels, the Learning Region network and its sub-networks, and are laid out in strategic planning documents

Long-term outcomes are defined, and indicators are mapped onto evaluation schemes

Outcomes are defined for the broader community, and indicators are mapped onto evaluation schemes

Outcomes and indicators are defined on different levels and mapped onto evaluation schemes

Good practice of measuring is documented and acknowledged by the network actors, that is stakeholders are feeling comfortable with measure sticks and perceive transformation (soft indicators)

Review of broader regional development strategies to determine influence of LR strategy 

CRSE- Institution Building of Regional Agency South East

http://www.crse.ro/doc/Seminarul32.pdf

Under the coordination of regional developing agencies, these strategic planning documents offer a very useful tool for measuring the progress of the learning region. A specific initiative of the Regional Agency South East (www.crse.ro) was to develop a special training module on Increasing the capacity of operating with complex evidences introducing the key concepts in the area of evidence-based policy planning and analysis.

Results and findings of monitoring, evaluation and review must be able to be widely understood

Evaluation and review results are made widely available

Results are stakeholder-specific

Number of network actors and target groups reached

Results of document analysis, such as on style of publications, different modes of communication targeted to the main user groups

Positive feedback from stakeholders, and inquiries indicating a gap in understanding

See criteria above.

Flexibility of partners to share information (not defensive)

Learning Region actors see the value-addedness of sharing insights and giving information

Number of documents shared

Quantity and quality of participation in forums, inscriptions to mailing lists, and use of means for knowledge sharing, such as databases

Results from stakeholders audits into the effectivity of knowledge sharing

Smaller networks created within the hub

Mutual agreement governing confidentiality and ownership of information exchanged

PREZENT! Creating incentives for participation to continuing training of employees at risk on the labour market – information, awareness raising and access to quality lifelong learning (POSDRU/62/2.3/S/22025).

www.invatapentrutine.ro

Information Exchange Hubs

The method of exchanging and sharing information is simple and powerful: due to the large dimensions of the partnerships, it was decided to create several information exchange hubs when four-five organisations are gathering together at a given time. At a second stage,  three of the organization attends the meeting of a different hub and are informed of new developments. In the third stage, the partners return to the hub and share the information with other organisations. This is particularly important when specific agreements should be made in a given time and only some partners are available. Also by using this method all partners have enough opportunities to offer their input and participate to the decision-making process.

Outcomes and impact are regularly checked, demonstrated and communicated to all members of the network and the community at large

There is mechanisms in place, which allow for periodic analysis, synthesis, demonstration and dissemination of outcomes and impact of LR initiatives based on monitoring and evaluation findings

Frequency and number of periodic reports

Communication campaign of PREZENT! Creating incentives for participation to continuing training of employees at risk on the labour market – information, awareness raising and access to quality lifelong learning (POSDRU/62/2.3/S/22025).

www.invatapentrutine.ro

Both face to face and online meetings with partners were planned, as well as public information campaigns for the beneficiaries of the project. A key success factor was exactly to combine internal and external communication channels so to use the occasions of developing activities for a specific target group to demonstrate and communicate within the network the progress of activities. Often these two areas are considered separate but, for example, in the case of PREZENT initiative, public caravans across all major cities in a region were used also as an opportunity to meet organisations form the network form that geographical area.

CRBI – Institutional Development of Regional Consortium Bucharest Ilfov

The communication channels among partners included publication of regular reports stating the progress of the project basesd on monitoring results. All these materials had a feed-back form attached so each organisation could indicate areas of interest or to raise particular questions in relation with a theme. At the end of the project, the partnership published a detailed report summing up all the main messages for the network and form the community at large.

Unintended as well as planned outcomes are documented and shared

Different types of outcomes are documented in database 

Number of database records according to types of outcomes, intended and unintended as well

See the core quality criteria related to the sharing of information