 |
|
|
 |
 |
 |
 |
 |
 |
 |
 process in which monitoring and evaluation are needed.&p[images][0]=http://epomm.eu/newsletter/v2/content/2016/0316/img/fb.jpg&p[url]=http%3A%2F%2Fepomm.eu%2Fnewsletter%2Fv2%2Feupdate.php%3Fnl%3D0316%26lan%3Den "share on facebook")
Dear reader, Regular monitoring, review and reporting is one of the seven principles for SUMP set by the European Commission in the Urban Mobility Package. In this e-update we will explore the phases of the sustainable urban mobility planning (SUMP) process in which monitoring and evaluation are needed. We will show examples from the ENDURANCE cities and include useful tools and documents, especially from the NISTO project that recently came to a close. | |
The final months of ENDURANCE |
|
 ENDURANCE experience exchange session in Barcelona | It is only three months until the end of the ENDURANCE project. It now has 296 activated cities online! This proves that the multi-angle support that we provide to the cities – through network meetings, trainings & workshops, policy transfers and on demand expert-delivery services is a success. This is largely thanks to our 25 national focal points (NFPs) who tailored that support to the local and national needs. The information of each ENDURANCE city can now also be downloaded as a PDF file. Our Portuguese NFP is the ‘uncrowned’ number 1 in SUMP awareness raising: 28 cities have started to or have improved their SUMP. Since 2015 there is a new dynamic for the development of PMT/ SUMP as a result of the requirement included in the Partnership Agreement between Portugal and the EC 2014-2020. It conditions the financing of urban mobility measures and actions to the prior performance of PMT/ SUMP. Short two page report here. On the ENDURANCE agenda:
|
What do the SUMP guidelines say about monitoring and assessment? |
|
 M&E components in the SUMP cycle, click to enlarge | Monitoring, assessment and evaluation reoccur in many steps of the SUMP planning cycle, as described in the European SUMP Guidelines (available in 9 languages):
|
Monitoring, assessment, evaluation: a wide range of different activities |
|
 Heat Map of cycle trips in Bologna during the European Cycling Challenge 2015. Photo courtesy of ECC.   Source: Active Access, www.eltis.org  The city of Maia, Portugal | The terms monitoring and evaluation are often mentioned in one breath and many people do not realise the difference between these two terms. We will provide our interpretation, also of other evaluation-related terms and some examples of projects and tools to illustrate them. Monitoring = Regular measurement of indicators to track progress towards goals. A systematic and more or less continuous data collection is the basis. The timely interpretation of the data – the evaluation – shows whether you’re on track or whether you have some undesirable developments. The recently started Horizon 2020 project CREATE aims to describe the policy evolution cycle over a period of at least four decades for several of its cities to also show how they reached “peak car”. Such a systematic collection of data is rare. Most ENDURANCE cities provided their modal split, some of them have longer timelines of these recordings (e.g. Vienna and Graz). It is all available on TEMS, the EPOMM modal split tool. An interesting way to collect data for more specific aspects of the mobility situation, is through the use of smartphone apps that track user behaviour. Examples are RouteCoach in Leuven, Belgium, the B-TRACK-B project and the Bicycle Counting App in Flanders and The Netherlands. The Cycling365 app used during the European Cycling Challenge also gathers data on cycling for the local transport planners. The raw data coming out of an app are not easily interpreted. The NISTO project developed some “Guidelines to convert sensor data from smartphones (e.g. GPS) into indicators that can be used in the evaluation”. Evaluation = Systematic determination of measure’s merit and significance during and after implementation. While the difference between monitoring data and evaluating data may be quite obvious, it is more difficult to distinguish the two when it is about process or more general terms like outcome or output. A good overview over the differences between monitoring and evaluation are provided in this Genuine Evaluation blogpost. Evaluation is part of a continuing management process consisting of planning, implementation, and evaluation; ideally each substituting the other in a continuous and simultaneous cycle until successful completion of the measure. Evaluation tells you how much and why you deviate from your goals and what you can learn from it. Related term:
Assessment = Judgement of a system, process or person’s performance, strengths and weaknesses. The SUMP process usually begins with a self-assessment of current planning practices. ENDURANCE developed a self-assessment questionnaire for its cities to help them identify their main needs for support. The questionnaire was also filled in by the ENDURANCE national focal points, to provide a peer review of the city’s SUMP status by an objective outsider. Related terms:
Appraisal = judging the value and importance of a measure or project – in the context of monitoring and evaluation this judgement is made before implementation, in order to be able to make an informed choice between different potential measures. This is also called ex-ante evaluation. (see also our e-update on smart packages of measures and the Konsult database). When the goals of a SUMP have been determined, the city has to select a suitable set of measures. For this, it needs to “appraise” the measures, which means to find out which measures have the most potential to realise the goals. An example: the ENDURANCE municipality of Maia, Portugal, had a municipal Technical Working Group and a Monitoring Commission composed of external entities to carry out monitoring and evaluation. For the ex-ante evaluation they developed different scenarios for the evolution of the mobility system through the technique of backcasting. Starting from the goals to be achieved in the future in terms of modal split and environmental impacts, the set of measures was adjusted so that they would lead to the achievement of these goals. A multi-criteria analysis was performed to choose between scenarios. Bremen, Germany, winner of the European SUMP Award 2014 for monitoring in SUMP, had three rounds in its ex-ante evaluation: an analysis of strengths, weaknesses, opportunities and threats (SWOT), a scenario analysis of five ‘extreme choice’ scenarios, and a cost-benefit analysis in collaboration with an external expert. The best scenario turned out to be the promotion of active mobility. (Source: Eltis, see also the case studies of the other finalists Ghent and Dresden, and special jury prize winner Thessaloniki) Related terms:
|
A challenging task for cities and municipalities |
|
 The city of Brasov, Romania  Tallinn, Estonia | The NFPs of the ENDURANCE project have detected many challenges for local authorities related to assessment, monitoring and evaluation in SUMP. Cătălin Frangulea, the SUMP Coordinator of the ENDURANCE city Brașov in Romania for instance reports about lacking mobility planning and evaluation expertise within local authorities. Technical staff need to know exactly what to ask and expect from the consultants, and, even more importantly, these staff need to be able to make good use of the products – the monitoring and evaluation results – as delivered by such consultants. The data can be obtained with a little effort, but the capacity to process the data is lacking. Our Austrian NFP, the Austrian Energy Agency, observed that local transport and mobility plans rarely include monitoring and evaluation measures. And even then, it typically does not follow a systematic approach but is based on individual requirements. Therefore monitoring and evaluation was a central focus in the guidance and workshops offered by ENDURANCE in Austria. The city of Vienna is one good example of an ongoing monitoring process and key policy evaluation studies to underpin the city’s Urban Development Plan (STEP2025). Read more > Our NFP in Estonia says the costs of surveys can pose a barrier. ENDURANCE supported the development of a detailed travel survey methodology for the Tallinn and Tartu region within an expert group at the Estonian Ministry of Economy and Communications. However, a first public tender failed, because the costs of the survey were higher than anticipated. Eventually, a more narrow travel survey, including household travel diaries of 4000 Tallinn residents, was carried out November-December 2015 and will be made available as open data. Both Tallinn city and the national government co-operate well together to find funding for the surveys, for instance via Interreg. Norwegian research on the evaluation of transport infrastructure projects shows that substantial resources are devoted to improving appraisal methods (cost estimation techniques, transport models, consumers’ valuation of the effects, etc.) and that most countries have guidelines for Cost-Benefit Analysis. Ironically, the amount of resources used for ex ante evaluation is rarely matched by the resources used for ex post evaluations.” (Ex post evaluation of transport projects – experiences from Norway) A further important part of appraisal is the different (conflicting) opinions of stakeholders. Citizen and stakeholder involvement is a central element of the above-mentioned strategy of SUMP award winner Bremen. It is also an essential part of the NISTO toolkit that helps users to identify synergies and disagreement between different stakeholder groups. “Involving stakeholder groups in the evaluation process and using monitoring data to inform the public are of growing importance but seen as difficult and potentially prone to pressure from interest groups,” writes the CH4LLENGE consortium in their paper Why is monitoring and evaluation a challenge in sustainable urban mobility planning? |
Conclusion and recommendations |
|
 Source: www.eltis.org | Monitoring and evaluation is important: for accountability (did we get value for money?), better decision making, optimisation along the way, benchmarking, proving success – and sometimes – showing the reasons for failure. Monitoring and evaluation should be an integral part of project management. As monitoring and evaluation is often neglected, we provide some tools and information we recommend to use.
|
Upcoming events |
|
For more events, please visit the EPOMM Calendar. | |
|
|
|
|
|
|
|