EPOMM e-update May 2014
  Language:   en | de | fr | it
ECOMM website allinx feedback subscribe unsubscribe fullscreen news archive
tweet share on facebook

Dear reader,

It has been said many times before in our e-updates and at the ECOMM conferences: evaluation is of vital importance. With evaluation, you know the status of your project, you can make informed decisions to steer your project, you can thus ensure successful implementation and finally demonstrate the merits of your project to ensure funding for future projects. However, good evaluation in mobility management is rare, standards vary widely and comparison between projects is most often impossible. In this e-update, we take a closer look at existing guidance materials, at tools that are currently under development and we report on EPOMMs' standard evaluation tool MaxEva.

This e-update was produced in collaboration with NISTO.

 

Evaluation: tips and tricks


At the start, it is important to define clear project goals and objectives against which the output and outcome of the project can be evaluated. Sound evaluation should cover the implementation (monitoring of progress while the project is running) as well as the post-project phase (do the outcomes stay when the project is finished?).

The following lists things to take into account when evaluating mobility measures, projects or plans:

  • Involve all relevant stakeholders preferably in all stages of the evaluation.
  • Plan the evaluation process and foresee the necessary budget for data collection and evaluation in advance.
  • Collect baseline data before project implementation to compare progress to the do-nothing scenario.
  • When defining the project objectives and targets use SMART-criteria (Specific, Measurable, Assignable, Realistic and Time-related) and relevant indicators.
  • Make sure that you will be able to separate the effects of the project itself and external factors like fuel price increase, weather or other policy measures.
  • Also evaluate the implementation process, this helps to understand how the implemented measure achieved the measured outcomes.
  • Try to use an evaluation system that is transferable to evaluate other projects.
  • Carefully prepare data collection in order to prevent failures in the data interpretation.
  • The presentation of the evaluation results should answer the pre-defined goals and objectives – and it should be understandable and relevant.
 

NISTO




The EU-project NISTO (2013-2015) – New Integrated Smart Transport Options – is a transnational cooperation project between five European countries (Belgium, France, Germany, Netherlands, United Kingdom). The NISTO consortium is developing an integrated evaluation framework for smart mobility, which will be tested by monitoring and evaluating five demonstration projects across the partner countries. Their approach will focus on five principles of smart mobility: mobility, economy, environmental quality, safety and customer satisfaction. In the end, this will result in a user-friendly online toolkit, which will help improve the supervision of smart mobility projects by planners and public authorities.

The NISTO partners are always looking for further relevant projects (contact NISTO). This project has received European Regional Development Funding (ERDF) through Interreg IVB. It is supported by the European Commission.

 

NISTO stakeholder survey:The need for more knowledge and expertise


NISTO made an overview of existing evaluation methodologies (Evaluation of Approaches and Recommendations) and conducted a survey among stakeholders involved in transport planning and evaluation from the five partner countries. The results were further discussed in several workshops in which 124 stakeholders participated. Main outcomes:

  • Social cost-benefit analysis and multi-criteria analysis are the best known and most commonly used evaluation methodologies.
  • The NISTO evaluation framework should combine the advantages of those methodologies.
  • Changing the modal split, accessibility, environmental quality, efficiency of transport and safety are the five preferred evaluation criteria.
  • Evaluating mobility projects is not common.
  • Stakeholders are not experienced enough with evaluation methods to choose the right method for the right project - there is need for more information about evaluation tools.
  • There is demand for a flexible evaluation tool that is adjustable to the complexity of the project.

Read more in the stakeholder survey report. As some questions still remain open, NISTO will organise two further stakeholder workshops in each of the regions in order to continuously involve the stakeholders in the development of the toolkit.

 

MaxEva and MaxSumo


Since its development in the EU project MAX, EPOMM has promoted and further developed the MaxSumo evaluation framework. This tool helps to design an evaluation that takes into account the whole implementation process: it assesses direct project outputs (such as number of households reached); awareness, attitudes and behaviour changes in the target groups; as well as the final impacts on the transport system and the environment.

The MaxSumo approach is mirrored in the MaxEva tool. MaxEva can be used as evaluation tool but also as benchmarking tool, as MaxEva is also a database that contains the evaluation results of currently over 160 mobility management projects from all over Europe. MaxEva offers very practical options:

  • Predefined default values per mode.
  • Fleet emission data of several countries.
  • Taking into account external factors – by personal assessment or through evaluation the evaluation methodologies “comparison group” and “control group”.
  • Automatic calculation of results.
  • See also “Why use MaxEva”.

European projects such as the Traffic Snake Game Network, SEGMENT and Push&Pull have used or are planning to use MaxEva and MaxSumo to evaluate and report their project results. EPOMM is continuously maintaining and upgrading MaxEva.

 

Other Evaluation tools and guidelines


There are other publications that can help you to set up a good evaluation strategy for your mobility projects.

Within the CIVITAS initiative, cities that introduce sustainable urban transport policy measures are required to carry out an evaluation of the projects. Their objectives are evaluated with the CIVITAS evaluation framework, which is based on monitoring 31 core indicators in 5 impact areas (Economy, Energy, Environment, Society, Transport). As the first CIVITAS projects started 10 years ago, there is a great potential to examine the long-term effects of mobility measures. For this purpose the CIVITAS Initiative is working on a “How-to-guide on long-term evaluation”.

In London, all travel plans have to be monitored with either TRICS or iTRACE. Obviously, this tactic generates a sound collection of intercomparable evaluation data.

Similarly, the Swedish Road administration made it obligatory to use an adapted MaxSumo method for for projects to get mobility management funding.

Other recommended guidance materials include:

 

Keeping scores


Using standard evaluation tools allows for comparison and benchmarking. EPOMM’s modal split database TEMS allows practitioners and cities to compare modal split data between 366 European cities. The tool makes it also easy to compare similar cities (of similar size, from same country, with similar modal split). Unfortunately, a lot of different methodologies are used to generate these data, making comparison indicative rather than scientific. With TEMS, EPOMM aims to promote efforts and discussions to make modal split data and surveys more standardised. EPOMM hopes cities will use TEMS as benchmarking tool for their development and EPOMM aims to foster comparison and a positive competition between cities for more sustainability.

The harmonisation problems are researched in the Study on Harmonised Collection of European Data and Statistics in the Field of Urban Mobility. On national level the SHANTI project looked at the harmonisation of transport data.

The European Commission recently published the first EU Transport Scoreboard. It compares Member State performance in 22 transport-related categories and highlights for most of these categories the five top and bottom performers. The Netherlands and Germany top the scoreboard with high scores in 11 categories, followed by Sweden, the UK and Denmark.

 

Upcoming events

For more events, please visit the EPOMM Calendar.
ECOMM website allinx feedback subscribe unsubscribe fullscreen news archive