Case study 1

1.1 Why use ROMA?

The DFID–ESRC Growth Research Programme (DEGRP) funds world-class scientific research on inclusive growth and related issues in low-income countries. ODI leads the Evidence and Policy Group (EPG) which supports DEGRP and its diverse projects to maximise the profile, uptake and impact of the programme’s research. All the projects have to develop an impact plan as part of their bid preparation and it was with this in mind that ODI decided to write a short guide on impact - a synthesised version of ROMA specifically targeted towards academic researchers to help the project teams.

The DEGRP programme funds a wide range of research on economic growth in low-income countries covering four themes: finance, innovation, agriculture, and the relationship between China and Africa. Many of the Principal Investigators are renowned researchers with good links to national and international policy-makers. The impact guide needed to be relevant to a wide variety of topics and to researchers with a good deal of experience.

1.2 How was ROMA applied?

‘Achieving Policy Impact’ was published in 2013. Based on chapter 2 of ROMA – developing a strategy – the guide explores several key areas: understanding the different types of impact that academic research could achieve; mapping stakeholders; considering what change is likely; and knowledge brokering. Academic researchers are more likely to be familiar with the concept of communicating their research than with the concept of achieving impact—though the emphasis put on impact by the UK’s Research Excellence Framework in 2014 is gradually changing this. The guide started with the assumption that achieving impact depends on having a good communication strategy.  Good communication depends on understanding what sort of brokering roles the project could fulfil, which in turn depends on developing a realistic theory of change. A realist theory of change depends on having a comprehensive stakeholder map. The guide reverses this logic to set out four steps: (1) an AIIM matrix, (2) an outcome mapping-based theory of change, (3) brokering roles, and (4) research communications.

Because all projects had developed impact plans as part of their programme plan, the DEGRP impact guidance was not prescriptive. The guidance was intended to help projects refine their understanding of how impact could be achieved, and make their approaches more appropriate to the specific needs of their stakeholders.

The guidance has been used by projects researching very different issues such as irrigation in Tanzania, risk perceptions in Uganda, financial sector development and supply chain management in sub-Saharan Africa and improving productivity in Bangladesh’s garment sector. It has also been well received in a number of workshops, and to a wider audience of grant holders from the DFID-ESRC Poverty Alleviation Programme.  Projects are given the guidance as they begin, but are not expected to refine their pathways to impact until a year later, to give them time to get into the field and get to know their real operating constraints.

The guidance is freely available on the website to others outside of DEGRP and is the most downloaded publication to date on the site.

1.3 How was ROMA adapted to the project context?

ROMA principles have been used in different ways. A review of the revised pathways to impact showed that in the main the guidance had been used to make projects’ anticipated achievements a good deal more realistic than what was set out in their initial project plans. One project that had drawn up a very detailed impact plan used the guidance to develop a theory of change to do some detailed differentiation of the needs of individual stakeholders – from procurement agencies needing more reliable and more suitable supplies to policy-makers wanting to find more scope for innovation and better supply chains. Another used the theory of change to nail down some very specific changes they would expect, like and love to see among both public policy-makers and private sector collaborators. Gratifyingly, by the end of the project the latter project had achieved its ‘like to see’ indicator and developed concrete ideas about how to achieve the changes the project team would ‘love to see’.

The guidance had a major effect on one project, causing it to completely rethink its approach to impact. The PI was initially openly sceptical about the need for academic research to consider issues of impact but having read the guidance, he recognised that the project did not understand enough about the agricultural policy-making process in Uganda to really know whether the recommendations they were developing from their research were locally and nationally relevant. Once initial research recommendations were finalised the team decided to embark on an extensive and highly structured stakeholder engagement exercise to turn them into policy recommendations. There were four stages to this: (1) small group discussions with the project’s direct collaborators, (2) larger workshops with district extensionists and local government officials, and (3) consultations with 90 national policy-makers, private sector representatives and development partners. At each stage the project’s recommendations were adapted and refined—the final stage (4) was a large multi-stakeholder workshop in Kampala which used group work to look at the wording of specific policy recommendations and discuss what might be needed for them to be implemented. The project team became so enthused by what they had learned from this ROMA-based approach that they decided to hire ‘policy brokers’ to help them ensure that the messages from the research continue to be heard within agricultural policy processes in Uganda.

Key to the success of the guidance has been that it is an approach rather than a template, and that the ROMA tools can be used in any order. The guidance placed less emphasis on chapter 1 of ROMA (understanding the context) or chapter 3 (developing a monitoring strategy) as the context for the research was set out in detail in the proposal documents and there is no requirement for projects to monitor their impact in detail. However, the DEGRP Evidence and Policy Group is working to develop a framework for monitoring programme-level impact for this very diverse portfolio of research.

1.4 What was achieved or learned by using ROMA?

The DEGRP example shows that ROMA is highly adaptable, being used by projects in very varied settings indeed. The fact that it is not necessary to know and use ROMA systematically from start to finish means that a small number of ROMA’s overall toolkit can be selected and presented in a non-prescriptive way.  This helps ensure that the overall approach remains relevant to the needs and concerns of academic researchers who previously have not had to think about impact in this way and who might find the full ROMA toolkit overwhelming.