Citizen science impact pathways for a positive contribution to public participation in science

Skarlatidou, Artemis; Haklay, Mordechai

doi:10.22323/2.20060202

1 Introduction

Of the different elements of the Science with and for Society efforts of the EU Horizon 2020 research programme, Citizen science is recognized as one of the Responsible Research and Innovation (RRI) methods. It is heralding a new era of public engagement with science, which aims at creating a shared responsibility between science, policy and society. Citizen science encompasses a wide range of activities from dedicating an hour a year to count birds in your garden, to carrying out biological experiments in Do-It-Yourself Biology. Participation in citizen science includes different levels of engagement [Haklay, 2013 ], different types of projects [Shirk et al., 2012 ] and different levels of technological use [Wiggins and Crowston, 2012 ]. We can look at this range of activities and consider what makes people participate in different activities, while also considering how we can assist them in identifying activities that suit their needs, abilities and interests, and fit within their care, work, and communal commitments. However, there are several unexplored questions such as: how do we go about understanding the role of different actors and the processes they go through in these journeys towards deeper engagement in science? How can we ensure that we have a grounded set of assumptions about the transformations that are required to make such journeys happen? Most importantly, how can we ensure that in such large-scale programmes the expected impacts are indeed achieved? Despite its several benefits, citizen science comes with some criticisms and limitations. One of them is that programme evaluation, in the wider citizen science context, is an area which received very little attention; there are only a few examples which discuss their methodological approaches to evaluation, together with lessons learned and reflections to eventually inform the future of the field.

In this paper we use the case of the EU-funded project Doing It Together Science (DITOs) to make the following contributions. First, we position citizen science alongside other modes of public engagement to explain how citizen science has the potential to enable upstream engagement and tackle research and innovation issues. By doing so we bring to the reader’s attention the necessity to better understand people’s journeys in their engagements with science. Using our novel concept for the ‘DITOs’ escalator framework’, we explain how activities can be designed to achieve wider and deeper public engagement with science, which is a common aim across many citizen science initiatives. Second, we discuss the need for evaluation mechanisms in citizen science and we present the theory and practical implementation of a top-level logic model. We use the DITOs logic model to provide insight into the design considerations in implementing the escalator framework, the programme’s assumptions and for demonstrating the impact pathways as an evaluation mechanism. By doing so, we hope to inspire others to use similar tools to monitor and evaluate citizen science initiatives, while sharing their lessons learned to subsequently improve our understanding of designing and running more effective citizen science programmes.

2 Engaging the public in science in the 21st century and the emergence of citizen science

Science has been through some profound changes in the last 50 years, in the way it is initiated and in terms of how it operates and is managed. First came the legacy of Bush’s [ 1945 ] ‘Science-The Endless Frontier’ formulation of science-society relationships until developments (e.g. controversial techno-scientific debates around topics such as nuclear weapons and the political changes that led to the commercialisation and industrialisation of scientific research) as well as incidents of scientific fraud that caused public suspicion and led to new formation of these relationships [Gregory and Lock, 2008 ; Jasanoff, 2003 ]. The post-war ‘social contract’ period [Jasanoff, 2003 ] is flooded with theories that attribute negative public attitudes to a disinterest and lack of knowledge around scientific issues and which therefore suggest that development of people’s understanding of scientific issues will improve public attitudes and science policies and lead to economic growth [Gregory and Lock, 2008 ; Royal Society, 1985 ]. Despite being popular at the time, the public understanding of science (PUS) movement proved to be problematic in the 1990s.

In the early 2000s a dialogic mode of cooperation and negotiation in science emerged, aiming to understand and integrate public needs and values in scientific practice. The main characteristic of the post-deficit era was the exercise of scientific citizenship through citizens’ juries, consensus conferences, public meetings, science festivals, etc. [Bauer, 2009 ; Gregory and Lock, 2008 ; Stilgoe, Lock and Wilsdon, 2014 ]. This led to a change from a dominant model of science communication to that of public engagement with science and, to a degree, in science policy. The same period saw a growing belief that public engagement could be the antidote to broader pathologies in policymaking, beyond science [Rowe and Frewer, 2005 ]. This model of engagement also came with its own critiques about issues of representativeness and inclusiveness, similar to wider debates on participatory democracy [Rowe and Frewer, 2005 ]; neglecting public input in the actual decision and policymaking processes (which can increase feelings of suspicion and distrust); restricting benefits and purpose to that of socialisation of innovation and technology acceptance [Elam and Bertilsson, 2003 ]; and the use of evaluation approaches to public deliberation which are based on constructs such as “literacy, attitudes, interests, and media attention” [Bauer, 2009 , p. 225].

In 2010 a new term emerged in European policy and it is currently implemented in national policies in the U.K., Netherlands and Norway, amongst other countries [Bauer, Bogner and Fuchs, 2016 ; Owen, Macnaghten and Stilgoe, 2012 ]. Responsible research and innovation (RRI) is described as a paradigm shift, a ‘new social contract’ that sets the basis for “a shared responsibility between science, policy and society, to ensure that science promotes socially beneficial action as well as freedom of thought” [European Commission, 2009 ]. In a European context, RRI reflects on the limitations of “managing ethically problematic areas of science and innovation” [Owen, Macnaghten and Stilgoe, 2012 , p. 752] and highlights the need for policy programmes which build on truly upstream dialogue and emphasise ethical and socially acceptable impacts. RRI pays special attention to the principles of inclusiveness in participation, responsibility in the (co)creation of innovation, anticipation that reflection brings to explore alternative pathways to innovation, and institutional reflexivity [Bauer, Bogner and Fuchs, 2016 ; Owen, Macnaghten and Stilgoe, 2012 ]. The main characteristic is that, while previous movements try to keep science independent from the socio-political and economic landscape, RRI sees science and society interacting in a way that they are simultaneously co-constructed [De Saille, 2015 ]. Fundamental to the RRI process are initiatives which enclose ideas, principles and the means for reshaping science in ways that make it more transparent and easily accessible for everyone. Although RRI is currently “one of the most visible, influential and disputed governance approaches” [Bauer, Bogner and Fuchs, 2016 , p. 56], it remains ill-defined in how it works in practice [Bauer, Bogner and Fuchs, 2016 ]. While the terminology of RRI has diminished within current high-level European Union discourse about science policy, with Open Science taking its place [Gerber et al., 2020 ; Shelley-Egan, Gjefsen and Nydal, 2020 ], the RRI discourse was central to the development of the logic model that we describe here.

One of the methodologies that attracted the attention of both researchers and science policymakers is citizen science, which is proposed as an RRI tool to enable upstream engagement and tackle research and innovation issues effectively [Bauer, Bogner and Fuchs, 2016 ]. The White Paper on citizen science in Europe [Socientize, 2014 ] makes an explicit reference to RRI and proposed citizen science as a method suitable for this purpose.

2.1 Citizen science: opportunities, limitations and the use of logic models in programme evaluation

Citizen science encompasses a collaboration and partnership between professional scientists and volunteers in scientific activities, which may take different forms, from data collection to comprehensive participation in problem identification, methodology design, analysis and dissemination of findings. Within this context, citizen science has the characteristics of upstream engagement which contributes to the creation of shared responsibility amongst scientists, policy actors, and the lay public.

The literature is flooded with claims about its potential benefits. Perhaps the most popular, with significant evidence to support it, is that citizen science enables the collection of data at previously unattainable scales [Cooper, Shirk and Zuckerberg, 2014 ]. Citizen science further helps with harnessing collective intelligence [Liu and Kobernus, 2017 ]. Ottinger [ 2010 , p. 244] argues that citizen science “has the potential to make environmental knowledge and policy more robust and democratic” and other scholars add that it may influence science so that it becomes more responsive to wider public concerns [Martin, 2006 ]. Moreover, citizen science may “rebuild and rekindle some of the public trust lost in institutional science” [Carr, 2004 , p. 847]. For the volunteers, participation can improve scientific literacy, knowledge and scientific-reasoning skills [Jordan et al., 2011 ] and change their attitudes to science and the environment [Brossard, Lewenstein and Bonney, 2005 ]. It may also improve a sense of community, the satisfaction of giving back to the world, joy and fulfilment [Cooper, Shirk and Zuckerberg, 2014 ]. Studies from the opposite spectrum emphasise concerns about citizen science: data and data quality issues and dealing with ‘data patches’ [Bonney et al., 2014 ]; privacy, security and ethical considerations in storing and sharing data [Liu and Kobernus, 2017 ]; cross-project facilitation and communication [Newman et al., 2012 ; Cooper, Shirk and Zuckerberg, 2014 ]; and the lack of systematic evaluations to mention a few.

There is indeed a growing recognition that most citizen science programmes lack systematic evaluations to demonstrate how impact is achieved (if at all) and within this context logic models have been suggested as a way to fill that gap [Reed et al., 2018 ]. In a ‘how-to-evaluate’ citizen science guide, Phillips et al. [ 2014 ] mention the importance of logic models and present an example. Wright [ 2011 ] explores the development of a logic model for the second Southern African Bird ATLAS project, which includes relationships amongst its various components. Chandler et al. [ 2017 ] describe a meta-analysis of evaluation data from 51 Earthwatch programmes that ran over seven years, focusing on production of literature from peer-reviewed publications to management plans and policies. The meta-analysis was based on measures of success (MoS) which, in turn, was based on a logic model approach. Their work provides great insight into the effectiveness of this approach and the lessons learned, but the actual logic model is not provided. Shirk and Bonney [ 2015 ] use a logic model approach for the development of a theoretical framework of the relationship between project design and its outcomes. Their example includes substantial information about potential citizen science outcomes and impacts which may apply in several contexts. Nevertheless, since this is not a practical implementation of a particular project, there is no discussion of the assumptions that may influence how the project is carried out and impact pathways are not considered.

Before we provide a deeper insight into the theoretical concepts that surround the logic models implementation, we present our case study. Here we discuss the theoretical concept of the ‘DITOs escalator framework’ which was used to design DITOs in a way that achieves wider and deeper public engagement — a common aim across many several citizen science initiatives.

3 Case study: Doing It Together Science

Doing It Together Science (DITOs) was a three-year H2020-funded coordination and support action (CSA), running from 2016 to 2019. The project performed over 800 events across Europe. The initial aim was to engage 290,000 people in person and 1.3m online, but the final online and offline activities resulted in engaging with over 4 million, overperforming by 166% our initial expectations. The vision was to develop and put into action frameworks, guidelines and practices to support the establishment of an integrated pan-European practice and policy for citizen science.

DITOs was organised into four themes; ‘environmental sustainability (ES)’ and ‘biodesign (BD)’ which run in parallel, and which intersected with another set of parallel themes, namely, ‘public engagement and capacity building’ and ‘policy engagement for RRI’. BD focused on aspects of biotechnology and DIY Bio, while ES included citizen science activities in ecology, biodiversity observations, and other environmental issues (e.g. community-led air quality monitoring). BD and ES involved activities such as: interactive travelling exhibitions, conferences and seminars, gaming competitions and online engagement activities, science cafes and public screenings, and DIY and DIT (Doing It Together) workshops. The ‘public engagement and capacity building’ theme, included: online activities, a travelling exhibition, conference, science cafes, workshops and online outreach activities through videos, blogs and photos. Finally, policy and RRI-related activities included European policy stakeholder round-tables and good practices workshops, delegation visits and a pan-European policy forum. DITOs paid special attention to the spectrum of citizen science with few events classified as collaborative and co-created and the majority of its activities covering methods that are commonly used to engage the public in science.

DITOs’ idea and plan aimed to address major challenges in the field of public engagement and participation with science. Its core design concept is based on the ‘DITOs escalator framework, shown in Figure 1 . Here we position the various practices of citizen science along a continuum with public engagement activities, which we then demonstrate with some evidence from the United Kingdom. Our aim is not to provide absolute and accurate numbers, but to gain a rough estimation of the scales of participation in different levels of engagement.

Figure 1 : DITOs escalator framework for public engagement.

Level 1 considers the whole population, about 65 million people. Because of the impact of science across society, the vast majority, if not all, will have some exposure to science — even if this is only in the form of medical encounters. Such encounters which are the result of living in a world dominated by science and technology are not, by themselves, part of engagement with science. However, the bare minimum of engagement is to passively consume information about science through newspapers, websites, and TV and radio programmes. We can gauge the number of people at this level from the BBC programmes Blue Planet II and Planet Earth II , both focusing on natural history, with viewing figures of around 14 million and 10 million, respectively [BBC, 2017 ; Jackson, 2016 ]. We can therefore estimate these ‘passive consumers’ at about 25% of the population.

The next level is active consumption of science — such as visits to the London science museum (U.K. visitors in 2017 — about 1.3m), a natural history museum (U.K. visitors in 2017 — about 2.1m), or attending events such as Pint of Science ( https://pintofscience.co.uk/ ) which attracted 24,000 people, so an estimation of 10% participation of the population seems justified. Next follows active but limited engagement in carrying out a scientific activity, hence taking part citizen science. Here, the Royal Society for the Protection of Birds (RSPB)’s annual Big Garden Birdwatch requires participants to dedicate a single hour in the year. The project attracted about 500,000 participants in 2017, With further estimated 170,000 people who carried out a single task on Zooniverse. We can therefore estimate participation at this level at about 1% of the population.

At the fifth level are projects that require remote engagement, such as volunteer thinking on the Zooniverse platform or volunteer computing on the IBM World Community Grid (WCG), in which participants download software onto their computer to allow processing to assist scientific research. The number of participants in WCG from the U.K. in 2017 was about 18,000 [IBM, 2018 ], and in Zooniverse about 73,000 regular volunteers in 2017 [Miller, 2018 ], thus estimating participation at 0.1% of the population. The sixth level requires regular data collection, such as participation in the British Trust of Ornithology Garden Birdwatch — about 6,500 active participants in 2017 [British Trust for Ornithology, 2018 ] — while around 5,000 contributed to the biodiversity recording system iRecord [Centre for Ecology and Hydrology, 2018 ]; it is reasonable to estimate that participation is about 0.01% of the population. The most engaged level includes DIY science participants, such as exploring DIY Bio, or developing their own sensors. This require sustained engagement and significant time investment. We can estimate this represents 0.001% of the U.K. population.

In summary, we can see that, as the level of engagement increases, the demands required of participants increase and the number of participants drops by an order of magnitude. Therefore, to enable a wider and deeper public engagement with science, as the key aim in DITOs, we considered an escalator-like process to create awareness of the various levels and assist people in moving up or down the engagement level. For example, due to changes in care responsibilities or life stages, people can become less active for a period of time and then choose to become more active later. The role of the events and activities in the DITOs project was to provide opportunities for movement across the escalator.

Later in this paper we discuss how this was applied and the project’s impact pathways to achieve this aim, using a logic model. In the next section we describe our logic model methodology.

4 Theoretical overview and the DITOs logic model

4.1 Conceptual overview of logic models as evaluation tools

Funding agencies put continuous pressure on programme coordinators to provide them with appropriate information that enables monitoring of a programme’s effectiveness and its overall performance [Gugiu and Rodríguez-Campos, 2007 ]. Logic models provide methodological tools to support this. Popular amongst programme managers and evaluators, logic models were initially introduced in the 1980s to support the identification of key activities and their outcomes and develop a programme theory to link those two parts. Due to the existence of different theories in logic model implementation, Clapham et al. [ 2017 , p. 97] argue that “the purpose of a logic model is to articulate the underlying assumptions about how the expected outcomes of a program will be reached in the short, medium and long term”. Frequently, an intervention will have multiple logic models, to correspond to different implementation levels [Rogers, 2014 ].

Logic models are particularly important for social interventions and community-based programmes as they create a basis to demonstrate logic, evidence and provide assurance they have realistic and achievable goals. Kaplan and Garrett [ 2005 ] summarise lessons learned from the use of logic models and their strengths in: identifying missing links in the causal chain; uncover key unexamined assumptions and mismatches in the priorities amongst key stakeholders; build consensus; and improve team communication. The symbols and language used in logic models can be problematic and development of programme assumptions — arguably the most valuable aspect — is very frequently left incomplete. Despite a few decades of using logic models, there is still ambiguity surrounding their implementation [Mayne, 2015 ].

Amongst the most popular logic models in the literature are the three approaches of the W.K. Kellogg Foundation [ 2004 ] logic models. First, the theory approach, explores “What issues or problems does the program seek to address? What are the specific needs of the target audience? What are the short- and long-term goals of the program? What barriers or supports may impact the success of the program?” [Gugiu and Rodríguez-Campos, 2007 , p. 340]. Second, the activities approach, links activities in a sequence across the whole programme implementation and is particularly helpful for evaluation especially in detecting barriers and obstacles for achieving programme impacts [Gugiu and Rodríguez-Campos, 2007 ]. Finally, the outcomes approach, links inputs to activities to outcomes, which is useful for monitoring the implementation of activities and their desired outcomes [Gugiu and Rodríguez-Campos, 2007 ].

Logic models have the following components: Inputs (e.g. financial, staff, material and immaterial resources); Activities (e.g. what will be produced with the resources, including tools, events and processes); Outputs (e.g. reports); Short-term impacts (i.e. within 1–3 years of project initiation); Medium-term impacts (i.e. within 3–5 years of project initiation); and Long-term impacts (i.e. 5+ years after project initiation).

4.2 DITOs logic model

The construction of the DITOs logic model was an iterative process with input from all project’s 11 partners. Several meetings and discussions were organized in the first year of the programme for this purpose, where consortium partners were provided with post-it notes and asked to write and stick on a white board their understandings of what are the project’s inputs, activities, outputs and impacts. The data provided was analysed and further processed by the paper’s authors in iterations, with feedback provided by the consortium and the project’s evaluation team at various stages.

We use a theory approach to develop but also to present and discuss in the next section a macro-level logic model for DITOs. Our logic model is a top-level model — with a ‘broad and shallow pathway’ [Taplin et al., 2013 ] which we developed as a reflective tool for articulating and communicating our assumptions, evaluating and subsequently demonstrating the programme effectiveness in the impacts we aimed to achieve. For the discussion we consider the various impact pathways, which we present at sufficient level of detail without increasing complexity. In the analysis, when applicable, we also consider ‘reach and reaction’ of the target groups that we intended to engage and thus those who ‘receive’ project goods and services; ‘capacity changes’ (short-term outcomes of those involved); ‘behavioural changes’ (medium-term outcomes of practice change); ‘direct benefits’ (medium-term outcomes as “improvements in the state of individual beneficiaries”); ‘external influences’ (“events and conditions unrelated to the intervention that could contribute to the realisation of the intended results”); anticipated ‘unintended effects’ (“positive or — more usually — negative unanticipated effects”); and assumptions which include rationale, reach, capacity change, behaviour change, direct benefits and well-being change assumptions [Mayne, 2015 ].

The DITOs logic model shown in Figure 2 ; it consists of inputs, activities, outputs and outcomes. We included participants/beneficiaries — those who are impacted by the action — as an additional component, which helps us structure and communicate our logic model more efficiently.

Figure 2 : DITOs logic model.

Inputs included financial resources (R1 in Figure 2 — DITOs’ 4m budget), of which a significant amount translates to staff in the people category (R2), which includes consortium partners, advisory boards, but also people outside the consortium who work closely on project activities, as well as people from the European Commission which monitored this project. Material resources (R3) included physical spaces for meetings and activities (e.g. the Natural History Museum in Brussels); tools and equipment (microscopes, phones or DIY kits); procedures, protocols, guidelines and other materials (e.g. how-to guides, management plans) and infrastructure services (e.g. communication software). The last category of intangible assets (R4) captures the knowledge, expertise, experiences and credibility that were brought into the project by partners and their networks. The overall design of the project focused on identifying activities that are already happening by the partners (e.g. a museum exhibition) and then providing additional resources and activities to open up the movement along the escalator.

Groups of DITOs activity beneficiaries included the public (Pub), science practitioners (Sci), and policymakers (Pol). Public engagement with science activities aims to involve people from all walks of life. Nevertheless, some people are harder to engage and many projects (including DITOs) make extra effort to reach those who are usually excluded due to their geographic location, gender, education and socioeconomic status. Science practitioners include scientists, science communicators, citizen science practitioners, designers and others involved in the planning and implementation of public engagement with science activities. Finally, the last category includes policymakers at different levels, including civil servants interacting with the project or using the outcomes, or involved in funding decisions.

The next component includes project activities, which activated the DITOs’ escalator. In their most generic form, 11 main activity components were essential in producing the desired outputs. First, the development of an outreach plan (A1) aimed to gather information from project partners about previous activities and build on their experiences (R4). Following these plans, events (A2-A5) were designed for the proposed beneficiaries (Pub, Sci, Pol). Because of the need to pay special attention to hard-to-reach audiences, they received dedicated activities (A3). Central to the project were planning communication and dissemination activities (A6) and setting up the tools and creating content (A7) (e.g. mailing lists, printed materials, project website). Like the previous category, this one involved a high number of nested activities, which took place at various parts of the project and produced substantial inputs for other activities.

The next activity was monitoring and evaluation of all events using both qualitative and quantitative methods (A8). A specific set of activities focused entirely on the development of DITOs innovation hubs: i.e. centres of excellence and a sustainable network for capacity building in citizen science (A9). In terms of achieving a sustained impact and because of the timescales of engagement, there is frequently a need to ensure that capacity to carry out engagement continues over time which was the role of DITOs innovation hubs. The penultimate category addressed the development and submission to the funder of project deliverables (A10), (27 in DITOs, mostly reports). Finally, project management and coordination (A11) included meetings, financial management and so on.

In our logic model we group outputs (and short-term, medium-term and long-term impacts) according to the beneficiaries to which they are most relevant. For example, all groups benefit from outputs such as reports, events, and communication and dissemination tools and materials that target their specific interest. Although we discuss pathways in more detail in the next section, it should be noted here that activities such as the outreach plan (A1), running public events (A2, A3), the communication and dissemination plan (A6), relevant content and tool creation (A7) which targeted members of the public in general, deliverables that are open to the public (A10), and overall project management (A11) all contributed to outputs that are beneficial for the public category (i.e. O1, O2, O3).

Because of its scale, DITOs outputs provide a good demonstration of the range of possible outputs in a citizen science project. These included 20 public reports available on UCL’s open access repository. Some are perhaps more beneficial to science practitioners and policymakers (O10) (e.g. reports in sustainable support for citizen science, evaluation terms of reference, templates). For policymakers, our policy briefs and reports from delegation visits or round-table events were of highest interest. Communication and dissemination channels, tools and materials were produced from various activities and aimed at different target audiences with different levels of interest in science and citizen science (O3, O6, O12); this included translations of citizen science materials into multiple European languages; channels such as #CitSciStories on Twitter where volunteer in citizen science activities (citizen scientists) shared their citizen science stories; DIY science postcards for DIY activities, and others. Outputs such as the design, planning and establishment of DITOs innovation hubs (O7) and an increase in the number of European Citizen Science Association (ECSA) memberships (O8) were more relevant to science practitioners. This was also true for project tools and mechanisms such as mailing lists, evaluation templates and the networks established through social media (O9).

The impacts of the project were set to short-term (during the lifetime of the project), medium-term (up to 5 years), and long-term impacts (up to 10 years). Some of them were set by the European Commission in the call text [see Göbel et al., 2019 ] while others were developed by the partners and other stakeholders. In the short term DITOs aimed to: improve the public’s awareness of science and RRI (ST1) and participation (ST2) in science and citizen science; improve visibility of science and citizen science (ST3) and increase people’s science knowledge and scientific literacy (ST4); support the development of methods and tools for public engagement (ST5); improve science practitioners’ knowledge and skills in terms of engagement with members of the public (ST6) and; contribute to the establishment of a pan-European centre of citizen science in the form of ECSA (ST7). On the policy side, the project aimed to improve policymakers’ awareness of public engagement with science and citizen science (ST8) and; provide policy support to guide agencies to set up schemes that take into account different levels of engagement (ST9, ST10).

In the medium-term activities DITOs aimed to: enable the engagement of citizens in shaping and conducting scientific research (MT1) and social and gender inclusiveness in science (MT2); enhance scientific and civic literacy (MT3); enhance the capacity of local science actors in public engagement in science (MT4) and; strengthen European cooperation and support in citizen science (MT5). At the policy level, the project aimed to increase the engagement of policymakers in shaping and conducting scientific research (MT6) and help ensure citizen science gains the support of policymakers (MT7). The long-term impacts are, again, unified across participation groups and include wider and deeper public engagement in science (LT1), maximising European innovation (LT2) and increasing societal input and external advice to research and innovation policies (LT3).

5 Discussion

To understand impact pathways and explain the way the components of the logic model work together towards materializing anticipated impacts, we need to start first with DITOs assumptions. Although various assumptions may be detected at various programme implementation levels, we focus here on assumptions that are common in citizen science. For example, the goal of achieving ‘wider public engagement’ incorporates strong assumptions about people’s inherent interests and motivations to engage with science. We carried out specific activities that further targeted beneficiaries such as women and girls and communities that are marginalised or without easy access to public science activities. Although many governmental and research reports stress the importance of engaging the ‘hard-to-reach’, existing practices show that most outreach efforts tend to include volunteers who are predisposed to participate and they already value science [Varner, 2014 ] so, while this can be true from level 2 or 3 of the escalator, special effort is required at the transition between levels 1 and 2.

The second assumption is that all public engagement with science activities improves science knowledge and scientific literacy and leads to positive attitudes towards science [Brossard, Lewenstein and Bonney, 2005 ]. Here, there are calls for more systematic evaluations and a need for further evidence to demonstrate this impact [Reed et al., 2018 ]. The development of sustainable networks for innovation and capacity building builds on the assumption that science practitioners and/or policymakers have a genuine and shared interest in exchanging knowledge and practices continuously, which might not always be the case [Powell and Colin, 2008 ]. Another assumption is funding availability for public engagement and citizen science in the short and long-term, which, considering the limited SwafS programme funding under which DITOs was set (about 0.60% of the Horizon 2020 budget), might be challenging.

Next, we should be cautious about assuming that all citizen science is, by default, positive [Stilgoe, 2016 ] or that policymakers will have the time and interest in getting involved in public discussions about science. Last but not least, it should be acknowledged that ‘impact occurs within a particular setting [Economic and Social Research Council, 2009 ] and DITOs’ logic model merges several public engagement activities which vary in their cultural, contextual settings and policy frameworks; therefore, impacts may not be as unified as discussed in the next section.

5.1 DITOs impacts at three engagement levels

We discuss below the impact pathways for the three types of DITOs beneficiaries: public, science practitioners and policymakers.

A. Public engagement impacts. The pathway towards ‘wider and deeper public engagement in science’ achieved with DITOs events, which covered a broad range of engagement with science, citizen science and DIY science at different levels of the escalator. Since the lower rungs have the highest number of participants, many of the activities targeted participants at the transitions from levels 1–2 and 2–3. Direct outputs of these activities contributed to improving public awareness of science and RRI (ST1), by increasing awareness of opportunities to participate more fully in scientific activity, since most participants have not heard about citizen science. This created the opportunity to increase participation in science and citizen science (ST2), which includes both access and exposure to scientific information and awareness of the opportunities and resources to organise themselves and engage in matters that are important to them. Some events specifically targeted women and girls, with a communication strategy that focused from the beginning on engaging these groups both online and offline. Social exclusion is addressed through a travelling exhibition in remote areas. By operating across the escalator, it was possible to improve the visibility of citizen science and DIY science (ST3). Various events on science topics (e.g. Bioblitzes, science cafes) resulted in improving participants’ science knowledge and scientific literacy skills (ST4) which also contributed to ST1.

The efficacy of the efforts was evaluated (A8) in a way that informed the development of ongoing events. Meeting short-term impacts meant that, by employing a variety of engagement techniques matching local contexts, citizens built their own capacities (in terms of knowledge and skill) to engage further with other citizens, but also scientists and policymakers in shaping and conducting research (MT1). The latter clearly depends on the willingness of more powerful actors to open up opportunities for participation, which further results in improved scientific and civic literacy (MT3). The aim is that, with both opportunities and networks established, personal and community relationships, gender and social inclusiveness in science and citizen science can be improved (MT2). It should be noted that it is the developments in the other parts of the mechanism that further enable the materialisation of these specific impacts in the medium and long term.

B. Impacts from engagement with science practitioners. These emerged from outputs that outlined and provided instructions and best practice guidelines (e.g. outreach plan, good practice guides, evaluation guidelines) for science practitioners who can use DITOs tools and methods for their engagement practices or to learn from failures and successes for their own designs (O4, O5, O6). Capacity building was and will be further enhanced through DITOs innovation hubs (O7), which are organisations capable of utilising citizen science at different levels as a result of the project. DITOs innovation hubs introduced five principles, which include communal — encourage collaboration and togetherness; self-organising and adaptive — allow innovators to set the agenda and trajectory of innovation; interdisciplinary knowledge transfer ; enabling innovators — hubs empower participants to transform into innovators; global impact — innovations tackle global-scale impacts. The expectation was that, through this effort, the value of knowledge-sharing that ECSA provides would increase and therefore its membership would also increase (O8). Finally, the project introduced a set of tools and mechanisms (O9) that are open to science practitioners such as use of social media (i.e. using DITOs social media to promote other public engagement projects), or evaluation forms and templates.

In the short-term, this effort to share methods, tools and, most importantly, lessons learned will further support science practitioners in adapting them to fit their needs beyond the project’s duration (ST5). DITOs events and other outputs were designed to contribute to capacity building of science practitioners, and the growth of ECSA will further increase knowledge, skills and capacity as well as opportunities for local science actors and decision-makers to get involved in similar initiatives (ST6) [Storksdieck et al., 2016 ]. This contributes towards turning ECSA into a pan-European knowledge and resource centre for citizen science (ST7). In the medium-term, innovation hubs, the availability of citizen science methods and tools, and the support provided by ECSA are expected to enhance capacity of local science actors (MT4). Sustainable capacity building through ECSA, cross-European fertilisation and innovation hubs can result in strengthening cooperation and support in citizen science and, more broadly, in science across Europe (MT5).

C. Impacts from engagement with policymakers. These emerged from the use of resources towards dedicated events such as Discovery Trips and stakeholder round tables, which engaged policymakers at local, regional, national and EU level. DITOs produced good practice guidelines and policy briefs and a targeted online communication and dissemination strategy for these stakeholders through printed media, videos, reports and newsletters. From these outputs in the short-term, the project improved policymakers’ citizen science awareness across Europe (ST8), with emerging evidence of the term used in policy reports [European Commission, 2017 ]. Policy-related outputs should result in further policy support for citizen and DIY science (ST9) which can be seen in the local support to citizen science in Horizon Europe programme. As usual with policy impacts, it is impossible to claim that DITOs is the main reason for this, but it is clear that the sustained engagement and exposure of science policy makers to citizen science within DITOs activities contributed to this. DITOs events at various levels of the ‘escalator’ and the involvement of policymakers led to appreciation of the escalator mechanism (ST10), with contribution to the framing of science and society questions to the Eurobarometer survey, or the integration of the model within a prominent report by the EC [European Commission, 2020 ]. The expected medium-term changes may push the boundaries of participation in science policymaking (MT6) and help appreciate and support the benefits of citizen science (MT7). Here, awareness that other levels of government utilise citizen science can push its adoption.

The final set of goals are long-term impacts, which are dependent on other projects and activities well beyond DITOs’ limited resources and duration. These long-term goals are wider and deeper public participation in science and greater awareness of RRI. Here the project addresses the challenge of deepening public engagement by having created a virtual ‘escalator’ that, as we demonstrated, enables people to decide which level of contribution is suitable for them, while gently exposing them and encouraging them to move to the next level. The goal to maximise European innovation is linked to the ongoing concern by European policymakers about the challenges from the US and Asia in scientific dominance [Hoenig, 2017 ]. The promotion of citizen and DIY science is carried out to encourage user-led and DIY innovation by drawing on the expertise of grassroots and DIY groups and organisations, which have a deeper understanding of local contexts. The DITOs escalator framework seeks to identify more of these groups and individuals and potentially offer them a pathway for utilising their efforts. DITOs further stimulates European innovation by building civil society’s capacity and response capabilities through the co-design and subsequent open access to research tools, methods and spaces. The final goal to maximise societal input and external advice to R&I policies is based on the wider assumptions behind the RRI efforts that were noted earlier [Owen, Macnaghten and Stilgoe, 2012 ].

6 Conclusions

In this paper, we demonstrated how logic models can be utilized in a large-scale project that focuses on public engagement in science and how pathways between the different components can be used to demonstrate and achieve impact. We have also positioned citizen science within the wider context of public engagement in science and explained how the use of these tools together with the novel concept of the ‘DITOs escalator framework’ can assist in achieving long-term goals which are common across many citizen science initiatives. By looking at the programme’s logic model at the top level, during its implementation, we were able to identify the right mechanisms at a strategic level to ensure that tools and instruments are in place to achieve the change that is required.

Important lessons learned from our experience include identifying and targeting all beneficiaries appropriately at all levels, and identifying opportunities and challenges throughout the project’s duration and designing relevant actions to address them (e.g. designing appropriate communication mechanisms and content creation for iterative evaluation activities that set the space for reflection and improvement from early design stages). We have learned that, while it is possible to envisage a model which tries to identify what each action aims to achieve, it is easier to miss those top-level impacts and consider the pathways which we believe are problematic areas of most existing public engagement programmes. A logic model can support this and, by considering the linear pathways (or, in our case, not so linear) to impact, gaps and misconceptions can be identified early enough and planned into programme design and delivery.

We would like to acknowledge that an abstract discussion of our programme’s logic model can be that it removes the context from specific public engagement events and activities and focuses on how the mechanism that we describe works in situated practices in a way that inspires others to follow a similar approach. One important note to make with respect to cultural contexts is that partners who organise local events do have extensive experience in their local cultural contexts and, by emphasising the network effect of the wider ECSA network, it is possible to tailor the framework to these contexts.

Acknowledgments

This research has received funding by: the European Union’s H2020 research and innovation programme DITOS (Doing It Together Science) under Grant Agreement No. 709443; the European Union’s H2020 research and innovation programme EU-Citizen.Science (The Platform for Sharing, Initiating and Learning Citizen Science in Europe) under Grant Agreement No. 824580; the European Union’s H2020 coordination and support action TIME4CS (Sustainable institutional changes to promote public engagement in science and technology) under Grant Agreement No. 101006201; the ERC Advanced Grant project European Citizen Science: Analysis and Visualisation under Grant Agreement No. 694767 and the National Co-ordinating Research Centre for Public Engagement (NERC) funded project Engaging Environments (NE/S017437/1).

References

: Bauer, A., Bogner, A. and Fuchs, D. (2016). D2.2. Societal engagement under the terms of RRI . PROSO project. URL: http://www.proso-project.eu/wp-content/uploads/proso_d2.2_societal_engagement.pdf .
: Bauer, M. W. (2009). ‘The Evolution of Public Understanding of Science — Discourse and Comparative Evidence’. Science, Technology and Society 14 (2), p. 221. https://doi.org/10.1177/097172180901400202 .
: BBC (2017). Blue Planet II tops 2017 TV ratings . URL: https://www.bbc.co.uk/news/entertainment-arts-42641146 (visited on 26th August 2018).
: Bonney, R., Shirk, J. L., Phillips, T. B., Wiggins, A., Ballard, H. L., Miller-Rushing, A. J. and Parrish, J. K. (2014). ‘Citizen science. Next steps for citizen science’. Science 343 (6178), pp. 1436–1437. https://doi.org/10.1126/science.1251554 . PMID: 24675940 .
: British Trust for Ornithology (2018). Garden BirdWatch submissions . URL: https://www.bto.org/volunteer-surveys/gbw/results/submissions (visited on 26th August 2018).
: Brossard, D., Lewenstein, B. and Bonney, R. (2005). ‘Scientific knowledge and attitude change: The impact of a citizen science project’. International Journal of Science Education 27 (9), pp. 1099–1121. https://doi.org/10.1080/09500690500069483 .
: Bush, V. (1945). Science — the endless frontier: a report to the president on a program for postwar scientific research. Washington, DC, U.S.A.: National Science Foundation. URL: https://www.nsf.gov/od/lpa/nsf50/vbush1945.htm .
: Carr, A. J. L. (2004). ‘Why do we all need community science?’ Society & Natural Resources 17 (9), pp. 841–849. https://doi.org/10.1080/08941920490493846 .
: Centre for Ecology and Hydrology (2018). Personal communication on iRecords statistics .
: Chandler, M., Rullman, S., Cousins, J., Esmail, N., Begin, E., Venicx, G., Eisenberg, C. and Studer, M. (2017). ‘Contributions to publications and management plans from 7 years of citizen science: use of a novel evaluation tool on Earthwatch-supported projects’. Biological Conservation 208, pp. 163–173. https://doi.org/10.1016/j.biocon.2016.09.024 .
: Clapham, K., Manning, C., Williams, K., O’Brien, G. and Sutherland, M. (2017). ‘Using a logic model to evaluate the Kids Together early education inclusion program for children with disabilities and additional needs’. Evaluation and Program Planning 61, pp. 96–105. https://doi.org/10.1016/j.evalprogplan.2016.12.004 .
: Cooper, C. B., Shirk, J. and Zuckerberg, B. (2014). ‘The invisible prevalence of citizen science in global research: migratory birds and climate change’. PLoS ONE 9 (9), e106508. https://doi.org/10.1371/journal.pone.0106508 .
: De Saille, S. (2015). ‘Innovating innovation policy: the emergence of ‘responsible research and innovation’’. Journal of Responsible Innovation 2 (2), pp. 152–168. https://doi.org/10.1080/23299460.2015.1045280 .
: Economic and Social Research Council (2009). Taking stock: a summary of ESRC’s work to evaluate the impact of research on policy and practice . Swindon, U.K. URL: https://esrc.ukri.org/files/research/research-and-impact-evaluation/taking-stock-a-summary-of-esrc-s-work-to-evaluate-the-impact-of-research-on-policy-and-practice/ .
: Elam, M. and Bertilsson, M. (2003). ‘Consuming, engaging and confronting science’. European Journal of Social Theory 6 (2), pp. 233–251. https://doi.org/10.1177/1368431003006002005 .
: European Commission (2009). Preparing Europe for a new renaissance: a strategic view of the european research area . First report of the european research area board — 2009. Brussels, Belgium. URL: http://publications.europa.eu/resource/cellar/89724e55-9579-4bd8-80cf-9a21ba645cbc.0001.02/DOC_1 .
: — (2017). LAB — FAB — APP — Investing in the European future we want . Report of the independent High Level Group on maximising the impact of EU Research & Innovation Programmes. Brussels, Belgium: Directorate-General for Research and Innovation. URL: https://op.europa.eu/en/publication-detail/-/publication/ffbe0115-6cfc-11e7-b2f2-01aa75ed71a1/language-en/format-PDF/source-77975731 (visited on 30th January 2019).
: — (2020). Citizen science and citizen engagement — achievements in Horizon 2020 and recommendations on the way forward . Brussels, Belgium. URL: https://op.europa.eu/en/web/eu-law-and-publications/publication-detail/-/publication/c30ddc24-cbc6-11ea-adf7-01aa75ed71a1 .
: Gerber, A., Forsberg, E.-M., Shelley-Egan, C., Arias, R., Daimer, S., Dalton, G., Belén Cristóbal, A., Dreyer, M., Griessler, E., Lindner, R., Revuelta, G., Riccio, A. and Steinhaus, N. (2020). ‘Joint declaration on mainstreaming RRI across Horizon Europe’. Journal of Responsible Innovation 7 (3), pp. 708–711. https://doi.org/10.1080/23299460.2020.1764837 .
: Göbel, C., Nold, C., Berditchevskaia, A. and Haklay, M. (2019). ‘How does citizen science “do” governance? Reflections from the DITOs project’. Citizen Science: Theory and Practice 4 (1), p. 31. https://doi.org/10.5334/cstp.204 .
: Gregory, J. and Lock, S. J. (2008). ‘The Evolution of ‘Public Understanding of Science’: Public Engagement as a Tool of Science Policy in the UK’. Sociology Compass 2 (4), pp. 1252–1265. https://doi.org/10.1111/j.1751-9020.2008.00137.x .
: Gugiu, P. C. and Rodríguez-Campos, L. (2007). ‘Semi-structured interview protocol for constructing logic models’. Evaluation and Program Planning 30 (4), pp. 339–350. https://doi.org/10.1016/j.evalprogplan.2007.08.004 .
: Haklay, M. (2013). ‘Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation’. In: Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI) in Theory and Practice. Ed. by D. Sui, S. Elwood and M. Goodchild. Berlin, Germany: Springer, pp. 105–122. https://doi.org/10.1007/978-94-007-4587-2_7 .
: Hoenig, B. (2017). Europe’s new scientific elite. London, U.K.: Routledge. https://doi.org/10.4324/9781315446042 .
: IBM (2018). World Community Grid team, personal communication . (Visited on 30th November 2018).
: Jackson, J. (7th November 2016). ‘Planet Earth II most watched natural history show for 15 years’. The Guardian . URL: https://www.theguardian.com/tv-and-radio/2016/nov/07/planet-earth-ii-bbc1-most-watched-natural-history-show-for-15-years (visited on August 2018).
: Jasanoff, S. (2003). ‘Technologies of Humility: Citizen Participation in Governing Science’. Minerva 41 (3), pp. 223–244. https://doi.org/10.1023/A:1025557512320 .
: Jordan, R. C., Gray, S. A., Howe, D. V., Brooks, W. R. and Ehrenfeld, J. G. (2011). ‘Knowledge Gain and Behavioral Change in Citizen-Science Programs: Citizen-Scientist Knowledge Gain’. Conservation Biology 25 (6), pp. 1148–1154. https://doi.org/10.1111/j.1523-1739.2011.01745.x .
: Kaplan, S. A. and Garrett, K. E. (2005). ‘The use of logic models by community-based initiatives’. Evaluation and Program Planning 28 (2), pp. 167–172. https://doi.org/10.1016/j.evalprogplan.2004.09.002 .
: Liu, H.-Y. and Kobernus, M. (2017). ‘Citizen science and its role in sustainable development: status, trends, issues and opportunities’. In: Analyzing the role of citizen science in modern research. Ed. by L. Ceccaroni and J. Piera. Hershey, PA, U.S.A.: IGI Global, pp. 147–167. https://doi.org/10.4018/978-1-5225-0962-2.ch007 .
: Martin, B. (2006). ‘Strategies for alternative science’. In: The new political sociology of science: institutions, networks and power. Ed. by S. Frickel and K. Moore. Madison, WI, U.S.A.: The University of Wisconsin Press, pp. 272–298.
: Mayne, J. (2015). ‘Useful theory of change models’. Canadian Journal of Program Evaluation 30 (2), pp. 119–142. https://doi.org/10.3138/cjpe.230 .
: Miller, G. (27th September 2018). Email, personal communication .
: Newman, G., Wiggins, A., Crall, A., Graham, E., Newman, S. and Crowston, K. (2012). ‘The future of citizen science: emerging technologies and shifting paradigms’. Frontiers in Ecology and the Environment 10 (6), pp. 298–304. https://doi.org/10.1890/110294 .
: Ottinger, G. (2010). ‘Buckets of Resistance: Standards and the Effectiveness of Citizen Science’. Science, Technology & Human Values 35 (2), pp. 244–270. https://doi.org/10.1177/0162243909337121 .
: Owen, R., Macnaghten, P. and Stilgoe, J. (2012). ‘Responsible research and innovation: From science in society to science for society, with society’. Science and Public Policy 39 (6), pp. 751–760. https://doi.org/10.1093/scipol/scs093 .
: Phillips, T., Ferguson, M., Minarchek, M., Porticella, N. and Bonney, R. (2014). User’s Guide for Evaluating Learning Outcomes in Citizen Science. Ithaca, NY, U.S.A.: Cornell Lab of Ornithology. URL: http://citizenscience.org .
: Powell, M. C. and Colin, M. (2008). ‘Meaningful citizen engagement in science and technology. What would it really take?’ Science Communication 30 (1), pp. 126–136. https://doi.org/10.1177/1075547008320520 .
: Reed, M. S. C., Duncan, S., Manners, P., Pound, D., Armitage, L., Frewer, L. J., Thorley, C. and Frost, B. (2018). ‘A common standard for the evaluation of public engagement with research’. Research for All 2 (1), pp. 143–162. https://doi.org/10.18546/rfa.02.1.13 .
: Rogers, P. (2014). Theory of change, methodological briefs: impact evaluation 2. Florence, Italy: UNICEF Office of Research. URL: https://www.unicef-irc.org/publications/747-theory-of-change-methodological-briefs-impact-evaluation-no-2.html .
: Rowe, G. and Frewer, L. J. (2005). ‘A Typology of Public Engagement Mechanisms’. Science, Technology & Human Values 30 (2), pp. 251–290. https://doi.org/10.1177/0162243904271724 .
: Royal Society (1985). The Public Understanding of Science. London, U.K.: Royal Society. URL: https://royalsociety.org/~/media/Royal_Society_Content/policy/publications/1985/10700.pdf .
: Shelley-Egan, C., Gjefsen, M. D. and Nydal, R. (2020). ‘Consolidating RRI and open science: understanding the potential for transformative change’. Life Sciences, Society and Policy 16 (1), pp. 1–14. https://doi.org/10.1186/s40504-020-00103-5 .
: Shirk, J. L. and Bonney, R. (2015). Informing a framework for citizen science within the U.S. fish and wildlife service. Ithaca, NY, U.S.A.: Cornell Lab of Ornithology. URL: https://ecos.fws.gov/ServCat/DownloadFile/56072?Reference=52383 .
: Shirk, J. L., Ballard, H. L., Wilderman, C. C., Phillips, T., Wiggins, A., Jordan, R., McCallie, E., Minarchek, M., Lewenstein, B. V., Krasny, M. E. and Bonney, R. (2012). ‘Public Participation in Scientific Research: a Framework for Deliberate Design’. Ecology and Society 17 (2), p. 29. https://doi.org/10.5751/ES-04705-170229 .
: Socientize (2014). White paper on citizen science in Europe . URL: https://ec.europa.eu/newsroom/dae/document.cfm?doc_id=6913 .
: Stilgoe, J. (8th November 2016). ‘Is citizen science the future of research or a recipe for bad science?’ The Guardian . URL: https://www.theguardian.com/science/political-science/2016/nov/08/is-citizen-science-the-future-of-research-or-a-recipe-for-bad-science (visited on 26th August 2018).
: Stilgoe, J., Lock, S. J. and Wilsdon, J. (2014). ‘Why should we promote public engagement with science?’ Public Understanding of Science 23 (1), pp. 4–15. https://doi.org/10.1177/0963662513518154 .
: Storksdieck, M., Shirk, J. L., Cappadonna, J. L., Domroese, M., Göbel, C., Haklay, M., Miller-Rushing, A. J., Roetman, P., Sbrocchi, C. and Vohland, K. (2016). ‘Associations for Citizen Science: Regional Knowledge, Global Collaboration’. Citizen Science: Theory and Practice 1 (2), pp. 1–10. https://doi.org/10.5334/cstp.55 .
: Taplin, D. H., Clark, H., Collins, E. and Colby, C. (2013). Theory of change — technical papers. New York, NY, U.S.A.: Act Knowledge. URL: https://www.actknowledge.org/resources/documents/ToC-Tech-Papers.pdf .
: Varner, J. (2014). ‘Scientific Outreach: Toward Effective Public Engagement with Biological Science’. BioScience 64 (4), pp. 333–340. https://doi.org/10.1093/biosci/biu021 .
: W.K. Kellogg Foundation (2004). Using logic models to bring together planning, evaluation and action — logic model development guide. Battle Creek, MI, U.S.A.: W.K. Kellogg Foundation. URL: https://www.aacu.org/sites/default/files/LogicModel.pdf .
: Wiggins, A. and Crowston, K. (2012). ‘Goals and Tasks: Two Typologies of Citizen Science Projects’. In: Proceedings of the 45 ^th Hawaii International Conference on System Science (HICSS-45) , pp. 3426–3435. https://doi.org/10.1109/HICSS.2012.295 .
: Wright, D. (2011). ‘Evaluating a citizen science research programme’. MSc dissertation. Cape Town, South Africa: University of Cape Town. URL: http://hdl.handle.net/11427/10904 .

Authors

Artemis Skarlatidou is a Senior Research Associate in the Extreme Citizen Science (ExCiteS) group at UCL. Her research interests include Risk Communication, Human-Computer Interaction (HCI) and User Experience aspects (e.g. usability, aesthetics, trust) of citizen science applications, geospatial technologies and their spatial representations for expert and public use. E-mail: a.skarlatidou@ucl.ac.uk .

Muki Haklay is a professor of Geographic Information Science at UCL. His research interests include public access, use, and creation of environmental information; interaction with geographical technologies; participatory mapping; and citizen science. He is the co-director of UCL Extreme Citizen Science group (ExCiteS) and co-founder of the social enterprise Mapping for Change. E-mail: m.haklay@ucl.ac.uk .