Beyond self-confidence: a participatory evaluation of personal change in Science Gallery's Mediators

---

1 Context

In 2008, a new exhibition space was opened in an old, abandoned car lot at the back of Trinity College Dublin, and Science Gallery was borne [Brunswick, 2017 ]. Science Gallery Dublin has a lofty mission: to ignite creativity and discovery where science and art collide. Its success has been such that this mission is now shared with an entire Network of Science Gallery venues: Science Gallery London (King’s College London), Science Gallery Bengaluru (Indian Institute of Science), Science Gallery Melbourne (University of Melbourne), Science Gallery Venice (Ca’ Foscari University of Venice) and Science Gallery Lab Detroit (Michigan State University).

Among the many things that Science Gallery sets out to do differently, its Mediator Programme has attempted to take a less didactic, more conversational approach to interactions with visitors and the Mediators “are the front line at Science Gallery Dublin” [Science Gallery Dublin, 2016 , p. 27]. Qualitative and quantitative surveys both have pointed to the success of this model with visitors, but little was known about the experience for the Mediators themselves [Fanning, 2015 ; Mordan and Hughes, 2015 ]. Mediators are an unusual and particularly interesting subset of individuals involved with Science Gallery: they are deeply engaged both as staff and participants in Science Gallery activities and mostly, though not always, represent Science Gallery’s key demographic ages 15–25. ¹ Although they are typically students or post-graduates, this is not a requirement of the role, and some Mediators were not in the process of studying while employed. New hires must commit to working for at least the duration of one exhibition (3–4 months); on average, a Mediator now stays with Science Gallery for two exhibitions, though some have stayed on for more than 3 years. As Science Gallery programming is concerned with both the arts and sciences [Bandelli and Zeijden, 2015 ], Mediators are not expected to be experts in either realm but rather must demonstrate a keen passion for learning and transdisciplinary approaches. Applicants to the Mediator role have sometimes heard about it through a friend, while others grow into the role through prior engagement during secondary school. Some enterprising young people search the application forms out themselves by asking for further information from staff. One way or another, the jobs are never widely advertised, as keen applicants generally come to Science Gallery. Recruiters for the programme are mindful to maintain gender balance, engage with students from a variety of academic backgrounds and institutions, and make space for those from underrepresented/underserved communities. The Mediator programme is not designed to create public engagement or museum professionals; rather, the hope is to equip young people with the skills required for the 21st century workforce: empathy, perspective, and a better sense of self. They are encouraged to go on and pursue careers that speak to their educational pursuits and passions. The programme therefore serves the dual purpose of both encouraging the public’s engagement at Science Gallery Dublin, while making a difference to the lives of the Mediators themselves.

2 Objective

Following participation in the Mediator Programme, anecdotal evidence points to multiple successes in both personal and professional development, but little had been done to understand the underlying reasons — is it the skill-set associated with being a Mediator, prolonged exposure to Science Gallery programmes, or simply that the Mediator Programme attracts high-achieving individuals? The specifically intended outcome of being a Mediator has never been defined, therefore we looked for a way of capturing both intended goals and unknown effects. The Mediator Programme has the potential to develop multiple skill sets, but requires much time and study; participation therefore has the potential to have both positive (e.g. increased interest in STEM, improved communication skills, enjoyment) and negative outcomes (e.g. burnout, reduced study time, frustration). In many ways, Science Gallery’s success is owed to its ability to marry art and science, and in its ability to shape the narratives that run across disciplines and boundaries. Using a simple quantitative survey alone felt like too much of a box-ticking exercise; we might miss the context behind the answers, the really beautiful stories. We needed to capture those.

When researching about how best to capture the stories and interrogate the Mediator programme, we landed upon the Most Significant Change (MSC) method [Davies and Dart, 2005 ], borrowed from the world of humanitarian and development projects. Initially developed to address complex programmes where the goals are not explicit, and where consequences might be unintended or even negative, MSC “is a qualitative and participatory form of monitoring and evaluation. It is based on the collection, systematic selection, and analysis of stories of important changes in people’s lives, resulting from development activities. MSC uses the concept of ‘monitoring without indicators’. The participatory fundament is captured through the involvement of project stakeholders and beneficiaries in the analysis of the data and the decision of the most significant among the changes, to be reported.” [Hidayati, 2018 ]. This technique does not measure the expected outputs of a programme, but stimulates identification of unexpected and unplanned programme achievements. MSC asks programme beneficiaries to identify what they think is the most significant change, which may be positive or negative, in themselves as a result/impact of the programme they are involved in.

3 Methods

Prior to the collection of the Mediators’ stories of reported changes, the Education Team at Science Gallery Dublin responsible for the Mediator Programme defined 7 broad categories they believed would represent the primary domains of change:

(A)

Health & Wellness, including Safety and Acceptance;

(B)

Civic Engagement, Interest in Social Justice, Emotional Empathy;

(C)

Critical Thought, Analysis and Problem-Solving;

(D)

Career Pursuits;

(E)

Academic and Research Pursuits;

(F)

Using Trans- and Multidisciplinary Approaches;

(G)

Confidence, Adapting and Interpersonal Skills.

To this, the Research Team at Science Gallery Dublin also added:

(H)

No Change and

(I)

Other, to allow for possibilities outside of the initially identified domains.

Thirteen Mediators participated in the research over the course of 4 weeks in May 2017. We asked each Mediator the question,

The question was answered by telling a brief “change story”. Respondents could choose to write the answer themselves, or to dictate while the interviewer typed. The respondent and interviewer completed a final review of the story to ensure it accurately represented what they were hoping to convey, and that the details provided ensured they were still relatively anonymous (as were any other individuals they made reference to in the story). Some respondents chose to put their names to the stories, while others decided to maintain anonymity. Respondents were also asked to complete a survey with demographic information and qualitative and quantitative data about their experience working as Mediators.

The stories were analysed using a participatory method known as “the selection process” that had stakeholders (staff ² ) choose the story that for them represented “the most significant change” across all 13 participants. A single layer selection (one group, one session) process was used for the review, rather than multiple stages, in order to maintain simplicity during this pilot phase. The selection process first asked stakeholders to categorise the stories of change across the previously identified Domains. Changes could additionally be identified as “positive” or “negative”. The process resulted in the choice of one story as the Story of Most Significant Change (see the “Story of Most Significant Change” section below, and appendix A for the full story.)

A secondary analysis was then completed by the research team through the use of content analysis. This secondary phase identified both trends and patterns including both the Story of Most Significant Change and the other non-selected stories. The data gathered from the survey also provided additional demographic information, as well as more detailed information about their experience as a Mediator, including perceptions of the values, knowledge and behaviour associated with the programme. We have used this information to provide additional data about the Mediator group, as the Most Significant Change method is generally not employed without some additional quantitative data.

4 Results

4.1 Who were our storytellers?

---

---

The sample above is broadly representative of the Mediator pool at any given time, as typically there are slightly more females than males employed as Mediators. During the year the research focused on, Science Gallery Dublin employed 13 female and 12 male Mediators, or a 52%–48% split. Since Science Gallery opened, the documented split is 57%–43%. Although Science Gallery Dublin does not keep records on the education levels of its Mediators, current staff agree that the education levels represented in the sample were also broadly correct: current undergraduate students represent a slight majority, while graduate students and recently graduate students make up the other 25%–35%.

4.2 Story of Most Significant Change

By: Responder 6 (Male, 21)

The selected account as MSC Story was Story 6 (see appendix A ), from a male respondent. The story was compelling, as it sketched a Mediator’s journey from start to finish and discussed the extremely positive benefit of his introduction to arts/science methods, which ultimately led to a better sense of direction in both his personal and professional life. However the story also alluded to some of what he perceived as the negative elements of the programme, including difficulty in communications between full-time/permanent staff and the Mediator team.

The majority of stakeholders involved in the selection process identified Story 6 with Domain (F) Using Trans- and Multidisciplinary Approaches , and only one stakeholder felt that it demonstrated an instance of the more frequently cited domain across all stories, which was (G) Confidence, Adapting and Interpersonal Skills . The stakeholders were compelled by Story 6 because unlike many of the other stories it contained both positive and negative elements (skills gained, confidence grown, but also stress, followed by a sense of loss when they stepped away from their role as Mediator).

Other beautiful stories and insights also came out of the process, and we have collected some of the quotes that resonated with us the most:

4.3 Secondary Analysis

Number of Stories Collected: 13

The most frequently identified primary domain of change was (F) Confidence, Adapting and Interpersonal Skills ; this was true for both male and female Mediators (see table 1 ). This identified domain of change does correspond strongly to quantitative data gathered from surveys given to the same group of Mediators; when asked to indicate their level of agreement with the statement “My involvement in the Mediator Programme has improved my comfort level speaking with others” 69.2% strongly agreed, and 30.8% agreed (none were neutral or in disagreement).

While each story had a “primary domain” (the domain agreed upon by the majority of reviewers), some stakeholders felt that stories could correspond to multiple domains. In total, the most identified domains overall were (G) Confidence, Adapting and Interpersonal Skills , (I) Other , followed by (B) Civic Engagement, Interest in Social Justice, Emotional Empathy , (D) Career Pursuits and (F) Using Trans- and Multidisciplinary Approaches .

The Most Significant Change Story cites (F) Using Trans- and Multidisciplinary Approaches as the primary Domain for most stakeholders, followed by (D) Career Pursuits .

Of the defined domains, (B) Civic Engagement, Interest in Social Justice, Emotional Empathy was the third most-cited by stakeholders. The potential power of marrying improved communication and confidence with real interest in communities, equity and inclusion is critical to Science Gallery’s mission. This finding suggests that the Mediator Programme is a key instrument to develop complex sets of skills among the participants. The “Honorable Mention” Story (appendix B ), which was discussed at length as the potential MSC, dealt quite explicitly with increased interest in social justice.

We looked also at the strength of the relationship between domains of change based on the number of stories that shared the same domains. The most commonly cited domain, (G) Confidence / Interpersonal Skills was frequently coupled with other domains, including (B) Civic Engagement / Social Justice , (D) Career Pursuits , (F) Using Transdisciplinary Approaches and (I) “Other” , which as noted above, encompassed various other types of domains. Domain (G) and (I) had the highest number of stories in common, 9, followed by (G) and (B) with 7 stories in common and (B) and (I) with 6 stories. Figure 1 illustrates the strongest relationships across domains.

---

---

5 Reflection and Conclusions

Story 6 resonated with our reviewers and became the story of Most Significant Change — many reasons were cited, but among them the sheer scope of change, the ability to recognise both positive and negative elements in their time at Science Gallery, and the length of the journey were all important contributors.

5.1 Excerpt from Story 6

Story 6 was in many ways an outlier, and we looked to the secondary analysis to give us a better view of the broader picture. Greater confidence in public speaking and improved communication (Domain G) is an important consequence of participation in the Mediator Programme, and the most cited across the change stories. We can recognise this as a strength of the programme, though it is not the sole or expressed goal of the programme. How can we sharpen this skill further, and how can we bring Mediators on a journey that recognises how to take those improved communication skills and use them to even greater advantage?

5.2 Lessons Learned

5.2.1 For the Mediator Programme

Intended Goals. We were delighted to see that one of the internal goals of the Mediator Programme, to increase confidence, was frequently cited in stories.

Likewise, the increased interest in social justice and equity frequently mentioned in stories was deeply heartening.

Unexpected Changes and Areas for Improvement. Given that (I) “Other” was identified as the primary domain twice, the research team is pleased that space was included for stakeholders to identify other, unintended outcomes. The Mediator Programme and its potential outcomes are complex enough that there will always be multiple unanticipated results.

In the non-selected stories, peer groups, peer support and peer learning were addressed frequently. The tacit learning structures within the Mediator team have not been formally explored, but their power as a community of practitioners likely deserves further attention.

The reviewing staff felt that the account chosen as the Story of Most Significant Change addressed some of their own key concerns with the programme, including:

• The perception that there are two, divided staff teams: Mediators, and everyone else
• Going beyond identifying confidence as a primary theme, which reviewers identified as “the bare minimum” a participant should walk away with
• Speaking to the difficulty that permanent gallery staff can have communicating with Mediators, as there are infrequent opportunities for introduction, working together, etc.
• Taking the reviewers through a journey from beginning to end of engagement with the programme
• Coming to an important conclusion: that sometimes one must move on from a project or role, and even though it may be the right conclusion, there may be a sense of loss to contend with.

5.2.2 The Most Significant Change Method

MSC requires a huge amount of time and resourcing. However, the payoff of stories that both confirm some of our intended goals and provide new avenues of exploration makes the method worth the effort. Our experience this time has inspired us to adjust a few elements for future:

• Invite more stakeholders that represent groups who are not staff (Board Members, funders, etc.)
• If possible, procure more interviews — the stories are so rich, there is something to learn from each one, and we wonder what is missed from those stories that were not told

A Appendix A. The story chosen as the Story of Most Significant Change

*Please note that the story below is written by the respondent, in their own words, and has not been edited for content or grammar.

STORY No. 6

What has been the most significant change in your life, over the course of the last year, due to participation in the Mediator Programme?

There are a number of reasons why I was interested in mediating and why I think the role suited me so well. I had always been interested in both the arts and sciences growing up and so when it came to deciding what I wanted to do in college, it was a fairly difficult decision. To me, the two concepts of art and science didn’t mesh very well together, and so there was a sense of loss in choosing one over the other. In the end I settled on doing a science course and while I was happy with that decision, I felt as though I were giving up an important aspect of my life, having been told by my mam and various others that “art could be kept up as a hobby.”

The most significant change that I had while working as a mediator was the discovery that there is a means to merge the areas of art and science, and that the two share a common element — a desire to discover, to broaden our understanding and to enrichen our experiences. This was the start of a long process of learning with the gallery, one which helped me develop into who I am today. It opened up the fantastic world of science communication and education to me, and introduced new ideas such as STE[Arts]M engagement to my life. Allowing me to learn about these and to nurture my creative side has helped me to become an incredibly well rounded person who now has the skills to do what they want in life. SciComm and education in particular are areas which I hope to return to later on in my career and which I will continue to advocate for.

This past year, I left the Science Gallery. There were a number of reasons motivating this, the foremost being that college was becoming far too stressful for me to continue. Other reasons included, unfortunately, a lack of respect from the full time staff for the mediator team and a feeling of inequality between them. Having dealt with tensions for quite some time, I thought it best to take a step back.

There was, again, a huge sense of loss here. The Science Gallery had become a huge part of my identity, and the people who worked there had become some of my closest friends. These past months since having left have been difficult in terms of not having an outlet for creativity and for learning new ideas that I could only ever engage with through the Science Gallery, and although I needed to focus more on my academics this year, and while that certainly has been beneficial to my studies, I feel as though my studies have suffered in other regards. There was a lack of curiosity in my approach to things, which is something that I always found in myself while working for the Gallery. Even still, I’ll carry the lessons that I learned there throughout my career.

B Appendix B. Honourable Mention Story

*Please note that the story below is written by the respondent, in their own words, and has not been edited for content or grammar.

STORY No. 11

What has been the most significant change in your life, over the course of the last year, due to participation in the Mediator Programme?

I had a realisation about what kind of public came to Science Gallery. Before working for Science Gallery, I didn’t know that 15–25 year olds were the specific audience, I had always felt perfectly at ease there! I realised who was coming on their own terms to see exhibitions, and I could see a pattern of well-to-do or middle class tourists and Dubliners. There was an awareness for me that the people coming, despite the gallery being free, were mostly middle class. Sometimes tours brought other kinds of people in. I realised that through other people’s eyes that the gallery could be intimidating, or seem strange. I recognised that there was much more to do to make the space democratic. To a certain extent, it made me more sensitive to who was participating in programmes, and it drew me to programmes that cater to kids from disadvantaged areas. During my degree, I did much more reading about equity, much more excited to do work with DEIS Schools, with MakeShop. It’s made me much more curious — particularly about individuals. It’s not just DEIS schools or the city centre, but even people from more rural areas — I could see the blank faces sometimes, the things that seemed strange to them. In personal terms, it’s really about curiosity, getting to know and understand people, how they see things.

References

Bandelli, A. and Zeijden, W. van der (2015). Studiolab: what has been learned. Ed. by A. Bandelli and S. Duensing. ISBN: 978-0-9926110-5-7. Dublin, Ireland: Science Gallery — Trinity College.

Brunswick, I. (2017). ‘Genesis of the Science Gallery’. In: Little Country, Big Talk: Science Communication in Ireland. Ed. by B. Trench, P. Murphy and D. Fahy. London, U.K.: Pantaneto Press, pp. 159–175. URL: https://ianbrunswick.wordpress.com/2017/06/02/229/ .

Davies, R. and Dart, J. (2005). The “Most Significant Change” Technique — A Guide to Its Use . URL: http://www.betterevaluation.org/en/resources/guides/most_significant_change .

Fanning, M. (2015). SECRET Evaluation. Dublin, Ireland: Behaviour&Attitudes.

Hidayati, N. (2018). The Most Significant Change Technique . URL: https://www.resultsinhealth.org/images/publications/Factsheets/Factsheet_ResultsinHealth_-_The_Most_Significant_Change_Technique_v2.pdf .

Mordan, C. and Hughes, S. (2015). HOME\SICK Evaluation. Dublin, Ireland: Celsius Research Cluster, Dublin City College.

Science Gallery Dublin (2016). Annual Review 2015. Dublin, Ireland: Science Gallery — Trinity College. URL: https://dublin.sciencegallery.com/files/yearly/SG_AR_15.pdf .

Authors

Katrina Enros has worked at the intersection of third-level education and the cultural sector for over a decade, most frequently on public engagement projects within universities. She completed her MA in Arts Administration and Policy (School of the Art Institute of Chicago, 2008) and an Hons. BA in Art History and Sociology (University of Toronto, 2004), as well as additional advanced diplomas in Learning & Development (CIPD, National University of Ireland, 2018) and Business Management (University of Toronto, 2014). Katrina has worked on cultural and public engagement projects with institutions across the globe, including Queen’s University, the University of Toronto and Trinity College Dublin. Katrina Enros is Manager for Global Engagement and Insights at Science Gallery International. E-mail: katrina.enros@sciencegallery.com .

Andrea Bandelli is a scholar in science communication and the executive director of Science Gallery International. He has developed, managed, evaluated and directed several international projects on science, art, democracy and public participation in collaboration with Ecsite, the European network of science centers and museums. He has a Ph.D in Science Communication from the Vrije Universiteit in Amsterdam on scientific citizenship and the role of public participation in science centers, and is a Cultural Leader at the World Economic Forum. E-mail: andrea.bandelli@sciencegallery.com .

Endnotes