1 Introduction

Participation in academic research can take many forms. Therefore, different typologies for members of the public who participate in academic research have been proposed. Often cited typologies are those by Bonney et al. [ 2009 ]. and Haklay [ 2013 ], who define different levels of participation, where the most comprehensive form of public participation in research is called co-creation or extreme citizen science. Co-creation as an approach to public participation in scholarly research aims at integrating citizens in all steps and decisions in the entire research process. Thus, it plays an important role in the democratization of research [Irwin, 1995 ].

An objective of the linguistic citizen science project “On everyone’s mind and lips — German in Austria” (abbreviated as IamDiÖ) is therefore to test the co-creation approach [Bonney et al., 2009 ] in the citizen humanities [Heinisch et al., 2021 ; Dunn and Hedges, 2017 ]. The project aims at involving volunteers in the entire research process.

The motivation behind this linguistic project is the fact that language is ubiquitous. Moreover, language and language varieties are an emotional and political topic since they are not only related to a person’s identity [Kummer, 1990 ; Thim-Mabrey, 2003 ] but also language ideologies [Woolard and Schieffelin, 1994 ]. Although language is omnipresent and IamDiÖ offers various types of tasks and engagement, recruiting (and retaining) participants proved to be challenging.

IamDiÖ combines different citizen science approaches and allows participants to select from a variety of different tasks characterized by different topics and different autonomy.

First, IamDiÖ engages in co-creation with a format entitled Question of the Month. Here, citizens can raise and answer questions related to the topic of German language in Austria supported by researchers.

Second, in a linguistic treasure hunt the project focuses on data collection and data analysis. Citizens take, save and tag pictures of written information in the public space with an app, thus contributing to the study of the linguistic landscape in Austria [Seltmann and Heinisch, 2018 ]. The linguistic treasure hunts are considered collaborative projects, because citizens participate in more than one research step, namely data collection and data analysis.

Third, it focuses on data generation through a meme contest, where citizens can combine text written in dialect with pictures that can be associated with Austria. However, the generation of memes may not be regarded as citizen science per se [Eitzel et al., 2017 ; Heigl et al., 2019 ].

Fourth, IamDiÖ invites participants to create an online dictionary for the language varieties used in Austria. According to lexicographical principles, participants create dictionary entries encompassing any form of standard or non-standard language, including standard German, dialects, youth language and languages for specific purposes.

2 Method

This study is based on a previous study [Heinisch, 2020 ] comparing the co-created approach (Question of the Month) and the collaborative approach (linguistic treasure hunts) in the IamDiÖ project. While the previous study focused on the comparison of the ‘success’ of these two approaches by introducing very general criteria for comparison, such as the number of participants, the quantity and quality of their contributions and the contribution to knowledge gain and academic progress, the present paper investigates the reasons for the difficulties in recruiting and retaining the participants in the Question of the Month task. The Question of the Month in IamDiÖ could not be realized in the way initially proposed. Therefore, the project could also not collect any personal data from the participants since the majority of persons participated only once in a setting that did not allow for the collection of personal data. Therefore, participant surveys are not an option for investigating the reasons why the initial project design could not be implemented as planned.

The literature review on failures in citizen science revealed only a few studies that explicitly used the word ‘failure’ to admit that citizen science did not work out as planned. One study piloted citizen science and showed that despite the high engagement and the large amounts of data collected, the participants did not achieve the educational goals the authors had wished for [Druschke and Seltzer, 2012 ]. Another study discusses different sources of failures in citizen science projects, thus paving the way for an error culture in the field of citizen science [Westreicher et al., 2021 ].

Although different success factors in citizen science projects [Freitag and Pfeffer, 2013 ] have been identified and success matrixes [Cox et al., 2015 ] have been proposed, not all of these categories are suitable for all project types. Moreover, projects may have different objectives making a general definition of success and comparisons difficult. However, from a methodological point of view, a comparative analysis requires the focus on certain criteria. Therefore, evaluation of citizen science projects receives growing attention [Kieslinger et al., 2018 ; MICS Project, 2020 ]. The evaluation of citizen science projects basically covers the three dimensions of participatory science: 1) scientific impact, 2) empowerment and learning of participants and 3) impact on society. These three dimensions can be assessed on two levels: process and feasibility, on the one hand, and outcome and impact, on the other [Kieslinger et al., 2018 , p. 81]. This framework is used in the current analysis of the Question of the Month strand of the IamDiÖ project.

3 Results and discussion

An extract of the initial research proposal for IamDiÖ states “The citizen science project “On everyone’s mind and lips — German in Austria” uses the general public’s knowledge, concerns and questions about the multi-dimensional field of German in Austria to address societal challenges. Together with citizens who are actual speakers of dialects and German varieties in Austria we engage in extreme citizen science, i.e. we design the entire research process (from knowledge gaining to dissemination) together with citizens. We encourage citizens to reflect on their language use or linguistic attitudes and to participate in all stages of the research process. Thus, citizens become part of finding new solutions to academic issues.” This statement shows that the project design is rather open and the target audience broad. The target audience is addressed later in the proposal in more detail, including schoolteachers, school students, university teachers, university students as well as associations in the field of language, cultural heritage, etc. as well as the means of communication used to reach them. The engagement strategies, especially for the Question of the Month, should include gamification aspects. The statement quoted from the project proposal also highlights the focus on co-creation and the aim of tackling societal challenges emerging from the use of language and of its varieties. The means to achieve this end is the Question of the Month.

3.1 Question of the Month

The Question of the Month should motivate citizens to ask (and answer) a research question about the topic of German in Austria. This can be very general questions, such as “Do dialects disappear?” or “What is the origin of the word ‘Oida’?” that can be become more specific during the research process. IamDiÖ set up a project website, where all the opportunities to participate in IamDiÖ, including the Question of the Month were explained in detail. When designing the website, the usability for a non-linguistic audience was of the utmost importance. Therefore, both the design was classical in the beginning and the language used on the website was geared towards a non-academic audience. The use of academic terms was reduced to the absolutely necessary minimum and if any terms from the domain of linguistics were used, an explanation was provided. Additionally, a glossary explaining the most important and/or frequent linguistic terms in an intelligible way was prepared.

On the project website, persons can submit their Question of the Month via a simple and GDPR-compliant web form, only asking for the name, an e-mail address and the question in which the person is interested. However, initially, persons had to write an e-mail to the project’s common e-mail address. The submission of the Question of the Month was piloted with students from associated universities who took part in university courses in which they either had to find a research question anyway, e.g. during seminars or in which the submission of a research question on the topic of German in Austria was subject to bonus points. Of course, the number of questions submitted by students for whom this was a mandatory part of the course was higher than for the bonus point group. However, while none of the students from the group for whom the raising of questions was a mandatory part actually submitted an answer to their questions (since only the question was mandatory), one student from the bonus point group was willing to undergo the entire research process and to find an answer to her question according to a predefined structure. This predefined structure reflects all the steps in the research process. Although this structure should ensure academic rigor and provide an insight into a scholarly approach to a question, the main aim was not to produce absolutely new research findings, but to provide a means of reflection. This structure required the participants to answer certain questions, e.g. What was the research question?, Why did you choose this research question? How did you answer this research question?, Which method did you select and use to collect and analyze your data?, What is the result?, What is the conclusion? What were the challenges?, etc. The initial idea was that the persons undergoing the entire research process are supported by the scholars in the project who help the participants to select a method, provide relevant literature and guide them through the various steps. However, those persons having found the IamDiÖ website and the relevant form to submit their question about German in Austria, were rather interested in receiving an answer to their question from the experts in the project than in conducting research on their own. A similar observation was made during science communication festivals that became the major means of recruiting participants for the Question of the Month. There, the visitors raised many questions but when they were invited to find an answer, they were not willing to contribute their time and effort to answer their own research questions.

The reasons that came to the fore when enquiring why the participants refused to go through the entire research process where that it participation seemed to be too time-consuming, that academic research seems daunting or that the scholars are the experts (and should either already know the answer or provide the participants with the answer). Moreover, the discussions showed the ignorance of the relevance of fundamental research since the participants were rather interested in the application of this type of research. Moreover, the persons acknowledged that they do not have the competences to answer a (or their) research question. However, these are only the reasons mentioned during the personal dialogue with the participants and they are not representative for all. Co-creation, thus, cannot be imposed by academic researchers. Through two science communication events, the Question of the Month could reach a high number of people in the initial research phase. A high number of volunteers specified research questions according to their interests. This showed also the societal relevance of the topic of language, ranging from the variation of language, language perception to language attitudes and language contact. The participants’ questions ranged from the use of ‘correct’ German, youth language, the influence of other German varieties and languages of migrants on the Austrian variety, etc. The discussions also revealed language myths that seem to prevail and that cannot be confirmed by research findings.

On the project’s social media channels, including Facebook and Twitter, the community was motivated to raise and submit their questions. However, compared to the face-to-face events, only a small number of questions were submitted via these channels. What was far more successful on social media was the voting for the Question of the Month. After having collected hundreds of questions during science communication festivals, these questions were clustered according to (research) topics, e.g. language contact, etymology or language history. Every month, two questions from the same cluster were selected from the project team and put to the vote on social media, where the community could select their favorite question. As mentioned before, the initial project design that intended that the participants answer their own questions could not be realized. Therefore, not the participants who raised the question answered it but the researchers. So, the Question of the Month became rather a science communication exercise than co-creation. The question that received the majority of likes from the community was then forwarded to a scholar working in the relevant field who was requested to answer this question according to the pre-defined structure introduced above in a comprehensible language. Then, members of the IamDiÖ project team reviewed the academic’s answer and suggested revisions to make the text more legible and shorter. Every Question of the Month was also embellished with a hand-drawn picture illustrating the topic covered. These answers in the form of blog posts on the IamDiÖ website received attention. In total, the Question of the Month was able to reach a rather broad audience measured in the number of questions raised, the number of persons voting on the Question of the Month on social media and the number of page views on the website containing the answered questions [Heinisch, 2020 ]. Moreover, journalists contacted the project team to conduct interviews with the researchers on the relevant topic. While the Question of the Month increased the visibility of the project and its researchers, it could not fulfil its original purpose and could not follow the co-creation approach. This demonstrates that the initial co-creation approach to the Question of the Month, i.e. that citizens raise their questions and find an answer on their own supported by researchers, was not easy to adopt. This required a change in strategy to meet the project’s overall objectives.

3.2 Assessment

The recruitment of participants is a recognized challenge in citizen science [Chu, Leonard and Stevenson, 2012 ; Crall et al., 2017 ; West and Pateman, 2016 ]. For the Question of the Month, the initial attempt to recruit participants via social media and allow them to enter their question in a web form could not be implemented as planned. However, face-to-face communication during science communication festivals helped to collect more than 500 potential research questions addressing the topic of German in Austria. Although several hundred participants raised questions through these recruitment measures and during university courses, only one participant answered her research question. In comparison to other citizen science projects in the field of social sciences and humanities that also ask the public to raise research questions and employ an online strategy [Ludwig Boltzmann Gesellschaft, 2019 ], IamDiÖ could gather more questions through personal dialogues than via its website.

The assessment of the Question of the Month according to the framework provided by Kieslinger et al. [ 2018 ] along the three dimensions ‘science’, ‘participants’ and ‘socio-ecological and economic dimension’ showed that the Question of the Month might be a format that is too open, for both the scientific process and the related feasibility as well as the target group alignment. While, in the scientific dimension, the project’s academic objective is to test the co-creation approach in citizen linguistics, the evaluation and adaptation element may require further improvement. The Question of the Month format, in its initial conception, adheres to the principle of joint knowledge creation in citizen science. However, although a plan B could be found for the Question of the Month, the initial co-creation approach could not be implemented. Data quality was not an issue according to the project design and a simple template was provided for the participants to follow the steps in an academic process and reflect on them. The data, i.e. the questions (and the related answers) are published on the website and are freely accessible. If participants answer their questions on their own, they also gain visibility through the publication of the results in the form of a blog post on the website. Additionally, synergies with other research projects and citizen science projects in the field of linguistics may facilitate the recruitment and retention of participants. A few academic articles were published on the Question of the Month as part of the project and IamDiÖ can rely on a research infrastructure provided by a larger project. This also means that the Question of the Month feeds into a larger virtual research environment making the outcomes more sustainable and FAIR (findable, accessible, interoperable and re-usable). While the Question of the Month can provide only limited new knowledge resources required for the advancement of science, it created a new means of science communication allowing interested persons to submit their question about a topic and researchers providing an academically sound answer to it in an intelligible form and accessible format. This can contribute to a better understanding of science in society. Whether the project contributed to institutional or structural changes would require further investigation.

Regarding the participant dimension, the target group alignment seems to be a major issue for the Question of the Month. Although the Question of the Month had an involvement plan that considered different target groups, it could not reach all the intended target groups as intended. Additionally, the training measures were not geared towards different groups but rather covered a superficial layer that may meet the needs of different target groups. Although the potential for the degree of involvement was high since participants could have participated in all project phases (except for the project proposal phase), it could not yield the expected success. A reason for this may be the vagueness of the project design itself and/or the clear communication of the (intended) project results. Another aspect that would need improvement according to the evaluation criteria is the regular feedback to participants. While the social media community receives constant information and feedback about the project, those who do not follow the project on social media are excluded from the information flow because it is unlikely that the participants visit the project website on a regular basis to obtain news about the project. The definition of the research questions together with the community and the empowerment of participants are at the core of participatory (action) research [von Unger, 2014 ] and therefore, participatory action research may also provide the framework for dealing with this openness and the resulting vagueness in the Question of the Month that makes the tangible outcomes and the benefits for the participants or society at large, difficult to grasp. Moreover, the facilitation and communication of the Question of the Month may need other approaches. On the one hand, reaching out to the associations in the cultural heritage domain or dialect associations (once again) could help reach a target audience that is willing to engage deeply in the research topic and the project. As a first step, it may be necessary to build a network. This can take the form of following the related associations or organizations on social media and liking and sharing their posts so that they become aware of the project. Additionally, they can be addressed individually by e-mail to inform them about the project and ask them how the project may contribute to their activities. Another option is that IamDiÖ project members volunteer to become a so-called school ambassador, which would allow the project to reach out to schools and to present the project at schools, and hopefully, also attract participants. The evaluation criteria on the outcome and impact level, namely knowledge and scientific literacy, behavior and ownership as well as motivation and engagement [Kieslinger et al., 2018 ] cannot be assessed at this moment since IamDiÖ does not have personal or survey data from the majority of the participants in the Question of the Month. While it is speculative that participants increased their understanding of science or changed their attitude towards science on the basis of one brief discussion during a science communication festival, it is likely that the person who had answered her research question, may have increased her academic literacy.

With regard to the third dimension, namely the socio-ecological and economic dimension, the Question of the Month may also lack target group alignment, as explained above. Although the project has a communication plan, it could be more specific with regards to a more detailed communication strategy for different target groups. Additionally, it could make further use of traditional media, such as newspapers, radio or television to increase the visibility of the project and invite participants to the project. However, since media campaigns usually result in a huge number of enquiries by members of the public, the days of the media campaign would require the availability of the staff to cope with all the enquiries and do justice to the interest of the members of the public. The Question of the Month itself is an innovative means of science communication, as addressed above. Additionally, the project also developed other means of science communication and engagement, such as comics, in which people can fill speech bubbles in any language and language variety they want. The Question of the Month is based on a two-way communication. However, as explained above, it became rather a means of science communication (from the experts to the members of the public). On the level of outcome and impact, the project topic being language does not align with the ecological impact, but it may have societal impact and wider innovation potential. While the Question of the Month was intended to contribute to the collective capacity of the participants in achieving common goals, e.g. finding an answer to a research question, it was hard to implement for the reasons already mentioned. The Question of the Month does also not stimulate political participation and does not have an impact on policy processes. Regarding the wider innovation potential, the Question of the Month rather uses social media or blog posts to engage with the public. However, the virtual research environment that is created by the associated project, would fulfil this criterium. Currently, the Question of the Month addresses sustainability and social innovation only marginally and does not address the factor of economic potential and market opportunities directly. Nevertheless, the participants in the Question of the Month raised the question of the application of the project results several times. One of the participants would like to use the dialectal data for speech recognition systems.

4 Conclusion

To conclude, although citizens are interested in the topic of (German) language use, which is documented by the high number of questions raised, the majority of the participants were not willing to contribute their time and effort to answering their own research questions. The implementation of a co-creation approach requires thorough preparation, including community building and the establishment of partnerships to align the project objectives with the target group and any wider societal impact and innovation potential.


This research received funding from Austrian Science Fund (FWF): TCS 57G.


Bonney, R., Ballard, H., Jordan, R., McCallie, E., Phillips, T., Shirk, J. and Wilderman, C. C. (2009). Public Participation in Scientific Research: Defining the Field and Assessing Its Potential for Informal Science Education . A CAISE Inquiry Group Report. Washington, D.C., U.S.A.: Center for Advancement of Informal Science Education (CAISE). URL: http://www.informalscience.org/public-participation-scientific-research-defining-field-and-assessing-its-potential-informal-science .

Chu, M., Leonard, P. and Stevenson, F. (2012). ‘Growing the Base for Citizen Science: Recruiting and Engaging Participants’. In: Citizen Science: Public Participation in Environmental Research. Ed. by J. L. Dickinson and R. Bonney. Ithaca, NY, U.S.A.: Cornell University Press, pp. 69–81. URL: http://www.cornellpress.cornell.edu/book/?GCOI=80140100107290 .

Cox, J., Oh, E. Y., Simmons, B., Lintott, C. L., Masters, K. L., Greenhill, A., Graham, G. and Holmes, K. (2015). ‘Defining and Measuring Success in Online Citizen Science: A Case Study of Zooniverse Projects’. Computing in Science & Engineering 17 (4), pp. 28–41. https://doi.org/10.1109/MCSE.2015.65 .

Crall, A., Kosmala, M., Cheng, R., Brier, J., Cavalier, D., Henderson, S. and Richardson, A. (2017). ‘Volunteer recruitment and retention in online citizen science projects using marketing strategies: lessons from Season Spotter’. JCOM 16 (01), A1. URL: https://jcom.sissa.it/archive/16/01/JCOM_1601_2017_A01 .

Druschke, C. G. and Seltzer, C. E. (2012). ‘Failures of engagement: lessons learned from a citizen science pilot study’. Applied Environmental Education and Communication 11, pp. 178–188. https://doi.org/10.1080/1533015X.2012.777224 .

Dunn, S. and Hedges, M. (2017). ‘From the wisdom of crowds to going viral: the creation and transmission of knowledge in the citizen humanities’. In: Citizen inquiry: synthesising science and inquiry learning. Ed. by C. Herodotou, M. Sharples and E. Scanlon. Abingdon, Oxon, U.K. and New York, NY, U.S.A.: Routledge, pp. 25–41. https://doi.org/10.4324/9781315458618-3 .

Eitzel, M. V., Cappadonna, J. L., Santos-Lang, C., Duerr, R. E., Virapongse, A., West, S. E., Kyba, C. C. M., Bowser, A., Cooper, C. B., Sforzi, A., Metcalfe, A. N., Harris, E. S., Thiel, M., Haklay, M., Ponciano, L., Roche, J., Ceccaroni, L., Shilling, F. M., Dörler, D., Heigl, F., Kiessling, T., Davis, B. Y. and Jiang, Q. (2017). ‘Citizen science terminology matters: exploring key terms’. Citizen Science: Theory and Practice 2 (1), pp. 1–20. https://doi.org/10.5334/cstp.96 .

Freitag, A. and Pfeffer, M. J. (2013). ‘Process, not product: investigating recommendations for improving citizen science “success”’. PLoS ONE 8 (5), e64079. https://doi.org/10.1371/journal.pone.0064079 .

Haklay, M. (2013). ‘Citizen Science and Volunteered Geographic Information: Overview and Typology of Participation’. In: Crowdsourcing Geographic Knowledge: Volunteered Geographic Information (VGI) in Theory and Practice. Ed. by D. Sui, S. Elwood and M. Goodchild. Berlin, Germany: Springer, pp. 105–122. https://doi.org/10.1007/978-94-007-4587-2_7 .

Heigl, F., Kieslinger, B., Paul, K. T., Uhlik, J. and Dörler, D. (2019). ‘Opinion: toward an international definition of citizen science’. Proceedings of the National Academy of Sciences 116 (17), pp. 8089–8092. https://doi.org/10.1073/pnas.1903393116 .

Heinisch, B. (2020). ‘Comparison of co-created and collaborative approaches to citizen science adopted by the citizen linguistics project ‘on everyone’s mind and lips — German in Austria’’. In: Proceedings of 5th Austrian Citizen Science Conference 2019 — PoS(ACSC2019) (Obergurgl, Austria, 26th–28th June 2019). https://doi.org/10.22323/1.366.0003 .

Heinisch, B., Oswald, K., Weißpflug, M., Shuttleworth, S. and Belknap, G. (2021). ‘Citizen humanities’. In: The science of citizen science. Ed. by K. Vohland, A. Land-Zandstra, L. Ceccaroni, R. Lemmens, J. Perelló, M. Ponti, R. Samson and K. Wagenknecht. Cham, Switzerland: Springer International Publishing, pp. 97–118. https://doi.org/10.1007/978-3-030-58278-4_6 .

Irwin, A. (1995). Citizen science: a study of people, expertise and sustainable development. 1st ed. London, U.K.: Routledge. ISBN: 978-04-1513-010-3.

Kieslinger, B., Schäfer, T., Heigl, F., Dörler, D., Richter, A. and Bonn, A. (2018). ‘Evaluating citizen science: towards an open framework’. In: Citizen science: innovation in open science, society and policy. Ed. by S. Hecker, M. Haklay, A. Bowser, Z. Makuch, J. Vogel and A. Bonn. London, U.K.: UCL Press, pp. 81–95.

Kummer, W. (1990). ‘Sprache und kulturelle Identität’. In: Ethnizität: Wissenschaft und Minderheiten. Ed. by E. J. Dittrich and F.-O. Radtke. Opladen, Germany: Westdeutscher Verlag, pp. 265–275. https://doi.org/10.1007/978-3-322-83695-3_13 .

Ludwig Boltzmann Gesellschaft (2019). Reden Sie mit! URL: https://tell-us.online/de#faq-was-ist-das-ziel-von-reden-sie-mit .

MICS Project (2020). MICS project factsheet . https://doi.org/10.5281/ZENODO.3981066 .

Seltmann, M. E.-H. and Heinisch, B. (2018). ‘How to speak German in Austria. Collaboration between two linguistic citizen science projects: “On everyone’s mind and lips — German in Austria” and “Lingscape” found each other’. In: Austrian Citizen Science Conference 2018 . Frontiers Media SA, pp. 82–85. URL: https://klf.univie.ac.at/fileadmin/user_upload/p_klf/Proceedings_OECSK2018.pdf .

Thim-Mabrey, C. (2003). ‘Sprachidentität — Identität durch Sprache. Ein Problemaufriss aus sprachwissenschaftlicher Sicht’. In: Tübinger Beiträge zur Linguistik: vol. 465. Sprachidentität — Identität durch Sprache. Ed. by N. Janich and C. Thim-Mabrey. Tübingen, Germany: Narr, pp. 1–18.

von Unger, H. (2014). Partizipative Forschung: Einführung in die Forschungspraxis. Wiesbaden, Germany: Springer VS. https://doi.org/10.1007/978-3-658-01290-8 .

West, S. and Pateman, R. (2016). ‘Recruiting and Retaining Participants in Citizen Science: What Can Be Learned from the Volunteering Literature?’ Citizen Science: Theory and Practice 1 (2), p. 15. https://doi.org/10.5334/cstp.8 .

Westreicher, F., Cieslinski, M., Ernst, M., Frigerio, D., Heinisch, B., Hübner, T. and Rüdisser, J. (2021). ‘Recognizing failures in citizen science projects: lessons learned’. In: Proceedings of Austrian Citizen Science Conference 2020 (ÖCSK 2020) — PoS(ACSC2020) (Vienna, Austria, 14th–16th September 2020). Trieste, Italy: Sissa Medialab. https://doi.org/10.22323/1.393.0007 .

Woolard, K. A. and Schieffelin, B. B. (1994). ‘Language ideology’. Annual Review of Anthropology 23 (1), pp. 55–82. https://doi.org/10.1146/annurev.an.23.100194.000415 .


Barbara Heinisch is a researcher and lecturer with a focus on translation studies at the University of Vienna. She graduated in technical translation and conference interpreting. Her research interests include specialised translation, usability, accessibility, localisation, technical documentation, terminology and citizen science. She is also running a citizen humanities project in linguistics. E-mail: barbara.heinisch@univie.ac.at .