Are we on the right path? Insights from Brazilian universities on monitoring and evaluation of Public Communication of Science and Technology in the digital environment

1 Introduction

Technology enables real-time monitoring of conversational interactions on digital and social networks, fostering dialogue channels. However, little is known about monitoring channels of communication that emphasize the relationship between science and society.

In this article, we present findings from a research study in which public communication managers from Brazilian federal universities explored a model for assessing the effectiveness of Public Communication of Science and Technology (PCST) in the context of digital activities.¹ PCST is defined by Burns, O’Connor and Stocklmayer [2003] as activities aimed at communicating scientific and technological information to the public in an accessible and engaging manner. The research aims to explore how these managers monitor and evaluate their activities, identifying suitable indicators for this purpose.

In 2022, an article in the Journal of Science Communication Latin America (JCOMAL) Aguiar Pereira and Salles-Filho [2022] introduced a monitoring and evaluation model of PCST based on the Theory of Change (ToC) and ideal types: Informative, Public Engagement, and Participatory. This model delineates a logical chain explaining communication between academia and society, identifying theoretical assumptions and measurement indicators.

Using this model, we asked communication managers to assess the relevance and application of 26 proposed indicators [Aguiar Pereira & Salles-Filho, 2022]. Through our research, we present a profile of Brazilian federal universities regarding PCST, focusing on their monitoring and evaluation of social interactions. In this article, our goal is to expand theoretical and empirical knowledge by exploring a topic often overlooked in scientific communication: the specific aspects of monitoring and evaluation, as well as organizational communication within universities. Our approach involves researching communication professionals in Brazilian universities, a region of Latin America that advances in PCST studies.

2 Objective

The study aims to elucidate how universities monitor and evaluate PCST, offering insights into their practices, typical metrics, and evaluation challenges. By posing questions regarding the relevance and applicability of the 26 indicators in both short and long-term scenarios, as well as the perceived risks involved, we seek to provide a comprehensive understanding of the PCST evaluation landscape, with a specific focus on its application within the digital environment.

3 Literature review

3.1 From models to practice

Despite efforts to increase public engagement [Massarani, Moreira & Lewenstein, 2017], measuring and understanding the significance of this engagement for public scientific communication remains a challenge [Pellegrini, 2021]. Terms such as outreach, engagement, participation, and knowledge ownership are common, but the scientific community needs to define how this monitoring and evaluation can be achieved through logical models that incorporate objectives and indicators providing evidence of this performance.

While bibliographic indicators dominate academia [Wouters, Zahedi & Costas, 2019], research on monitoring and evaluating social communication is recent and sporadic [Pellegrini, 2014; Castelfranchi & Fazio, 2021]. Most studies that analyze metrics related to science-society interaction aim to understand the public perception of science and the impact of activities on the understanding of science and technology (S&T), especially those in museums and festivals [Grand & Sardo, 2017].

Carneiro, Resce and Sapkota [2022] advocated for a streamlined framework to monitor scientific production progress, emphasizing a growing body of literature that underscores the web and social media as platforms fostering broad public discourse, information dissemination, and engagement across various topics. They underscored the importance of selecting indicators for this purpose, highlighting the need to incorporates PCST in the evaluation of scientific production.

Engagement lacking evaluation holds limited value [Watermeyer, 2012]. Enhancing engagement in PCST is achievable when adopting evidence-based communication choices, as advocated by Besley, Newman, Dudo and Tiffany [2020] and Jensen and Gerber [2020]. Engagement alone does not ensure an understanding of how the public receives, comprehends, or applies communicated information. Different forms of engagement and participation depend on institutional, local, and cultural contexts. This approach goes beyond mere accessibility of scientific knowledge, aiming to understand its actual impact on the public.

Despite the infancy of data-driven strategic communication [Wiencierz & Röttger, 2019; Economou, Luck & Bartlett, 2023], Volk and Zerfass [2021] note the increasing utilization and importance of social media analytics tools.² Social media analytics tools play a crucial role by providing insights into various aspects of social media performance. These tools analyze metrics such as engagement rates, audience demographics, reach, impressions, and sentiment analysis,³ among others. However, they highlight a lack of knowledge and training in using these tools, with about two-thirds of professionals acquiring skills through practical experience. Fitzpatrick and Weissman [2021] acknowledge the importance of monitoring and measurement tools but cite challenges in establishing systematic evaluation models. A deficit exists in analytical skills and technical knowledge needed to analyze large datasets and make data-driven decisions [Macnamara, Lwin, Hung-Baesecke & Zerfass, 2021; Economou et al., 2023].

However, the challenge extends beyond local perceptions. Jensen [2014] emphasizes that low-quality evaluation methods are routinely used even in well-resourced scientific communication institutions in developed countries. Such practices lead to questionable data, misleading conclusions, and reduced effectiveness of scientific communication. Scientific communication institutions often uncritically consume and produce evaluation research, assuming that measuring complex outcomes is straightforward.

Pellegrini [2014] holds a similar view, viewing evaluation planning and design in science and technology communication as intricate tasks involving various factors and utilizing both quantitative and qualitative methods. Evaluations also vary significantly between knowledge transmission activities and those aiming to facilitate dialogue among social actors [Pellegrini, 2021].

In recent years, academic and research organizations, including universities, have expanded and professionalized communication efforts, diversifying strategies [Schäfer & Fähnrich, 2020]. They have intensified media relationships and aimed to safeguard reputations by allocating more resources to institutional communication sectors. Furthermore, scientists and communicators, especially in research centers and universities, have moved beyond questioning their roles to embrace this integration as a synergistic alliance [Calcagnini & Xanthoudaki, 2016; Trench, 2017]. They now perceive public scientific communication not just as an isolated responsibility of institutional communication departments, but as complementary to the research process, understanding that effective communication increases public engagement and promotes broader societal impact.

Public science communication has professionalized, transitioning from amateurism to a more strategic approach [Trench, 2017]. Universities encounter challenges in the evolving digital media landscape and academia’s transformation, necessitating communication sectors to expand and diversify practices to reach broader and more diverse audiences [Davies & Horst, 2016; Weingart & Joubert, 2019; Fürst, Volk, Schäfer, Vogler & Sörensen, 2022]. Moreover, communication must extend beyond presenting facts to establishing emotional connections between scientists and the public [Joubert, Davis & Metcalfe, 2019].

Cancino et al. [2021] propose understanding the institutional format of universities based on their complexity, characterized by diverse actor networks and sociotechnical contexts. In a dissertation on communication policies in Brazilian federal universities, Martins [2021] observed that only 30% of the universities in the sample ($N=34$) have organizational communication policies, which include guidelines for evaluating results and societal impacts. Having a communication policy aligns with the idea advocated by the NRC [2016, p. 69], as evaluating a project requires clearly defined objectives and expected outcomes.

Entradas et al. [2020] conducted a transnational study involving 2,030 research institutes in universities and scientific organizations across eight countries, including Brazil. They distinguish between three different formats of communication: public events-making (hereafter referred to as public events), traditional news media access (traditional media), and the use of new media channels (new media).⁴ And they found traditional media formats prevailing, complemented by social media. However, they revealed social media, particularly Facebook, is popular among Brazilian institutes. Despite this, Brazil still lags in achieving a more dialogic and relational communication approach [Massarani, 2012, 2022]. Public communication within Brazilian universities mostly follows a linear and informative model of PCST aligned with the dissemination model [Bastos, 2020].

Similarly, Barba, Castillo and Massarani [2019] identified limitations in organizational structures and constrained resources impacting the efficacy of science communication to broader audiences. These limitations involve the lack of comprehensive documentation such as reports, and the necessity for robust impact assessment mechanisms. Gascoigne and Schiele [2020] aimed to assess the practical application. Using data from 14 indicators, they noted the accelerated evolution of PCST models overall despite notable country differences.

While many studies explore engagement goals with society, few link communication practices to professional structure [Weiner et al., 2021; Besley et al., 2020; Jensen & Gerber, 2020; Olesk et al., 2021]. Despite a growing academic debate, empirical evidence is needed to determine if communication from higher education institutions prioritizes social objectives like knowledge dissemination and dialogue or organizational goals and self-promotion [Entradas et al., 2020; Fürst et al., 2022].

Metcalfe [2019] proposed reflecting on the objectives and nature of engagement activities concerning PCST models. She compared 515 activities with the characteristics of the three dominant models: deficit, dialogue, and participation. The Australian study showed that most activities combined objectives from the deficit and dialogue models.

Selecting indicators to evaluate and compare university communication is challenging. De Filippo, Benayas, Peña and Sánchez [2020] opted for altmetrics, using Altmetric.com mentions to assess science and research presence in Spanish universities, aiming to transcend scientific boundaries. However, they solely quantify mentions without analyzing comments. Alternative metrics are advocated for social science assessment by Priem, Taraborelli, Groth and Neylon [2010], Wouters et al. [2019], and Sugimoto, Work, Larivière and Haustein [2017]. However, understanding of these metrics in Brazil remains limited.

4 Method

Initially, it is essential to clarify that university communication managers validated indicators as part of a comprehensive research project. Aguiar Pereira and Salles-Filho [2022] detailed the conceptual model, based on the recognition that no single model can encompass the diverse objectives of an institutional communication program. The uniqueness of the proposed framework lies in aligning various PCST models with the most relevant set of indicators.

To ground the study, we outlined three ideal types of PCST to establish a reference and define long-term objectives: 1) Informative, 2) Public Engagement, and 3) Participatory. These types were derived from established PCST models in the literature, notably those by Bruce Lewenstein [2010, 2016], and subsequent adaptations by authors such as Bucchi and Trench [2021a, 2021b]. To enrich the Participatory type, we incorporated elements from the Social Appropriation of Science and Technology (ASCyT) model, reflecting the Ibero-American perspective proposed by Colciencias [2010]. This model has been analyzed by scholars such as Polino and Cortassa [2016], Daza-Caicedo et al. [2017], and Ortiz [2021].

Using the concept of ideal types [Weber, 1999], we categorized the PCST models into three types. The following descriptions follow the detailed model proposed by Aguiar Pereira and Salles-Filho [2022].

In the Informative type, we grouped the models described in the literature as the Deficit and Contextual Models [Lewenstein, 2010, 2016; Brossard & Lewenstein, 2009; Bucchi & Trench, 2021a, 2021b]. This type is characterized by the dissemination of information to a mass audience, with an emphasis on mediating to make academic language accessible. The audience is considered passive, and the flow of information is unidirectional.

In the Public Engagement type, we grouped the models described in the literature as Lay Knowledge and Public Engagement [Lewenstein, 2010, 2016; Brossard & Lewenstein, 2009; Bucchi & Trench, 2021a, 2021b]. The key concept in this type is interaction. More actors are included in the communication process, beyond academia. The production incorporates situated lay knowledge in a bidirectional and multidirectional flow, aiming for dialogical communication. Although participation is present in the concept of engagement, it is in the next type that it takes the form of political engagement and the appropriation of science and technology in the broader sense of scientific culture.

The Participatory type incorporates the concept of Social Appropriation of Science and Technology (SAST). It represents an integrated strategy for knowledge production, its democratization, and citizen participation, as well as a critical level that involves public policy [Castelfranchi & Fazio, 2021]. The model of public participation or engagement includes consensus conferences, citizen juries, deliberative technology assessments, science workshops, deliberative polling, and other techniques [Castelfranchi & Fazio, 2021]. It aims to enhance public participation activities with the commitment to “democratize” science by providing public groups with some form of empowerment [Lewenstein, 2016]. Among the objectives, Metcalfe [2019] described collective learning and problem-solving, the integration of different opinions, forms of relationships, and cultures, as well as critical reflection on science and its institutions.

4.1 Construction of the conceptual model

The PCST monitoring and evaluation model was built from the Theory of Change (ToC) [Weiss, 1998; Anderson, 2006; Funnell & Rogers, 2011; Vogel, 2012; Mayne, 2015]. ToC was employed to delineate the assumptions explaining steps toward long-term objectives and connections between activities and products in each intervention phase. Based on this framework, indicators applicable to monitor and evaluate the performance of each communicative process phase were identified.

ToC outlines how a complex change initiative unfolds over time, illustrating the interconnected components necessary to achieve the desired outcome [Anderson, 2006]. Theoretically, we aim to deduce logical pathways to the anticipated outcomes and impacts [Funnell & Rogers, 2011], tailored to each ideal type of PCST. Drawing inspiration from the ‘AMEC Integrated Evaluation Framework’ by the International Association for the Measurement and Evaluation of Communication, we introduce specific adaptations to the PCST field.

For each model, we applied the standard program logic sequence — inputs, outputs, outtakes, and outcomes — and devised a set of indicators for each stage. Following Amec’s taxonomy,⁵ communication activities served as inputs (actions taken), short-term results represented what the target audience perceived (outputs), long-term results (outtakes) denoted the direct benefits to the audience, and impact signified the anticipated outcome in the logical progression of change. In essence, distinct metrics of effectiveness were employed for each model — reach, engagement, and participation. It is essential to emphasize that the model doesn’t address transitions between models but rather the anticipated changes within each proposed model.

For this construction, we conducted a literature review and consulted guiding documents, previous plans, and evaluations. Practically, we prioritized guiding questions and aimed to integrate indicators from social media management and meaningful representations of Brazilian reality. Each result (or precondition) along the change path was assigned an indicator to evaluate the program’s intended effects’ magnitude [Vogel, 2012].

Finally, we revisit the concepts proposed by Mayne [2015], who defines communication outcomes as involving capability change assumptions, which pertain to “the events and conditions required for results to lead to changes in knowledge, attitudes, skills, aspirations, and opportunities”. Additionally, we align impact with Behavior Change Assumptions, encompassing “the events and conditions necessary for changes in the capabilities of target groups to result in actual behavior changes” [Mayne, 2015, p. 124]. Moreover, we correlate these two concepts with different PCST types, presenting distinct assumptions to attain the desired outcomes. The former is linked to social engagement (Type 2), while the latter pertains to the social appropriation of knowledge resulting in behavioral changes (Type 3).

These effects are not directly measured but through indicators providing information on the achievement of objectives or deviations, acting as proxies for the intended results [Morra Imas & Rist, 2009]. Despite the broad scope of the studied object, our focus was on activities (inputs) conducted in the digital environment. However, certain indicators encompass results from other media, including print media, institutional policies, and perception surveys.

4.2 Indicator selection

Indicators guide success identification in each change phase [Anderson, 2006], requiring validation and practical testing for coherence [Olesk et al., 2020]. Social media platforms provide essential performance indicators, such as likes, shares, and engagement [Latorre-Martínez, Orive-Serrano & Íñiguez-Dieste, 2018]. However, metrics like likes and comments lack qualitative depth [Macnamara, 2018; Olesk et al., 2020]. Our framework integrates three dimensions, encompassing three PCST forms: Informative, Public Engagement and Participate/Appropriate. We find ToC’s contextual nature and adjustable form most suitable: “an approximate, flexible guide offering perspectives, not a static forecast” [Vogel, 2012, p. 29]. Based on this model, we tailor which aspects to include in each institutional context.

4.2.1 Type 1 indicators — Informative (scientific dissemination)

In this dimension, we present the activities that typically constitute a PCST program in Brazilian public universities, mainly linked to disseminating scientific knowledge. The activities reinforce the recurring notion of a knowledge deficit, with the public lacking a general understanding of complex scientific subjects. Despite receiving interactions around science content, mainly through likes, there is a lack of active listening to these comments and other forms of engagement across the various channels of scientific dissemination [Aguiar Pereira & Salles-Filho, 2022]. Active listening encompasses a deeper engagement strategy where individuals or organizations actively seek to understand and empathize with their audience’s perspectives, concerns, and feedback. Thus, as short-term results, we expect the public’s reach and attention, leading to three medium/long-term results: a more significant presence in regional, national, and international media; a more substantial web presence; and an improvement in national and international rankings. In this type, the keywords are audience reach and attention.

Based on the theory of change, utilizing the concept of short- and medium-term outcomes as well as the prospecting of assumptions for achieving defined objectives, the model proposed by Aguiar Pereira and Salles-Filho [2022] suggested a set of eight indicators for Type 1: 1) Regularity and number of posts; 2) Presence on social media platforms; 3) Publication reach; 4) Number of likes on the publication; 5) Growth rate (followers, fans, visits); 6) Inclusions in local and regional media; 7) Inclusions in national media; 8) Mentions tracked by altmetrics.

4.2.2 Type 2 indicators — Public Engagement

In this category, the engagement of different audiences with the presented content is the primary characteristic. Production integrates localized lay knowledge in a bidirectional and multidirectional manner, fostering dialogic communication. With a focus on interaction, audiences feel less inhibited in expressing opinions and engaging actively. While participation is inherent in engagement, it evolves into political engagement in subsequent stages, encompassing broader scientific culture appropriation [Lewenstein, 2016; Daza-Caicedo et al., 2017]. Short-term outcomes in public engagement include increased interactions, ranging from likes to comment and shares, leading to a qualitative transformation of these interactions. For medium to long-term effects, we evaluate social participation through social network conversations, the proliferation of citizen science projects, and heightened academic involvement in the media.

For this group, the model suggest this set of indicators: 9) Number of comments; 10) Number of mentions in comments; 11) Engagement rate (interactions/reach); 12) Number of post shares; 13) Sentiment analysis; 14) Relationship between academic and non-academic audience; 15) Site traffic from social networks; 16) % of non-academic individuals in comments; 17) Research projects with social participation; 18) Inclusions in external media on non-institutional topics.

4.2.3 Type 3 indicators — Participate (technical-scientific citizenship)

In this domain, we envision fostering a dialogic rapport with diverse stakeholders (students, press, professional associations, and marginalized groups), culminating in a more participatory scientific citizenship. Here, the emphasis lies on informed participation. Communication endeavors to actively involve citizens in decision-making processes regarding scientific and technological policies [Lewenstein, 2016; Polino & Cortassa, 2016; Daza-Caicedo et al., 2017]. Information flow is multidirectional, originating from institutional communication, academics, and responses to societal demands.

Short-term outcomes encompass PCST initiatives tailored to specific segments, enhanced research-extension interplay, and the formation of novel interaction networks. In the medium to long term, outcomes are linked to the progression of Public Perception of S&T metrics,⁶ integration of PCST objectives and metrics into institutional planning, and the proliferation of projects/research mentioned in patents, varieties, legislation, and public policies. The detailed construction of the model and these assumptions can be found in Aguiar Pereira and Salles-Filho [2022].

For Type 3, we selected the indicators: 19) Students and faculty trained for (S&T); 20) of social participation in suggestions for S&T topics; 21) Number of extension projects with communication channels; 22) Number of former students participants in institutional networks; 23) Number of public discussion forums; 24) Public Perception of S&T research; 25) Number of S&T goals and indicators in institutional strategic planning; 26) Presence and reach of communication channels for Graduate Programs.

In summary, we affirm the alignment of each indicator set with the three delineated types: Type 1 prioritizes “reach”, aiming to capture the audience’s attention. Type 2 emphasizes “interaction”, akin to the social conversation around science elucidated by scholars like Bucchi and Trench [2021a, 2021b]. Lastly, Type 3 underscores “participation” through behavioral change, encompassing engagement in specific communications, such as comments and mentions, and active listening to propositions and perceptions about science itself.

In Type 1, “likes” serve as an indicator representing audience engagement with the content. For Type 2, the engagement rate encompasses various interactions with the content (likes, reactions, comments, and shares), indicating active discourse around the presented topic. In Type 3, indicators include the presence and communication channels of postgraduate programs, signifying participation from segmented audiences with content tailored specifically for that purpose.

4.3 Survey application

The survey⁷ encompassed communication managers from all 69 Brazilian federal universities, yielding a sample of 51 responses (73.9%). Data collection occurred from July 7 to September 2, 2022, via Google Forms. The response rate exceeded typical online survey expectations [Sheehan, 2001], particularly given the institutional sampling units rather than individual ones [Entradas et al., 2020]. The questionnaire targeted professionals overseeing the communication sector, holding various titles such as coordinators, directors, secretaries, superintendents, department heads, and press officers, among others — with only one response per university. Federal universities were selected for study due to their pivotal role in scientific knowledge production in Brazil, notably through postgraduate programs [Souza, De Filippo & Casado, 2017].

The institutions exhibit varying founding ages, sizes, geographical locations, areas of excellence, communication methods, and strategic objectives. The cohort of 51 universities collectively hosts around 1,800 postgraduate programs, catering to student populations ranging from 4,000 to 60,000, with offerings spanning from seven to 120 undergraduate programs [Geocapes, 2022].

Employing descriptive statistics derived from the survey data, we structured and delineated the sample’s profile. Additionally, communication team managers were tasked with assessing a list comprising 26 recommended indicators for monitoring PCST, rating their a) relevance and b) degree of application or potential application within their institution.

Subsequently, we present a perceived risk matrix associated with the proposed model, indicating managers’ perceptions of the main barriers and risks to achieving the results and objectives of PCST programs, as well as difficulties in monitoring and evaluating these actions. Our methodology resonates with Hermann-Pawłowska and Skórska’s [2017] assertion that advanced evaluation frameworks should integrate contextual factors and their associated risk perceptions.

5 Results

From the survey, we present the profile of Brazilian federal universities regarding the management of PCST, particularly in monitoring and evaluation within the digital environment. It’s noteworthy that data pertains to 51 respondent universities. It is important to highlight that presenting the social media profile helps in understanding how PCST is addressed in the institutional profiles of universities.

5.1 Profile of sampled universities

Regarding digital and social media use, YouTube and Instagram are used across all universities, followed by Facebook (98%), Twitter (84.3%), and LinkedIn and WhatsApp (58.8%). Also, 39.2% use Flickr and 31.4% use Telegram. TikTok is used as a communication tool in eight universities (15.7%).

Predominantly, universities rely on manual approaches using platform-provided resources for monitoring and evaluation of platform use. Only 17.6% reported using monitoring tools or software, with 11.8% not systematically monitoring these communication forms. Among monitoring practices, 43.1% indicated comprehensive monitoring of all engagement types and consistent responses to comments and mentions.

For 43.1% of respondents, the main driver for monitoring is to address issues like complaints, criticism, or inappropriate comments, while 25.5% seek to understand the demographics of those interacting with the posts. A majority (51%) stated they respond to the most pertinent comments even during monitoring, while 9.8% never engage with comments. Furthermore, 35.3% conduct sentiment analysis on comments to gauge positivity, negativity, or neutrality. Additionally, 21.6% categorize topics that elicit higher engagement, and 19.6% identify influential individuals, groups, and communities of interest.

Regarding performance reporting for digital media and social networks, 35.3% do not generate such reports. Among those who do, 31.4% produce monthly reports, 7.8% produce weekly reports, 5.9% produce semi-annual reports, and only 2% produce daily reports. For 37.3%, the focus is more on content delivery, with monitoring occurring sporadically. Just 13.7% claim comprehensive social media control, employing proprietary tracking indicators. 9.8% utilize commercial tools for social media monitoring.

Concerning the audience, 41.2% perceive it as broad without segmentation. While acknowledging various audiences, 37.3% identify targeted communication for specific groups needing improvement, with undefined reached segments. The majority of the audience accessing and engaging with S&T information (72.5%) represents an academic profile, including students, professors, and researchers. 11.8% perceive the audience as the general society (non-academic), and 15.7% lack knowledge about the audience’s profile.

Regarding national and international rankings, 86.3% of universities monitor global rankings and communicate their outcomes. Merely 3.9% involve communication professionals in monitoring working groups or committees. For 21.6%, PCST enhances university performance and positioning in rankings, while 13.7% view this ranking type as unrelated to the communication sector.

Concerning media relations, most universities (90.2%) maintain contact lists and periodically issue press releases, with only 11.8% employing specialized companies for press contact management and 19.6% for media mention monitoring. For 49%, media mentions mainly result from university suggestions, while 15.7% reserve exclusive S&T content for media of interest. 21.6% believe independent news topics drive most media mentions unrelated to institutional content.

5.2 Assessment of the indicators

In a specific section of the questionnaire, communication managers were invited to assess the relevance and application of the set of 26 indicators suggested by the model. All the indicators refer to S&T content, specifically focusing on how PCST can be measured across various actions and objectives.

To measure relevance, we used a five-point scale (1, 3, 5, 7, 9), where 1 means minimal relevance and 9 means maximal relevance. This interval scale includes descriptions for levels of relevance, ranging from not relevant to very relevant [Malhotra, 2010]. Odd-numbered scales include options for uncertainty or neutrality [Croasmun & Ostrom, 2011]. Since our scale aimed to show various aspects of a complex concept across different factors, we did not test reliability for each part [Malhotra, 2010]. In the following table, we present the frequency of responses based on the managers’ perceptions.

To improve clarity regarding relevance, we consolidated scores 7 and 9 as highly relevant. The indicators are categorized according to the percentage of the total of these merged scores, as depicted in Figure 1. The indicators were marked with different colors for Type 1, Type 2, and Type 3, allowing visualization of how the indicators are perceived at different levels of reach, engagement, and participation.

For Type 1 indicators — Inform — mostly garnered average relevance scores (5) and significant relevance scores (7 and 9). Figure 2 shows that “Publication Reach” had the highest percentage of scores 7 and 9 (80.3%) within this category. Conversely, the “altmetric-tracked mentions” indicator had the lowest percentage of scores 7 and 9 (21.5%). Tracking mentions through altmetrics can be done using different platforms, the main ones being Altmetric.com, PlumX, and Impactstory. Essentially, they capture mentions of scientific articles from social media data, media coverage, and academic platforms such as Mendeley. Thinking of PCST as the conversation around science and technology, altmetrics can be useful in identifying mentions on social media and analyzing their occurrence and access profile [Wouters et al., 2019].

In Type 2 — Engagement — the “Number of shares” indicator received the highest percentage of scores 7 and 9 (74.5%), while the “% of non-academic individuals mentioned in comments” had the lowest perceived relevance at 23.5%.

Regarding the perception of Type 3 — Participation/Appropriation indicators, “Presence and reach of communication channels for postgraduate programs (PPG)” and “Number of extension projects with communication channels” had the highest percentages of scores 7 and 9 (47.0%), whereas the “Number of discussion forums” indicator was deemed the least relevant (17.5%).

5.3 Application of indicators

Table 2 shows the range of indicator usage, including the number and percentage of universities monitoring each indicator, their short-term and long-term plans to use them, or the perceived inability to use them due to institutional constraints.

Among Type 1 indicators, the majority boast application rates surpassing 50%, with four exceeding 70%. However, the “Mentions tracked by altmetrics” indicator is an outlier, mentioned by only one university. Application rates for Type 1 indicators range from 45% to 78.4%.

For Type 2 indicators, application rates range from 5.8% to 74.5%. Among the ten indicators, four have application rates below 20%, two fall between 20% and 50%, and four exceed 50%. Notably, indicators such as “Percentage of non-academic individuals in comments”, “Relationship between academic and non-academic audience in interactions”, and “Number of mentions in comments” face higher perceived impracticality (above 30%).

In Type 3, most indicators have application rates below 20%, except for the “Presence and reach of PPG communication channels”, which received 21.5% of “already applies” responses. Some institutions deem certain indicators impossible, while others have already implemented them.

5.4 Risk matrix and relationship with the model

Mayne [2015] underscores the impact of external factors, both positive and negative, throughout the ToC process. While prioritizing risk assessment during strategic planning is ideal, evaluating potential risks in established programs also holds significance. Drawing from the risk management framework advocated by the Committee of Sponsoring Organizations of the Treadway Commission [COSO, 2017], a risk matrix was constructed to facilitate analysis.

In our questionnaire to communication managers at Brazilian federal universities, we solicited their evaluation of factors that could impede the attainment of anticipated PCST outcomes and impacts, indicating their perceived risk intensity as low, moderate, or high. These findings are depicted in Figure 2.

The top five factors perceived as high risk include 1) Reduced communication team professionals and 2) Financial resources, both with 82.3%; 3) Insufficient availability of professionals for qualitative assessments (66.6%); 4) Limited tools for tracking engagement (39.2%) and 5) Lack of coordination among institutional sectors (35.2%).

Factors with lower perceived high-risk percentages include: “Sanitary limitations (social distancing)” (0%), “Discrepancy between publication and audience engagement times” (13.7%), and “Normative instructions and information security guidelines” (15.6%). Notably, social distancing due to the COVID-19 pandemic was not deemed a risk to achieving expected PCST outcomes and impacts. This observation follows approximately two years of remote work during the pandemic.

Finally, managers ranked five factors by perceived importance as barriers to evaluating science communication effectiveness in their institution. Responses showed a wide variance in perceptions, yet an order of importance was discernible according to the majority’s perception, as depicted in Figure 3.

According to the research findings, the prioritized sequence of obstacles to monitoring and evaluating PCST is as follows: foremost — Implementation Cost; secondly — Time of Execution; thirdly — Lack of experience among team professionals; fourthly — Absence of internal evaluation guidelines; and lastly — Lack of credibility of results.

Thus, it’s clear that the risk matrix for achieving PCST objectives and its monitoring and evaluation obstacles must consider a comprehensive set of indicators, their applicability, and varying difficulty levels across different institutional contexts. Once again, we emphasize the adaptable nature of the proposed model. Tailoring the intended outcomes and indicators to fit the institution’s reality is a decision influenced by associated risks. The perceptions of risks also exhibit diversity, reflecting our approach of employing distinct scales.

6 Discussion: how do we know we are on the right track?

Our study presented perceptions regarding potential indicators for inclusion in a PCST monitoring and evaluation model. The discussion we propose involves reconsidering the theoretical framework underpinning this assessment. Consistent with Macnamara [2017], we emphasize the importance of understanding the objectives behind communication activities and their impact on human attitudes and behaviors. This entails considerations such as whether the goal is to reach a broad audience or foster engagement with the content, and whether quantity or the effective participation of specific groups is prioritized in social media conversations.

Our approach aligns with Metcalfe’s advocacy [2019] for integrating multiple PCST models. Proposing a practical evaluation of the PCST models (deficit, dialogue, and participation), the author concluded that the models are not mutually exclusive but coexist in PCST programs, including a mix of approaches. Our study indicates that the activities and objectives of Types 2 and 3 can coexist and are reliant on Type 1 activities rooted in the diffusion model. Our research unveils a symbiotic coexistence of these three activity types, mirrored by a similar pattern in monitoring and evaluation indicators. Universities prioritize indicators from the informational model (Type 1), followed by engagement indicators (Type 2), with Type 3 indicators being the least prominent.

Our findings support the idea that technology has improved communication, but there remains a need for clear policies and strategies to guide its use in the PCST context. While most surveyed universities utilize technology for content dissemination on media platforms, consistent performance monitoring is irregular and often marginalized within communication departments. Notably, only 13.7% of universities employ tracking metrics for comprehensive social media control. Monitoring and evaluation efforts predominantly focus on Type 1 — Informational, echoing observations made by Massarani [2012] a decade ago, highlighting the prevalent trend in science communication favoring the deficit model in the Brazilian context.

According to Massarani’s study, more than a decade later, our findings continue to support the notion that informing society remains a primary focus for most universities, with less emphasis placed on receiving questions, suggestions, or explaining processes. These results underscore the persistence of an informational and instrumental bias, aligned to the core idea to promote scientific literacy⁸ and consolidate institutional structures to enhance media visibility, as noted by Metcalfe [2019].

An important observation is the increased collaboration between science communicators/journalists and professors/scientists/researchers, as noted by Schäfer and Fähnrich [2020]. This trend matches the growing interest in PCST, shown by more content created by academics on institutional platforms. This type of communication is perceived to improve understanding and attract more attention from society.

Our study points to an increasing focus on monitoring, with 43.1% of universities monitoring social media engagement, including tracking complaints, criticisms, or inappropriate comments. Additionally, 25.5% identify characteristics of those interacting, 35.3% conduct sentiment analysis, 21.6% categorize themes and subjects, and 19.6% identify key influencers, groups, and communities of interest.

These findings suggest that while universities traditionally focused mainly on Type 1 activities, a significant portion has integrated monitoring mechanisms aligned with Type 2 — Engagement. This trend is reinforced by the perceived relevance and application of indicators related to conversational aspects, reflecting a more interactive approach [Gascoigne & Schiele, 2020].

However, the majority of universities face human resource constraints, with professionals often juggling multiple roles and lacking the necessary training to manage these interactions effectively, as noted by Fitzpatrick and Weissman [2021]. Notably, 43.1% of universities lack professionals trained in digital media/social media management within their teams. Additionally, planning for these endeavors and their systematic assessment often requires improvement, hindering the improvement cycle in communicative practices. These authors advocate for clearly delineating objectives, emphasizing strategies and tactics that support their realization, including focusing on actions and criteria for program monitoring and subsequent evaluation.

In this context, the target audience assumes prominence. It’s crucial not only to map and understand the audiences engaged with by organizations/universities but also to comprehend their perspectives and engagement in the communication process. Our model’s proposed indicators reveal a lower perception of relevance for those linked to audience characteristics. However, “reaching non-academic audiences” emerges as the second most critical factor for PCST, as indicated by 58.8% of respondents. This disparity underscores the necessity for better alignment between expected outcomes and the actions and indicators guiding the communicative process.

Each university must ascertain its science communication strategy and direction, evaluate feasible options within its context, and delineate desired outcomes. Herein lies the relevance of the Theory of Change. This framework enables a reflective examination of our actions, methodologies, achieved outcomes, and envisioned future states. Prioritizing Type 1 strategies doesn’t guarantee Type 3 outcomes; rather, it involves defining strategic steps toward desired changes and selecting pertinent indicators aligned with priority factors.

Based on the survey, it’s evident that Type 3 indicators are least implemented in Brazilian universities. Despite their limited application, they signal the essential behavioral changes highlighted by Mayne [2015] and endorsed by scholars studying PCST models, indicating participatory communication. It is important to remember that these assumptions encompass communication involving behavioral change, where citizens are included in the decision-making processes regarding science and technology policies.

Aligned with Lewenstein [2016], we emphasize the importance of sparking discussions within the scientific community on integrating society into science and technology dialogues. Dialogue fosters emotional connections between scientists and the public while understanding the institutional and social policy aspects is crucial for assessing PCST’s role in public engagement.

Our findings present a current panorama of PCST monitoring and evaluation in Brazilian federal universities, crucial entities in the nation’s science and technology landscape. It underscores that the perceived relevance of indicators doesn’t always directly translate into implementation due to diverse influential factors. The universities’ profiles reveal areas for improvement, such as inadequate training for digital media management and limited social media monitoring resources.

Ultimately, the model’s adaptability to diverse institutional contexts is crucial for tailoring it to specific perceived risks and objectives. The wide variation in indicator perception underscores the necessity of customizing the monitoring and evaluation model for each institution. The study’s significance lies in identifying trends and anticipating their potential application in universities.

7 Limitations and future research

This study acknowledges the challenge of assessing the intangible nature of communication’s results and impacts. Measuring communication effects proves complex, and indicator selection, though rooted in theory, retains a degree of subjectivity. We suggest refining and updating indicators as the model diffuses and is applied. Establishing causality in evaluation poses complexity, particularly in distinguishing action effects amidst numerous factors, especially without experimental methods. The solution involves systematic monitoring to establish causal relationships between actions and outcomes.

Another limitation concerns the selection of three primary types of PCST (informative — engagement — participatory) potentially overlook other unanticipated possibilities. However, we propose that adapting to each institutional context entails reflecting on the cross-cutting nature of these objectives. In essence, even if an institution aims to enhance its reputation, both reach and engagement should be deemed crucial for achieving this objective.

This study’s contributions suggest three research pathways. Adapting and implementing the model in case studies would enable assessing indicator relevance in communication planning and exploring their practical application, measurement techniques, weighting, and usage trends. Another avenue involves examining the role of automated tools in assessing PCST in digital media. Lastly, adopting the theoretical framework to other organizational structures, such as Science and Technology institutions and strategic programs, would facilitate evaluating PCST’s role and impact in projects and investigating its integration into funding methodologies.

References

Aguiar Pereira, C. M. G. & Salles-Filho, S. L. M. (2022). Tipos ideais e Teoria da Mudança: proposição de modelo de avaliação para a comunicação pública de ciência e tecnologia. JCOM – América Latina 05 (02), A03. doi:10.22323/3.05020203

Anderson, A. A. (2006). The community builder’s approach to Theory of Change. A practical guide to theory development. The Aspen Institute Roundtable on Community Change. New York, NY, U.S.A. Retrieved from https://www.theoryofchange.org/pdf/TOC_fac_guide.pdf

Barba, M. L. P., Castillo, J. P. G. & Massarani, L. (2019). Public engagement in science: mapping out and understanding the practice of science communication in Latin America. Anais da Academia Brasileira de Ciências 91 (1), e20171000. doi:10.1590/0001-3765201920171000

Bastos, A. (2020). Engajamento público em controvérsia científica: o caso da pílula do câncer (Ph.D. Thesis, Federal University of Minas Gerais, Faculty of Philosophy and Human Sciences, Belo Horizonte, Brazil). Retrieved from http://hdl.handle.net/1843/34138

Besley, J. C., Newman, T. P., Dudo, A. & Tiffany, L. A. (2020). Exploring scholars’ public engagement goals in Canada and the United States. Public Understanding of Science 29 (8), 855–867. doi:10.1177/0963662520950671

Brossard, D. & Lewenstein, B. V. (2009). A critical appraisal of models of public understanding of science: using practice to inform theory. In L. Kahlor & P. Stout (Eds.), Communicating science: new agendas in communication (pp. 25–53). doi:10.4324/9780203867631

Bucchi, M. & Trench, B. (2021a). Rethinking science communication as the social conversation around science. JCOM 20 (03), Y01. doi:10.22323/2.20030401

Bucchi, M. & Trench, B. (Eds.) (2021b). Routledge handbook of public communication of science and technology. doi:10.4324/9781003039242

Burns, T. W., O’Connor, D. J. & Stocklmayer, S. M. (2003). Science communication: a contemporary definition. Public Understanding of Science 12 (2), 183–202. doi:10.1177/09636625030122004

Calcagnini, S. & Xanthoudaki, M. (2016). The researcher as communicator: a competitor or an ally? Spokes 22. Retrieved from http://www.ecsite.eu/activities-and-services/news-and-publications/digital-spokes/issue-22

Cancino, R., Orozco, L. A., Medina, J., García, M., Coloma, J., Bustos, F. & Alister, C. (2021). Methodology for the governance and management of the university as a complex adaptive system based on the three missions: research, education, and connecting with its surroundings. In L. A. Orozco, G. Ordóñez-Matamoros, J. H. Sierra-González, J. García-Estévez & I. Bortagaray (Eds.), Science, technology, and higher education: governance approaches on social inclusion and sustainability in Latin America (pp. 253–285). doi:10.1007/978-3-030-80720-7_10

Carneiro, B., Resce, G. & Sapkota, T. B. (2022). Digital artifacts reveal development and diffusion of climate research. Scientific Reports 12 (1), 14146. doi:10.1038/s41598-022-17717-8

Castelfranchi, Y. & Fazio, M. E. (2021). Comunicación pública de la ciencia. Foro Abierto de Ciencias Latinoamérica y Caribe. UNESCO. Montevideo, Uruguay. Retrieved from http://forocilac.org/wp-content/uploads/2021/04/PolicyPapers-CILAC-ComunicacionPublicaCiencia-ES.pdf

Colciencias (2010). Estrategia nacional de apropiación social de la ciencia, la tecnología y la innovación. Departamento Administrativo de Ciencia, Tecnología e Innovación, República de Colombia. Bogotá, Colombia.

Committee of Sponsoring Organizations of the Treadway Commission (COSO) (2017). Gerenciamento de riscos corporativos — estrutura integrada. IIA Brasil.

Croasmun, J. T. & Ostrom, L. (2011). Using Likert-type scales in the social sciences. Journal of Adult Education 40 (1), 19–22.

Davies, S. R. & Horst, M. (2016). Science communication: culture, identity, and citizenship. doi:10.1057/978-1-137-50366-4

Daza-Caicedo, S., Maldonado, O., Arboleda-Castrillón, T., Falla, S., Moreno, P., Tafur-Sequera, M. & Papagayo, D. (2017). Hacia la medición del impacto de las prácticas de apropiación social de la ciencia y la tecnología: propuesta de una batería de indicadores. História, Ciências, Saúde – Manguinhos 24 (1), 145–164. doi:10.1590/s0104-59702017000100004

De Filippo, D., Benayas, J., Peña, K. & Sánchez, F. (2020). Communication on sustainability in Spanish universities: analysis of websites, scientific papers and impact in social media. Sustainability 12 (19), 8278. doi:10.3390/su12198278

Economou, E., Luck, E. & Bartlett, J. (2023). Between rules, norms and shared understandings: how institutional pressures shape the implementation of data-driven communications. Journal of Communication Management 27 (1), 103–119. doi:10.1108/JCOM-01-2022-0009

Entradas, M., Bauer, M. W., O’Muircheartaigh, C., Marcinkowski, F., Okamura, A., Pellegrini, G., … Li, Y.-Y. (2020). Public communication by research institutes compared across countries and sciences: building capacity for engagement or competing for visibility? PLoS ONE 15 (7), e0235191. doi:10.1371/journal.pone.0235191

Fitzpatrick, K. R. & Weissman, P. L. (2021). Public relations in the age of data: corporate perspectives on social media analytics (SMA). Journal of Communication Management 25 (4), 401–416. doi:10.1108/JCOM-09-2020-0092

Funnell, S. C. & Rogers, P. J. (2011). Purposeful program theory: effective use of theories of change and logic models. San Francisco, CA, U.S.A.: Jossey-Bass.

Fürst, S., Volk, S. C., Schäfer, M. S., Vogler, D. & Sörensen, I. (2022). Organizational and societal goals in tension? A survey of communication practitioners at Swiss higher education institutions. JCOM 21 (07), A06. doi:10.22323/2.21070206

Gascoigne, T. & Schiele, B. (2020). Introduction. A global trend, an emerging field, a multiplicity of understandings: science communication in 39 countries. In T. Gascoigne, B. Schiele, J. Leach, M. Riedlinger, B. V. Lewenstein, L. Massarani & P. Broks (Eds.), Communicating science: a global perspective (pp. 1–14). doi:10.22459/CS.2020.01

Geocapes (2022). Distribuição de programas de pós-graduação. Sistema de Informações Georreferenciadas — CAPES. Retrieved from https://geocapes.capes.gov.br/geocapes/

Grand, A. & Sardo, A. M. (2017). What works in the field? Evaluating informal science events. Frontiers in Communication 2, 22. doi:10.3389/fcomm.2017.00022

Hermann-Pawłowska, K. & Skórska, P. (2017). A comprehensive review of the theory-based evaluation concepts. In J. Pokorski, Z. Popis, T. Wyszyńska & K. Hermann-Pawłowska (Eds.), Theory-based evaluation in complex environments (pp. 21–34). Warsaw, Poland: Polish Agency for Enterprise Development. Retrieved from https://en.parp.gov.pl/storage/publications/pdf/20017_theory-based-evaluation.pdf

Jensen, E. A. (2014). The problems with science communication evaluation. JCOM 13 (01), C04. doi:10.22323/2.13010304

Jensen, E. A. & Gerber, A. (2020). Evidence-based science communication. Frontiers in Communication 4, 78. doi:10.3389/fcomm.2019.00078

Joubert, M., Davis, L. & Metcalfe, J. (2019). Storytelling: the soul of science communication. JCOM 18 (05), E. doi:10.22323/2.18050501

Latorre-Martínez, P., Orive-Serrano, V. & Íñiguez-Dieste, D. (2018). Medición y análisis de la audiencia social de las televisiones autonómicas en Facebook y Twitter / Measurement and analysis of the presence in Facebook and Twitter in the regional television broadcaster’s context in Spain. Profesional de la Información 27 (5), 1061–1070. doi:10.3145/epi.2018.sep.10

Lewenstein, B. V. (2010). Modelos de comprensión pública: la política de la participación pública. ArtefaCToS 3 (1), 13–29. Retrieved from http://hdl.handle.net/10366/120832

Lewenstein, B. V. (2016). Can we understand citizen science? JCOM 15 (01), E. doi:10.22323/2.15010501

Macnamara, J. (2017). Evaluating public communication: exploring new models, standards, and best practice. doi:10.4324/9781315391984

Macnamara, J. (2018). A review of new evaluation models for strategic communication: progress and gaps. International Journal of Strategic Communication 12 (2), 180–195. doi:10.1080/1553118x.2018.1428978

Macnamara, J., Lwin, M. O., Hung-Baesecke, C.-J. F. & Zerfass, A. (2021). Asia-Pacific Communication Monitor 2020/21. Strategic issues, competency development, ethical challenges, and gender equality in the communication profession. Results of a survey in 15 countries and territories. Asia Pacific Association of Communication Directors (APACD), European Public Relations Education and Research Association (EUPRERA). Hong Kong, Brussels.

Malhotra, N. K. (2010). Pesquisa de marketing: uma orientacao aplicada. Porto Alegre, Brazil: Bookman.

Martins, C. V. O. (2021). As políticas de comunicação nas organizações públicas: um olhar sobre as universidades federais brasileiras (Ph.D. Thesis, Universidade do Minho, Instituto de Ciências Sociais, Braga, Portugal). Retrieved from https://hdl.handle.net/1822/75375

Massarani, L. (2012). Comunicação da ciência e apropriação social da ciência: algumas reflexões sobre o caso do Brasil. Revista Uni-pluri/versidad 12 (3), 92–100. Retrieved from https://www.arca.fiocruz.br/handle/icict/45659

Massarani, L. (2022). 30 years of PUS: reflections from Latin America on the academic field of science communication. Public Understanding of Science 31 (3), 323–330. doi:10.1177/09636625221076203

Massarani, L., Moreira, I. & Lewenstein, B. (2017). A historical kaleidoscope of public communication of science and technology. JCOM 16 (03), E. doi:10.22323/2.16030501

Mayne, J. (2015). Useful theory of change models. The Canadian Journal of Program Evaluation 30 (2), 119–142. doi:10.3138/cjpe.230

Metcalfe, J. (2019). Comparing science communication theory with practice: an assessment and critique using Australian data. Public Understanding of Science 28 (4), 382–400. doi:10.1177/0963662518821022

Morra Imas, L. G. & Rist, R. C. (2009). The road to results: designing and conducting effective development evaluations. World Bank Group. Washington, DC, U.S.A. Retrieved from http://documents.worldbank.org/curated/en/400101468169742262/The-road-to-results-designing-and-conducting-effective-development-evaluations

National Research Council (NRC). National Academies of Sciences, Engineering, and Medicine (2016). Effective chemistry communication in informal environments. doi:10.17226/21790

Olesk, A., Renser, B., Bell, L., Fornetti, A., Franks, S., Mannino, I., … Zollo, F. (2021). Quality indicators for science communication: results from a collaborative concept mapping exercise. JCOM 20 (03), A06. doi:10.22323/2.20030206

Olesk, A., Renser, B., Franks, S., Schofield, B., Villa, R., Zollo, F., … Bell, L. (2020). D2.1: Key performance indicators for quality assessment in science communication. European Commission & Quest Project. QUEST Consortium.

Ortiz, J. M. E. (2021). Cómo medir la apropiación social de la ciencia y la tecnología: la definición de indicadores como problema. Innovar 31 (80), 153–166. doi:10.15446/innovar.v31n80.93672

Pellegrini, G. (2014). The right weight: good practice in evaluating science communication. JCOM 13 (01), C03. doi:10.22323/2.13010303

Pellegrini, G. (2021). Evaluating science communication. In M. Bucchi & B. Trench (Eds.), Routledge handbook of public communication of science and technology (3rd ed., pp. 305–322). doi:10.4324/9781003039242

Polino, C. & Cortassa, C. (2016). Discursos y prácticas de promoción de cultura científica en las políticas públicas de Iberoamérica. Trilogía: Ciencia Tecnología Sociedad 8 (15), 13–24. Retrieved from https://ssrn.com/abstract=3528218

Priem, J., Taraborelli, D., Groth, P. & Neylon, C. (2010). Altmetrics: a manifesto. Retrieved from http://altmetrics.org/manifesto

Schäfer, M. S. & Fähnrich, B. (2020). Communicating science in organizational contexts: toward an “organizational turn” in science communication research. Journal of Communication Management 24 (3), 137–154. doi:10.1108/jcom-04-2020-0034

Sheehan, K. B. (2001). E-mail survey response rates: a review. Journal of Computer-Mediated Communication 6 (2), JCMC621. doi:10.1111/j.1083-6101.2001.tb00117.x

Souza, C. D., De Filippo, D. & Casado, E. S. (2017). A colaboração científica brasileira no contexto do programa Ciência sem Fronteiras. In A. Cuevas-Cerveró, S. Sánchez-Cuadrado, M. T. Fernández-Bajón & E. Simeao (Eds.), Investigación en información, documentación y sociedad: perspectivas y tendencias (Vol. 2, pp. 299–310). Universidad Complutense de Madrid, Departamento de Biblioteconomía y Documentación.

Sugimoto, C. R., Work, S., Larivière, V. & Haustein, S. (2017). Scholarly use of social media and altmetrics: a review of the literature. Journal of the Association for Information Science and Technology 68 (9), 2037–2062. doi:10.1002/asi.23833

Trench, B. (2017). Universities, science communication and professionalism. JCOM 16 (05), C02. doi:10.22323/2.16050302

Vogel, I. (2012). Review of the use of ‘Theory of Change’ in international development. UK Department for International Development (DFID). Retrieved from https://www.theoryofchange.org/pdf/DFID_ToC_Review_VogelV7.pdf

Volk, S. C. & Zerfass, A. (2021). Management tools in corporate communication: a survey about tool use and reflections about the gap between theory and practice. Journal of Communication Management 25 (1), 50–67. doi:10.1108/JCOM-02-2020-0011

Watermeyer, R. (2012). From engagement to impact? Articulating the public value of academic research. Tertiary Education and Management 18 (2), 115–130. doi:10.1080/13583883.2011.641578

Weber, M. (1999). A “objetividade” do conhecimento nas ciências sociais. In G. Cohn (Ed.), Weber — Sociologia (pp. 79–127). São Paulo, Brazil: Editora Ática.

Weiner, M., Fitzsimmons, A., DiStaso, M., Draper-Watts, P., Ehrhart, C., Gilfeather, J., … Stacks, D. (2021). The communicator’s guide to research, analysis, and evaluation. Institute for Public Relations. Retrieved from https://instituteforpr.org/communicators-guide-research-2021/

Weingart, P. & Joubert, M. (2019). The conflation of motives of science communication — causes, consequences, remedies. JCOM 18 (03), Y01. doi:10.22323/2.18030401

Weiss, C. H. (1998). Evaluation: methods for studying programs and policies (2nd ed.). Upper Saddle River, NJ, U.S.A.: Prentice Hall.

Wiencierz, C. & Röttger, U. (2019). Big data in public relations: a conceptual framework. Public Relations Journal 12 (3), 1–15.

Wouters, P., Zahedi, Z. & Costas, R. (2019). Social media metrics for new research evaluation. In W. Glänzel, H. F. Moed, U. Schmoch & M. Thelwall (Eds.), Springer handbook of science and technology indicators (pp. 687–713). doi:10.1007/978-3-030-02511-3_26

Authors

Cibele Maria Garcia de Aguiar. Graduated in Social Communication (Journalism), master in Business Administration, and Ph.D. in Science and Technology Policy at the State University of Campinas (Unicamp). Journalist at the Federal University of Lavras (UFLA) since 2011. Member of the Laboratory of Studies on Research and Innovation Organization — GEOPI.
@CibeleAguiar10 E-mail: cibele.aguiar@ufla.br

Sergio Luiz Monteiro Salles Filho. Ph.D. in Economics from State University of Campinas — UNICAMP (1993). Full professor at the Department of Science and Technology Policy of the Institute of Geosciences at the University of Campinas, Brazil. Leader of the Laboratory of Studies on Research and Innovation Organization — GEOPI and coordinator of the project Research of the Research: indicators, methods, and evidence of impacts.
E-mail: sallesfi@unicamp.br

Sérgio Parreiras Pereira. Agricultural engineer, master, and Ph.D. in Agronomy (Phytotechnics) from the Federal University of Lavras (UFLA). Scientific researcher at the Campinas Agronomic Institute (IAC) and coordinator of the Social Coffee Network, the largest network for sharing news, science, and technology about the coffee sector.
@pereiracafes E-mail: sergiopereira@iac.sp.gov.br

Fernando Antonio Basile Colugnati. Background in statistics from State University of Campinas (Unicamp) and Ph.D. in Science by Federal University of São Paulo (Unifesp, 2005) with a fellowship at CDC, Atlanta-USA (2004) and post-doctorate in Science, Technology and Innovation Evaluation at Unicamp (2013). Associate professor at Medicine School of Federal University of Juiz de Fora (UFJF). Member of the Reference Center in Research, Intervention, and Evaluation in Alcohol and Other Drugs (CREPEIA), member of the International Society for Pharmacological and Outcomes Research (ISPOR), and the International Society of Biometrics (IBS).
@FernandoColug E-mail: fernando.colugnati@medicina.ufjf.br

Endnotes