HomeEvidence, Evaluation, and Learning …An Assessment of the Current State of Monitoring and Evaluation in Public Diplomacy hide An Assessment of the Current State of Monitoring and Evaluation in Public Diplomacy November 2022In 2021, the Office of Policy, Planning, and Resources for Public Diplomacy and Public Affairs (R/PPR) conducted a qualitative study on the current state of monitoring and evaluation (M&E) in the field. The study found that practices, understanding, and expectations of M&E expressed by field-based public diplomacy (PD) practitioners vary. The study identified several gaps and barriers to be addressed through guidance and capacity-building. Background Department of State PD practitioners and Congress have highlighted the need for more robust support for M&E of PD programs. R/PPR is planning several initiatives to support improvements in the quality of PD programs, including M&E efforts. To inform these initiatives, R/PPR conducted a series of in-depth interviews with field-based PD Foreign Service Officers (FSO) and Locally Employed Staff (LE Staff). These assessed their current practices and understanding of programmatic M&E, data collection, and their perspectives on future support for M&E. Findings Interviewees expressed: Differing perspectives on how PD practitioners define and measure program outcomes and interpret PD goals. Reliance on ad-hoc observation and intuition instead of systematic data collection. Lack of time to conduct M&E and a need for more specialized M&E staff and direct consultation services. A perception that DC offices do not understand the local context and constraints when answering M&E-related data calls. Challenges with data privacy laws and cultural norms in collecting sensitive information. Recommendations Adopt and promote common standards for conducting M&E at post. Clarify expectations for M&E supported by capacity-building and guidance. Communicate with practitioners about M&E-related support offices and resources. M&E data calls should focus on clear policy connections and programmatic outputs. Provide exemptions when restrictions prohibit collecting sensitive information. Research Questions What practices do field-based PD staff have when conducting programmatic M&E? What barriers do field-based PD staff face when conducting programmatic M&E? What are the expectations for field-based PD M&E set by PD Leadership in DC? What additional support do PD staff need to improve conducting M&E? Approach, Methods, & Data Between September 2021 and January 2022, the R/PPR Research and Evaluation team conducted a qualitative survey, which included 29 interviews with field-based PD staff. This survey included 19 interviews with LE Staff and 10 interviews with FSOs. The research team drew a random sample of embassies and consulates within regions to ensure a diversity of PD section size. They used guided coding and content analysis to analyze the qualitative data. Many interviewees demonstrated a weak understanding of the foundational concepts around M&E, which required the research team to adjust the interview protocol and spend time level-setting on core terminology. The data collected is best used to understand and characterize interviewees’ perceptions, attitudes, understanding, and behaviors around a particular topic to understand their experience better. The findings and recommendations discussed in this report are solely based on the interviews conducted during this project. Public Diplomacy section staff in Indonesia present an example outreach and engagement plan while attending a R/PPR Monitoring and Evaluation training at Embassy Jakarta. Detailed Findings 1. Differing perspectives on how PD practitioners define and measure program outcomes and interpret PD goals. High levels of decentralization, discretion, and little M&E guidance for PD section-level planning and design made implementation of standardized processes for appropriately defining program outcomes, developing indicators, and collecting data challenging. PD staff depended on implementing partners with low M&E capacity to define and measure program outcomes. As such, program outcomes were widely defined as either direct outputs, number of programs completed, or amount of social media coverage. 2. Reliance on ad-hoc observation and intuition instead of systematic data collection. Field-based PD program staff commonly relied on observation, intuition, or qualitative anecdotes to collect program information. The lack of more systematic or rigorous M&E standards was exemplified in non-standardized data collection processes, overreliance on qualitative data to measure results, and unsystematic methods for acquiring and using partner data for PD programmatic decision-making. 3. Lack of time to conduct M&E and a need for more specialized M&E staff and direct consultation services. There was limited time for staff to conduct appropriate M&E practices. PD staff noted they were often requested to perform tasks and services outside of their primary jobs, areas of expertise, and to other sections limiting their bandwidth. Additionally, PD leadership requested implementation of programs that were not identified in their Public Diplomacy Implementation Plans (PDIP), nor in other strategic planning frameworks. The result was a wariness to add new requirements around M&E, fearing that they would further strain PD sections already subject to myriad demands not always aligned or coordinated. Interviewees also reported that there was a lack of specialized M&E staff, and few capable in-country M&E partners. More M&E staff is needed in the field and to increase the use of departmental M&E guidelines and templates. The ad hoc nature of these requests on staff with limited time combined with few knowledgeable M&E staff in the field made thoughtful and intentional M&E difficult. 4. A perception that DC offices do not understand the local context and constraints when answering M&E-related data calls. Data calls from PD leadership in DC skewed expectations for field-based PD staff as to the type of M&E data that is most important to collect–mostly outputs and anecdotes. Staff also noted a lack of application of departmental or government-wide technical guidance when discussing standards and expectations for M&E with DC. However, PD staff have observed a positive shift in tone from DC for better M&E practices but noted skepticism about actual interest in M&E. Finally, field-based staff see PD leadership in DC as lacking an understanding of the local context, for example, when requesting data that might not exist in a country. 5. Challenges with data privacy laws and cultural norms in collecting sensitive information. Data collection is difficult for logistical as well as regulatory and cultural reasons. Many countries have remote populations with restrictive data collection and privacy laws combined with traditional cultural/ethnic group complexities that have impeded thorough data collection of sensitive information like ethnicity, religious affiliation, or gender. Public Diplomacy Framework Learn & Adapt The PD Framework to achieve U.S. Foreign Policy outcomes includes five initiatives – apply policy in context, analyze audiences, develop plans, manage effectively, and learn and adapt. The PF M&E study puts the Learn & Adapt principle of the PD framework into practice. This principle aims to assess whether PD initiatives and activities meet their objectives, report on outcomes, analyze and share results, and use data to adjust plans and inform future efforts. Findings by the Numbers According to a recent survey of PD staff in the field, which supports the findings of this study: 48%of respondents report that evidence is more important than personal opinions in making decisions. 37%report their section has the right skills to collect useful data on their programs. 41%responded they struggle to collect data on PD programs. 40%noted it being difficult to access research or M&E support. Learning Summary In January 2023, R/PPR staff led an Evaluation Community of Practice Meeting where they presented the findings of this study. Representatives from M&E groups, including USAID, The Millennium Challenge Corporation (MCC), Bureau of International Narcotics and Law Enforcement Affairs (INL), and Bureau of Educational and Cultural Affairs (ECA), attended and provided their perspectives and suggestions: Clearly define the goals of PD and PD interventions. Determine what success looks like for M&E implementation and communicate it. Determine who will do M&E at post and in DC and define how they will interact together. General expectations for M&E should be low given current conditions, and it will take many years to make modest improvements. Use short, pithy research briefs to distill compiled information.