HomeEvidence, Evaluation, and Learning …Demarche Analysis Signals Need for Data Consistency Across the Department hide Demarche Analysis Signals Need for Data Consistency Across the Department November 2022A 2022 analysis commissioned by the Department of State demonstrated challenges regarding the ability to measure the effectiveness of universal demarching for gaining support for United Nations General Assembly (UNGA) resolutions due to differing formats. Demarching is the act of sending a “demarche”, which is a formal diplomatic communication of a government’s official position to another government or international organization. Issues with data quality in demarche cables impeded obtaining nuanced and accurate results, and further study is recommended. Background The Bureau of International Organization Affairs (IO) and the Office of Management Strategy Solutions (M/SS)’s Center for Analytics (CfA) developed and applied new data tools alongside social science research methodology to study the effectiveness of multilateral demarches on voting coincidence at UNGA. The report addresses the question: “Which diplomatic interventions in multilateral fora are effective for achieving U.S. foreign policy objectives?” Findings IO and M/SS/CfA’s analysis concluded the following: More research is needed to identify the conditions under which demarching may be effective. Issues with data quality in demarche cables was an impediment to obtaining nuanced and accurate results. Recommendations Conduct resolution-by-resolution analysis. Analyze vote coincidence in other UN bodies and other intergovernmental organizations. Analyze alternative approaches to gaining support via demarche, such as targeting likely swing votes. Standardize and formalize demarche request and response language. Further investigate universal demarching on voting coincidence in UNGA. Evaluation Questions Does the number of demarche requests sent to a country indicate how often a country will vote with the United States? Does a post’s response rate to demarche requests indicate how often that post’s country will vote with the United States? Does the timeliness of a post’s responses to demarche requests indicate how often a country will vote with the United States? Does the position level of the response drafter at post indicate how often that post’s country will vote with the United States? Detailed Findings Overall, although the findings questioned whether current demarching practices advance U.S. Government interests at UNGA, the study faced several data quality limitations that impeded accurate results. Below are the findings for the evaluation questions. Question 1: There was a statistically significant, positive correlation between demarche request frequency and a country’s likelihood of voting with the United States on UNGA resolutions. However, the substantive significance is minimal. Question 2: There was no apparent correlation between a post’s responsiveness and the host nation’s voting alignment with the United States on UNGA resolutions. Question 3: There was no clear linear relationship between response timeliness and voting coincidence with the United States. Question 4: There was not a clear relationship between drafter position level and voting coincidence. However, further study is recommended given data availability limited the interpretation of these results. Further work is needed to standardize and formalize – and therefore, improve – data quality. Approach, Methods, and Data Data Collection: The team used the corpus of government diplomatic cables available on the Department’s Sensitive but Unclassified network OpenNet and their metadata as their primary data source. The final dataset comprised 334,859 diplomatic cables from across the Department, including 2,276 unique cables identified as demarche requests and responses (between January 1, 2020, and December 31, 2021). Data Analysis: The team conducted analysis in three parts: trend analysis, distribution analysis, and hypothesis testing. Data Visualization: As a byproduct of this study, the research team designed and built an automated demarche tracker, which will assist IO and the Department by automating demarche and demarche response tracking. Department of State Learning As a result of the analysis, the following has taken place or is set to occur: The Department developed one demarche tracker to increase the utility of and access to cables and will continue to assess how to best leverage demarching to achieve desired multilateral outcomes. The Bureau of International Organization Affairs will work with the Office of Foreign Assistance and the Bureau of Budget and Planning to socialize research results for the Department’s Learning Agenda.