Brian Cox on “Summarizing the Detailed Assessment of the Sexual Assault Prevalence Statistics at the Center of the Military Justice Reform Movement”
In today’s post Professor Brian Cox introduces a study of critical importance to the ongoing debate about the proposal now in Congress to eviscerate commanders’ authority in military justice matters. Brian is one of the nation’s foremost experts on this issue, so when his research guts the oft-given interpretation of a key statistic upon which the proposal advocates depend, everyone ought to take notice. Just read this section of his post and you’ll realize why his work is so important:
Misusing the data to claim that the survey report does present an estimate of actual prevalence, however, is not supportable by either the methodology or the actual underlying data. This assertion – that an estimated 20,000 actual sexual assault offenses occur in the military each year – has long been the cornerstone upon which the movement to divest commanders of court-martial disposition authority is built. The claim is not supported by the data – it never has been.
Read Brian’s full post here:
Summarizing the Detailed Assessment of the Sexual Assault Prevalence Statistics at the Center of the Military Justice Reform Movement
by Brian Cox
As both chambers of Congress get set to debate the respective versions of the bills involving military justice reform that may soon emerge from markup in committee, it is vitally important that members of Congress and the constituents they represent possess a clear understanding of the various challenges the legislation is intended to address.
The assertion that an estimated 20,000 service members are sexually assaulted each year despite all prior endeavors in the military to improve prevention and response efforts lies at the very core of the long-running movement to divest commanders of disposition authority for selected offenses. But what is the source of this estimate, and how effective is the estimate as a measure of the performance of the military justice system to hold offenders accountable?
These questions inspired me to investigate the source of the statistics, and my findings were recently published in an essay that is available here. While the full analysis is available in the essay, this guest post provides an opportunity to summarize the analysis.
Central Sexual Assault Prevalence Statistics in Context
Senator Kirsten Gillibrand summarizes the case in favor of adopting the Senate version of the draft legislation well in a recent blog post that is available here.
In that post, Senator Gillibrand begins by asserting that there “is an epidemic of sexual assault in the military and the current military justice system has proven incapable of addressing it.” In support of this assertion, she notes that “an estimated 20,500 service members are sexually assaulted every year, while “less than one in ten of the cases considered are sent to trial, and just a small fraction of those cases end in conviction.”
The solution to this problem, according to Senator Gillibrand, is “to change who is responsible for handling these cases” since “commanders are also not trained lawyers.” Senator Gillibrand asserts that commanders are failing to correct the “epidemic” since “by nearly every measure things are moving in the wrong direction.”
While many points Senator Gillibrand makes in the blog post warrant further critical scrutiny, both the full essay I recently published and this summary of the essay focus on the very first thread in the tapestry she describes in support of the measure that would divest commanders of disposition authority for selected offenses. That is, that the military experiences an “epidemic of sexual assault” since an “estimated 20,500 service members are sexually assaulted each year.”
This statistic – an estimated 20,000 service members are sexually assaulted each year – has emerged as the cornerstone upon which the case to divest military commanders of court-martial disposition authority is built.
A current page on Senator Tammy Baldwin’s website, for example, claims that “almost 21,000 service members were sexually assaulted in 2018 despite repeated efforts to end [the] scourge of sexual harassment and assault in the military.” Likewise, Representative Jackie Speier recently opened a press conference in support of the House version of the draft legislation by asserting, “We’re here today because each year 20,000 service members are sexually assaulted.” A recent video op-ed posted to The New York Times website displays “20,500” to accompany an assertion by Col. (ret.) Don Christensen, president of the advocacy group Protect Our Defenders, that “thousands of service members report being raped or sexually assaulted each year.”
So, the claim is clear: Each year, an estimated 20,000 service members are raped or sexually assaulted – and military justice reform is a central part of the solution to this “epidemic.” But what is the source of this claim, and how useful are these statistics as a measure of the performance of the military justice system?
Identifying the Source of the Prevalence Estimate
In short, the source of the claim that approximately 20,000 service members are raped or sexually assaulted each year can be traced to a biennial survey that was last administered in 2018. The 2016 and 2018 iterations of this survey were administered and analyzed by the DoD Office of People Analytics. For 2016, the top-line estimate of sexual assault prevalence was about 14,900, and the estimate for 2018 was about 20,500.
Rand Corp. administered the survey in 2014, and the top-line estimate that emerged from that iteration is about 20,000. In what is described as a “fact sheet” posted by Protect Our Defenders in 2015, the organization notes that “the sexual assault rate remains the same as 2010.” Note that the “fact sheet” describes this as the sexual assault rate, because this is precisely how the statistic is used by core proponents of the measure to divest commanders of court-martial disposition authority.
The problem, though, is that the “estimate” derived from the biennial survey does not present a close approximation for the actual rate of sexual assaults. How might one determine this fact? It’s easy – the survey report says so…on the first page.
Both the 2016 and 2018 iterations that were administered by DoD OPA are referred to as the Workplace and Gender Relations Survey of Active Duty Members, or WGRA. For ease of reference, the full essay primarily analyzes only the 2018 survey report since it is the most recent.
As the 2018 survey report notes, because a survey does not investigate “the specific outcomes of particular allegations” and because “all references to ‘experiences’ of sexual assault…are based on behaviors endorsed by respondents’ self-reports,” the survey affirms that “references to ‘sexual assault’…do not imply legal definitions.” This is the first indication that the survey report does not present an estimated prevalence of actual sexual assault offenses. It says it doesn’t – on page 1.
With this initial disclaimer from the survey report in mind, what are some of the central limitations inherent in using the report as a precise estimate of actual sexual assault offenses or as a useful measure of the effectiveness of the military justice system? The full essay explores several, which are summarized in the following section.
Describing Some Central Limitations
One central limitation is the low response rate relative to the entire active-duty population eligible for the survey. Out of 1,285,990 active-duty DoD servicemembers in the paygrades of E1 to O6 (the population eligible for the survey), the voluntary survey was made available to 694,441 service members.
Of that sample size, responses were received from only 102,109. As a function of the total eligible population, this equates to a response rate of just 7.9%. The survey administrators adjust the response rate slightly using a formula that is summarized here, but even the slightly-higher-than-actual weighted response rate amounts to only 9% of the total eligible population.
As a function of the total eligible population, this decidedly low response rate introduces a significant risk of statistical bias as the total number of responses is expanded to cover the entire eligible population. The full essay explores a number of factors that present a significant risk of statistical bias, and these factors are summarized in this section.
First, as the full essay analyzes, the survey can only measure perspectives of survey respondents based on self-reports. For some types of sexual assault offenses, the perspective of the alleged offender is obviously criminal.
For others, such as sexual assault when the potential victim is incapable of consenting to sex due to alcohol intoxication and the alleged offender knew or reasonably should have known of this condition, the perspectives of both participants are incredibly important in the endeavor to determine whether the sexual act was criminal. The WGRA report, by design, can only capture the perspectives of survey respondents; a determination of verifiably criminal misconduct requires a more wholistic approach in many circumstances.
A second limitation inherent in purporting to use WGRA survey results as a proxy for the prevalence of actual sexual assault offenses is that the report cannot control for the “unique military context” that is characteristic of the surveyed population. This limitation is analyzed here in the full essay.
In summary, though, “individual, social, organizational, or environmental factors” contribute to differences “in terms of both victimization and perpetration.” The inability of the WGRA survey to control for these factors means that top-line estimates may be distorted as the survey responses are expanded to account for the entire active duty population.
A third limitation inherent in purporting to utilize the WGRA survey results as an approximation of sexual assault prevalence is a phenomenon known as non-response bias. This is a statistical bias that can be introduced into survey results when a particular subset of the population is more likely to respond to the survey.
Non-response bias is analyzed here in the full essay, and my analysis is consistent with similar observations in other studies – both in relation to the WGRA report and to a study of sexual assault prevalence in the Canadian Armed Forces.
These are some of the central limitations that are inherent in claiming that the WGRA report represents an estimate of the prevalence of actual sexual assault offenses. These limitations, of course, do not impugn the findings reflected in the report since the report does not purport to present an estimate of actual offenses.
Misusing the data to claim that the survey report does present an estimate of actual prevalence, however, is not supportable by either the methodology or the actual underlying data. This assertion – that an estimated 20,000 actual sexual assault offenses occur in the military each year – has long been the cornerstone upon which the movement to divest commanders of court-martial disposition authority is built. The claim is not supported by the data – it never has been.
Usefulness in MJ Reform Discussion and the Way Ahead with Prevalence Data in Proper Context
With these limitations in mind, how useful is the WGRA prevalence data as a measure of the effectiveness of the U.S. military justice system? To borrow an observation from the full essay: “In short, the WGRA survey reports are of extremely limited probative value as a measure of the effectiveness of the military justice system. This is true regardless of whether disposition authority for sexual assault offenses is vested in commanders or military lawyers.”
This is the case because the low response rate relative to the entire eligible population coupled with several identifiable factors that introduce a significant risk of statistical bias mean that the survey report in no way presents a precise “estimate” of sexual assault prevalence. Likewise, the WGRA survey report does not present – nor does it purport to present – data involving actual sexual assault offenses, which of course is the domain of the military justice system.
Finally, as the full essay describes further, a significant proportion of the affirmative survey responses involve misconduct that quite likely would not be referred to court-martial – regardless of whether a commander or a lawyer has that authority. There are methods by which the effectiveness of the commander-centric military justice system can be measured, but the WGRA survey reports are most decidedly not one of them.
Incidentally, some studies that are better tailored to measure the performance of the military justice system have been conducted. The most recent of these, published in October of last year, determined that “there is not a systemic problem with the initial disposition authority’s decision either to prefer a penetrative sexual offense charge or to take no action against the subject for that offense.”
This study, conducted by the Defense Advisory Committee on Investigation, Prosecution, and Defense of Sexual Assault in the Armed Forces, analyzed actual casefiles that were initially considered by commanders. The authors of that report concluded that “in 94.0% and 98.5% of cases reviewed, respectively, those decisions [by commanders] were reasonable.”
Combining these findings with the analysis of the usefulness of the WGRA survey report data as a measure of the commander-centric military justice system supports the conclusion that members of Congress, and the constituents they represent, must fundamentally reevaluate what they have come to believe about the prevalence of sexual assault in the military and the performance of the military justice system in responding to sexual assault offenses.
The same is true, incidentally, for U.S. service members – all the way up the ranks. As CJCS Gen. Mark Milley recently observed, for example, “If we had 20,000 casualties in Afghanistan last year or 20,000 casualties in Iraq, we’d be taking some pretty radical actions to correct that.” If victims of sexual assault are assessed as casualties, the “estimated” casualty rate of about 20,000 per year is not supported by actual data. It isn’t now, and it never has been.
My next step is to publish the findings of my study of the specific, calculated techniques by which the WGRA survey report data (and previous versions of the survey) have been misused over the years to support the movement to divest commanders of court-martial disposition authority. I am also finishing a critical assessment of the DoD Independent Review Commission report that convinced Secretary Lloyd Austin and President Joe Biden to endorse the measure to remove court-martial disposition authority from commanders in relation to sexual assault offenses and “related crimes.”
For now, perhaps the analysis of the source of the prevalence “estimate” at the center of the military justice reform movement can provide some clarity to the debate involving military sexual assault prevention and response and involving court-martial disposition authority. Important decisions lay ahead involving the future of the U.S. military justice system. These decisions must be based on a full understanding of the available data, even if such decisions have not been incredibly well informed in the past.
About the author:
Brian L. Cox is an adjunct professor of law at Cornell Law School and a visiting scholar at Queen’s Law in Ontario. Brian retired in 2018 from the U.S. Army after 22 years of military service. He served as an airborne infantry soldier, combat camera operator, airborne infantry officer, and for seven years as a military legal advisor. His combat deployments include Iraq from 2003-2004 as a combat camera operator and Afghanistan from 2013-2014 as the chief of international and operational law for Regional Command-East in Afghanistan.
Brian also served as a military prosecutor, federal prosecutor, brigade judge advocate, and military magistrate while he was a military legal advisor. His military awards, decorations, and qualifications include the Ranger Tab, Senior Parachutist Badge, Pathfinder Badge, Air Assault Badge, Bronze Star Medal, Meritorious Service Medal, Basic and Advanced Collateral Damage Estimation Certification, and Joint Firepower Certification. Brian holds an LL.M. from Queen’s Law and a B.A. (International Relations) and J.D. from the University of North Carolina.
The views expressed by guest authors do not necessarily reflect the views of the Center on Law, Ethics and National Security, or Duke University.
Remember what we like to say on Lawfire®: gather the facts, examine the law, evaluate the arguments – and then decide for yourself!
