Open Data Projects Are Fueling the Fight Against Police Misconduct

A growing number of accountability groups are aggregating and distributing police misconduct databases on their own. 

NEW YORK, NY - JULY 02:  New York City Police Academy cadets attend their graduation ceremony at the Barclays Center on July 2, 2013 in the Brooklyn borough of New York City. The New York Police Department (NYPD) has more than 37,000 officers; 781 cadets graduated today.  (Photo by Andrew Burton/Getty Images)
New York City Police Academy cadets attend their graduation ceremony at the Barclays Center on July 2, 2013, in New York. Photo: Andrew Burton/Getty Images

Samantha Seda’s client, a 16-year-old foster child from Far Rockaway, New York, had no criminal history when he was arrested in September, accused of having pulled out a gun and fired one shot in the air. Even though he had no priors and no relatives who could post bail, a judge set the amount at $100,000, and as he sat in jail for over a month, the boy lost his spot at the foster home where he had been living.

Seda, a Legal Aid attorney representing adolescents charged as adults in Queens, thought the allegations against her client were dubious and was looking for a way to get him out on bail. That’s when she decided to look into the officers named in the complaint against him. What she discovered stunned her.

The arresting officer, she learned, had been sued several times, and in the 1990s, he had been part of a group of officers working a narcotics operation that was accused of planting drugs on people and stealing drugs from suspects. Some of the officers went to trial and were convicted on felony charges, but most settled, costing the city some $1.2 million in damages to their victims. The officer she was researching was acquitted in court, but he had been named in connection with at least nine separate misconduct cases and had settled at least two, she told The Intercept.

Intrigued, Seda looked into the second officer, a sergeant who claimed he had watched her client pull out a gun and shoot it. He was also named in a pending lawsuit, in which a driver alleged the officer had stopped him, assaulted him, and arrested him with no legal justification.

“I thought, wow, that’s outrageous. Two cops in one case that are dirty,” Seda said. When she looked into the third officer, another sergeant who said he had found a gun on the street after her client ran, she was almost expecting to find something.

And in fact, that officer had also been sued, for assaulting a young black man moments after he walked out of his sister’s house. That lawsuit alleged the officer had thrown the boy to the ground and “roughed him up” before arresting him and taking him to the police precinct, also without legal justification. The boy was never charged with anything — but his family sued the city, demanding $100,000 in damages.

The accusations against the officers didn’t automatically imply their guilt in those cases, Seda said, but they certainly raised red flags.

Seda took her findings to a judge and argued that they undermined the credibility of the officers in question — and he promptly dropped her client’s bail and released him until trial. But only months earlier, the case might have played out quite differently.


A map produced from Legal Aid Society data shows the incidence of lawsuits against the NYPD between June 2015 and May 2016, broken down by demographics.

Graphic: The Legal Aid Society

Tracking Police Misconduct

In the past, when Seda wanted to learn more about an officer connected to one of her cases, she would scour legal databases and news archives in search of any relevant information. If she somehow knew an officer’s history would matter to the case, she might file a public records request for complaints filed against him or try to subpoena public agencies for that information — a cumbersome process for a public defender handling dozens of cases, and one with no guarantee of success. “It would have been very hard and it would have taken months,” she said. “It may have never happened.”

Most police misconduct goes unreported, particularly in less extreme cases and in more disenfranchised communities, but complaints filed with police departments and civilian review boards, as well as lawsuits, can point to significant histories of abuse tied to specific officers and precincts. In most cases, however, a citizen who becomes the victim of police abuse has next to no way of knowing if that officer is a repeat offender or has a history of targeting certain people, say, or sexual harassment.

As is the case with most police departments across the country, the NYPD does not disclose internal disciplinary records to the public. Even though cities spend millions in public funds to settle lawsuits filed against officers, the public has little access to what the settlements reveal about problematic officers and precincts. Meanwhile, the officers themselves rarely face consequences and often return to the streets quickly, their histories shielded in anonymity.

But that situation is beginning to change — as a growing number of police accountability groups are starting to bypass the departments by aggregating and distributing misconduct history databases on their own.

Earlier this year, Seda was trained in using the Cop Accountability Project, a database created by New York’s Legal Aid Society that pools civil rights lawsuits, criminal court decisions, and a variety of other public and private sources like attorney notes and social media content to compile misconduct profiles on nearly 9,000 New York City officers. The database assembles a wealth of information that could otherwise take months to gather, as well as some that wouldn’t be available anywhere else, and it has proved to be a game-changer for the attorneys using it.

In the past, individuals could file public records requests with the Civilian Complaint Review Board, which is normally the first recipient of citizen complaints regarding police use of force, abuse of authority, and other misconduct. But since October 2014, when the New York board’s executive director was fired and accused of releasing records protected by law, the CCRB has denied all public records requests on the grounds that they violate New York’s strict privacy protections for law enforcement officers. A spokesperson for the board declined to comment on the change in policy or the former chief’s departure but wrote in an email to The Intercept that the CCRB “believes that transparency and accountability builds trust between police and the community it serves.”

The CCRB also recently launched its own Data Transparency Initiative, an interactive online data tool that includes information on more than 190,000 allegations of police misconduct, involving more than 63,000 victims and some 36,000 NYPD officers. But unlike the Legal Aid database, that data paints only a macro picture of the issue and cannot be connected to individual officers or incidents. Legal Aid’s Cop Accountability Project began as an informal, handwritten list of officers Legal Aid lawyers knew had a history of misconduct. Over the years, as lawyers amassed information to help build stronger cases for their clients, the list grew into a spreadsheet, then eventually into a cloud-based relational database that recently became available to Legal Aid attorneys through a mobile app.

“We are essentially trying to find a way to collect data to document events that government doesn’t want documented in a public way,” Cynthia Conti-Cook, one of the curators of the database, told The Intercept.

Although it keeps growing every day, the database is by nature incomplete, Conti-Cook notes, and lacks access to any internal documentation the NYPD may have on its officers. In fact, Legal Aid is currently fighting the city in court to open up officer misconduct files to public records requests. The group has also been fighting a New York law known as “50-a,” which protects officers from exactly the kind of scrutiny and accountability the public is demanding by guaranteeing the confidentiality of all law enforcement personnel records, essentially blocking any judicial review of officers’ histories and possible patterns of misconduct.

In a statement released last week, Mayor Bill de Blasio outlined some proposed amendments to 50-a. The NYPD did not respond to a list of questions from The Intercept but referred us instead to a recent official statement by Commissioner James O’Neill expressing support for de Blasio’s proposed reforms. “I believe in transparency. I also believe that making information about disciplinary proceedings public will help us build trust with the community,” O’Neill said.

But police accountability advocates were quick to point out that the proposed changes were just a way to kick the can down the road; they were “not substantive” and fell short of a “genuine commitment to full transparency.” “If the administration is serious about police accountability,” the New York Civil Liberties Union wrote in response to the mayor’s statement, it will “just start releasing records.”

Until that happens, the Legal Aid database remains the most comprehensive accountability tool available to lawyers.


A map produced from Legal Aid Society data shows the incidence of lawsuits across the city’s boroughs between June 2015 and May 2016.

Graphic: The Legal Aid Society

“What the Legal Aid Society is doing will not replace legislative reform, no matter how large its database grows,” Conti-Cook wrote. “Defender-driven data, while filling a gap in access to records, is not the ultimate solution.”

Yet for all its limitations, the database has already achieved a lot. Lawyers in Legal Aid’s network have successfully used it to “change the narrative in the courtroom about what happened during a specific encounter between a client and an officer,” Conti-Cook explained in an academic paper presenting the project. “Expanding the definition of police accountability data from official disciplinary complaints to other sources that similarly document misconduct events has changed who controls the definition of misconduct, and therefore who controls the narrative of what is happening between police and the communities they serve.”

And those victories haven’t stopped in the courtroom. In response to the Legal Aid database, the NYPD itself has expanded the definition of misconduct it uses to manage risk internally to include allegations raised in lawsuits in addition to the department’s internal affairs bureau and the city’s civilian review board. These investigations remain inaccessible to the public, but they are now informed by a broader set of sources.

Perhaps most importantly, beyond tracing individual officers’ histories of misconduct and singling out the so-called bad apples, the Cop Accountability Project has also highlighted broader trends in New York’s police-community relations.

The database revealed, for example, that between June 2015 and May 2016, a single Brooklyn precinct — the 75th Precinct in East New York — was sued in federal court at least 47 times, more than double the amount of any other precinct. The second most sued precinct was the 73rd, in Brownsville, Brooklyn. In total, over that time frame, NYPD officers were sued in federal court 966 times. The NYPD is sued about 4,000 times a year, mostly in state courts — a dramatic increase in litigation that in 2014 alone cost the city $216 million in settlements. The increase in lawsuits marks an opposite trend from that noted by New York’s Civilian Complaint Review Board, which reported that from 2006 to 2015, the number of complaints steadily declined, from 7,663 in 2006 to 4,461 in 2015 — possibly suggesting that a growing number of citizens, frustrated with the redress process available to them, might be resorting to suing the police.

The Legal Aid database also showed that the 10 precincts with the highest numbers of lawsuits were concentrated in Brooklyn. Citywide, 82 percent of plaintiffs filing lawsuits were black, while only 2 percent were white. Thirty-three percent of the lawsuits were filed over encounters that took place on the street, and 47 percent alleged excessive use of force, with 64 percent of those cases requiring hospitalization.

The analysis also gave a breakdown of the charges officers made against the plaintiffs, which were overwhelmingly for “resisting arrest” and “disorderly conduct,” two of the vague charges regularly used in questionable police stops.

Legal Aid made those findings public as part of a push for greater police transparency, but the bulk of the database is currently available only to lawyers within the group’s network. As word of the database’s existence spread, other attorneys began reaching out to Legal Aid for access to its data, sometimes contributing more information from their own records. In one case, a judge even suggested that an attorney seek access to the database, prompting the officer’s attorney to also demand to see the file Legal Aid had on him.

Conti-Cook said the entire database has not been made public because it is built, in part, on confidential sources and information that’s protected by attorney-client privileges. But the goal, ultimately, is to distill the majority of the data that is in the public record and open up access to all New Yorkers.

“We’re dreaming of a website,” she told The Intercept.

Thousands take part in the Justice for All March and Rally on Pennsylvania Avenue to the US Capitol in Washington, DC, on December 13, 2014, to protest the killings of unarmed African-Americans by police officers and the decisions by Grand Juries to not indict them. AFP PHOTO / SAUL LOEB        (Photo credit should read SAUL LOEB/AFP/Getty Images)

Thousands take part in the “Justice for All” march on Pennsylvania Avenue in Washington, D.C., on Dec. 13, 2014, to protest the killings of unarmed African-Americans by police officers and the decisions by grand juries not to indict them.

Photo: Saul Loeb/AFP/Getty Images

A Killer Cop With a Past

On October 20, 2014, a group of Chicago police officers shot and killed a black teenager named Laquan McDonald. Police said that the officers fired in self-defense, the boy had a knife, and he was killed by a single shot to the chest. For over a month, the official narrative was the only narrative.

It was not until a whistleblower tipped off journalist Jamie Kalven that the story was false and there was “horrific” video to prove it that the real narrative began to emerge. Kalven obtained a copy of the autopsy report, which showed that McDonald had been shot 16 times, including several times in the back, and that police officer Jason Van Dyke had unloaded his weapon “execution style” while McDonald lay on the ground. A month later, the city settled with McDonald’s family for $5 million before they even moved to sue.

It took several more months of protest and public pressure before details of the official cover-up were revealed, and more than a year for the city to release dashcam video of the shooting. Finally, Van Dyke was charged with first-degree murder. Officials then sought to paint him as a lone “bad officer” whose actions shouldn’t reflect on the department.

Eventually, it emerged that Van Dyke had been accused of misconduct at least 17 times before he killed McDonald, including several allegations of brutality. Yet none of those complaints — one of which cost the city a $500,000 civil settlement — had resulted in any disciplinary action. Between 2012 and 2015, the city of Chicago paid $210 million in police misconduct settlements — with just 124 of the city’s roughly 12,000 police officers accounting for $34 million in payouts. The Chicago Police Department did not respond to a request for comment.

Public disclosure of Van Dyke’s long complaint history was made possible by litigation brought by Kalven against the city of Chicago back in 2007. It took seven years of litigation for an Illinois Supreme Court judge to rule that documents bearing on allegations of police abuse, including citizen complaints, were public information and therefore subject to public records requests. Kalven’s group, the Invisible Institute, whose mission is to hold public institutions accountable, quickly moved to obtain records from Chicago’s Independent Police Review Authority as well as the department’s internal affairs division — which came to some 56,000 misconduct complaints against 8,500 police officers since 2011.

But the Invisible Institute didn’t stop there. Driven by a commitment to the notion that information about public institutions should be open, free, and accessible to all, the group processed all the complaints and then published them on a user-friendly, interactive website.

“We positioned ourselves as advocates for data, transparency, and accountability,” Rajiv Sinclair, who works on the database, told The Intercept. “We share all the data with everyone.”


Charts and map visualizations generated from police data obtained by the Invisible Institute.

Image: Citizens Police Data Project

As is the case with New York’s Legal Aid database, the Invisible Institute’s Citizens Police Data Project allows people to search for the complaint records of individual officers, but it also tells broader stories about police trends in the city. For instance, the data shows that young black men on the South and West sides of Chicago consistently file more complaints than any other group in the city — but it also shows that their complaints have a much lower “sustain rate,” Sinclair said, meaning they are much less likely to be investigated and followed up on.

That’s partly because Chicago police will not investigate a complaint until the citizen filing it signs a sworn legal affidavit — a stressful prospect for many whose interactions with law enforcement are defined by mistrust. As a result, nearly 60 percent of complaints are thrown out. A quarter of complaints were dropped because citizens filing them couldn’t identify the officers. A new data initiative launched last week, OpenOversight, is hoping to fill that gap by providing a database of officers’ photos and badge numbers.

“There’s a lot of intimidation in that process,” Sinclair noted, referring to the affidavit requirement. “Many of those cases are going to be an officer’s word versus a citizen’s word. And if you’re a black teenager, you know how undervalued your word is, you know that if you say one thing and the officer says another thing, they’re just going to say that you’re lying.”

Whose Data Is Public Data?

The Invisible Institute plans to expand the scope of its project to add new data sources like use of force records, tactical response reports, and a wider set of misconduct complaints, but it also hopes to combine those with more contextual data around individual officers. The ambition is to be able to follow problem officers through their careers, to see how patterns change as they move between units and commanders, and to explore the intricate networks of relationships that model the Chicago Police Department’s behavior.

But the primary goal remains easily accessible accountability. The group recently developed a Twitter bot that extracts officers’ names from news stories posted on Twitter and automatically responds to followers with those officers’ profiles and complaint histories. They also run workshops based on the database with high school students in the city’s most heavily policed neighborhoods — the same kids “who gets stopped by police every day,” Sinclair said.

“Those users are the most important to us, and they have been underserved by the predominant trends in civic data transparency and police data transparency,” he added.

In fact, as open data initiatives have multiplied in recent years, some government agencies have gestured toward transparency by releasing data sets for scholars and analysts to report on broad social trends. But most of that data, like the data released by New York’s Civilian Complaint Review Board, is “de-identified,” meaning it can’t be traced back to specific officers. And ultimately, for the people whose encounters with police build up those data sets, the big picture matters less than the name and history of the officer who abused them.

“A lot of data is very disconnected from the people who we think are the most important users, and those are citizens who have had encounters with police and their lawyers,” Sinclair said. “One of the most important uses for our data set is for lawyers and citizens to be able to look up an officer. … Data can support your case, make it more clear that what happened to you is true.”

A similar initiative, which focuses on officers’ patterns of traffic stops rather than complaint histories, is North Carolina’s Open Data Policing NC. The project is made possible by North Carolina’s uniquely detailed reporting requirements for traffic stops, which require officers to file detailed reports on all stops, including reasons for the stop, details on whether a search was conducted, as well as the race, age, and gender of the driver.

That data was always public, but like much public information, it was not readily accessible, essentially voiding its public utility.

“There’s nothing that I’m publishing that you couldn’t have always gotten,” Ian Mance, an attorney with the Southern Coalition for Social Justice and curator of the database, told The Intercept. “But if you requested the database from the state, it would be such a large file it would literally melt down your laptop if you tried to open it.”

Until the website launched — covering 25 million traffic stops made since 2002 by the state’s 300 largest enforcement agencies — police were required to compile the data, but then “it kind of just disappeared,” Mance said. Departments never looked at it and didn’t have systems set up to analyze the data from officers’ stop patterns, he said, adding that he’s been trying, with some success, to convince police departments of the site’s benefits, pitching it as an “auditing tool” they can use for internal reviews.

While the database doesn’t publish individual officers’ names, it lists their ID numbers — making it possible for a citizen who was pulled over to identify his stop by department, date, or other circumstantial data, and then, through a hyperlink on the officer’s ID, to populate a page with his “career enforcement data,” including any patterns suggesting bias.

“So if you are a driver who suspects you’re being stopped for an illegal, race-based reason, you can test your hypothesis by going on the site,” Mance explained.

Like the Chicago and New York initiatives, the North Carolina project quickly revealed broader patterns of enforcement, including racial disparities in traffic stops. And the data has already informed some policy changes. In Greensboro, for instance, city officials changed their protocols to end stops for low-level traffic violations, and in Durham, the city council passed a policy mandating that officers get written consent for searches not justified by probable cause.

Protesters march in Charlotte, North Carolina, on September 23, 2016 following the shooting of Keith Lamont Scott by police three days earlier and subsequent unrest in the city.Hundreds of protesters were out again on Friday night calling for the release of the videos amid a greater presence of National Guard troops, but the atmosphere was calmer than during previous days. / AFP / NICHOLAS KAMM (Photo credit should read NICHOLAS KAMM/AFP/Getty Images)

Protesters march in Charlotte, North Carolina, on Sept. 23, 2016, following the shooting of Keith Lamont Scott by police three days earlier.

Photo: Nicholas Kamm/AFP/Getty Images

As with Chicago and New York, the data has also been used in court. “We’ve had cases where there were terrible claims of racial profiling and officers weren’t able to offer a coherent explanation of why a certain person came to their attention and why they stopped them,” Mance said. “Their career enforcement data became immediately relevant to answering questions of whether they targeted this person for illegal, race-based reasons.”

“Open data is the future, there’s no putting the cat back in the bag,” he added. “This is something police departments, whether they like it or not, are going to have to contend with, because we’re reaching a point where the public expects to have access to this kind of information.”

Yet many data projects are ultimately contingent on agencies’ cooperation — either willingly and because local legislation declares the data a public record, or after long-fought legal battles like Kalven’s.

As demands for accountability increase, so does pushback. As police departments across the country started to enroll in body camera programs, local legislatures also began to discuss bills to keep that footage secret. In Georgia, legislators moved to shield bodycam footage from public records requests and extended the restriction to officers’ misconduct records that were previously available. In New York, the Civilian Complaint Review Board’s reversal of its policy of disclosing records upon request came just months after the killing of Eric Garner sparked widespread protests in the city and demands for more accountability.

North Carolina, despite having one of the most transparent traffic stop reporting requirements in the country, is no exception. Last month, for instance, the Charlotte-Mecklenburg Police Department faced an onslaught of criticism for its refusal to release video of the police killing of Keith Lamont Scott. Scott’s killing came just days before a revision to North Carolina’s public records law — allowing police to withhold video like that of Scott’s killing from the public — was set to kick in.

“I think these efforts are very shortsighted and people see them for what they are,” Mance said, “which is an attempt by people in power to keep the public from having a better understanding about the way that they are being policed.”

A National Clearinghouse for Police Complaints

So far, efforts to use open data to track officer misconduct and hold police departments accountable are few and limited. But with 18,000 law enforcement agencies across the country, each ruled by different sets of local regulations, there is no uniformity in how police misconduct is tracked, if at all, and in a majority of places, the public has no access to records of complaints filed against police.

That might be about to change, as a civil rights group is preparing to launch a nationwide website that will allow users to file official reports of officer misconduct and abuse, while also making those reports immediately public. For years, Flex Your Rights, the group behind the Open Police Complaints initiative, ran “constitutional literacy” workshops, educating the public on their rights during police encounters. But they soon realized that wasn’t enough.

“The events of the last few years have shined an important light on the fact that people are going to have bad police encounters regardless of whether they ‘flex’ their rights perfectly or not,” Steve Silverman, the group’s founder and director, told The Intercept.

Instead, the new website, which is set to go live early next year, will aim to bring some uniformity and accountability to the police complaint process, by compiling research on different departments’ requirements and allowing users to either file complaints directly online — for those departments that choose to participate in the initiative — or upload the complaints they file on their own and share those reports publicly.

The idea is to simplify the complaint process for citizens and minimize the need for people to file grievances in person at a police station, “where they’re bound to have another bad encounter,” Silverman said. The site will populate a report by asking users dozens of questions and will offer varying degrees of transparency, allowing users to publicly disclose anything from bare, anonymous details of the incident to full narrative accounts including names of officers and victims. It will also give police departments an opportunity to respond and update each report with the findings of their investigations.

“We’re trying to use open data in order to encourage better responsiveness but also to track how well and how poorly departments are responding to complaints, as one of the big reasons why police complaints processes suck is because they’re very unresponsive,” Silverman said. “Of course there are going to be some police departments and individual officers who chafe at the idea that their name might be attached to a public complaint. But if that’s the case, they should go ahead and properly and swiftly investigate the complaint and they’ll have an opportunity to post their findings.”


The Open Police Complaints web app, scheduled for a 2017 release.

Image: Open Police Complaints

The group said it would protect the site from misuse and false reports — and potential libel suits — by requiring that all police complaints be officially filed before they can be shared publicly, and by flagging “frequent flier” users and filling the site with disclaimer notices reminding readers that the reports are allegations. It will also offer tools to connect citizens filing complaints with attorneys in their areas, as well as to other relevant resources.

“We hope that by providing this kind of information in a public way, we’ll urge departments to take action,” Silverman said.

But even if departments refuse to engage with the site — and Silverman and Morgan Lesko, the project’s main developer, expect it will take a few years to compile a significant volume of complaints and get departments to take notice — the reports will be accessible and easily searchable by anyone.

“In the worst case scenario — that we don’t get appropriate feedback from police departments — we still have these collected online,” said Lesko. “And it’s all published in one place; it’s not just lost on Facebook somewhere, scattered around in rant form. It’s a public record.”

Join The Conversation