This is the third part in a four-part series Recidiviz has written to document the use of technology in the criminal justice ecosystem and share specific recommendations for practitioners in tech procurement, integration, and deployment (see Part I, Part II). As always, we welcome thoughts and feedback.
The criminal justice system is awash with new tech tools and inherently a difficult place to establish data-driven feedback loops. Practitioners looking to use technology must wade through a dizzying array of options to determine what to use, when to upgrade, and how to implement changes and track impact. Drawing on both what we’ve learned from building Recidiviz and successful practices from the private sector and government, we created this guide to help criminal justice practitioners procure and deploy technology effectively.
Before looking at technology solutions, we should clearly articulate a) the need we’re trying to address, b) the people who would use the solution, and c) how it could fit into their workflow.
For example, imagine our goal is to increase the efficiency of re-entry case managers to reduce deferred parole hearings. It might be tempting to jump straight to looking at checklist or project management software, or designing something in-house. But this skips to the solution, before we really understand the problem. If the main issue holding back re-entry readiness is that vocational and rehabilitation is often required for parole, but only available in certain facilities, there are a lot of options to explore to fix the issue, and many don’t require new tools at all — e.g., expanding programming availability across facilities might help more than a to-do list tool would have. Another common issue is understanding the problem, but not how our approach would fit into existing workflows: we might rush to procure a great to-do list app, only to find that case managers only have time to check their to-do’s during lunch time, when away from their computer, and the app we procured isn’t built for smartphones. Knowing the right place to intersect with their workflow might have helped us find the right solution.
One area to be especially wary is with shiny new technology, like blockchain or deep learning, where exciting buzzwords can obscure that the tool doesn’t actually address the problem at hand. Thinking about the problem, and the user, rather than starting the solution can help weed out costly missteps and lead to more adoption within the agency. Without adoption, shininess doesn’t matter much. As an added bonus, this process will also lead to a better RFP that treats user-experience as a first-order priority.
Custom-built solutions may seem like a good way to meet a specific or immediate need, but they can lead to negative consequences in the long term. They also make it less likely that the new system will ever be able to communicate with other parts of the justice system. When products and data are integrated across agencies, rather than in individual silos, we can look at improvements and impact more holistically.
We should ask questions like: Does the new bodycam software output videos in a format that is well-supported by the police department’s database, or with other video software? Can other organizations, e.g. correctional facilities, easily read or access information from the probation office’s new record management system?
Choosing a custom solution also makes it more difficult to switch vendors if product requirements change down the line, leading to lock-in. When only one vendor in the world knows how your software works, you’ll only ever be able to work with that one vendor. Before committing to any tool, and especially a highly custom one, the following considerations may help to improve long-term efficacy:
It is often a red flag if a tech vendor can’t produce a clear answer to how a tool works or how an algorithm weighs specific factors in a predictive decision. This is unfortunately still common — many companies have historically resisted disclosure, calling the workings of their tools protected trade secrets.
Take the example of probabilistic genotyping software, which is used by investigators to predict the probability that a genetic sample came from a specific individual. Practitioners should ask: In what types of cases will the software likely fail? What kind of mistakes does it make (e.g., can it fail in both directions — saying the samples came from the same person in cases where they didn’t, and other times saying they didn’t come from the same person when they actually did)? What are the consequences of these mistakes (e.g. leading to punitive outcomes or opportunity denial)? How often do mistakes happen? If the tool’s output is challenged, do I (or the vendor) have enough information to determine and explain what went wrong?
Technology has the potential to make the criminal justice system more fair and efficient. However, a growing body of research has shown that certain types of technology can be discriminatory. Luckily, we can assess this up-front and use what we learn to inform procurement decision-making.
All tools, especially those that use machine learning, should be assessed for performance (i.e. accuracy) across protected characteristics (i.e. race, gender, age, sexual orientation, and socioeconomic status). A good example is the set of analyses the National Institute of Standards and Technology conducted for facial recognition software, evaluating 189 algorithms from 99 different developers for bias across race, age, and sex. It’s a good sign if a tool has already been evaluated by another agency or academic group that has no financial ties and a history of objectivity.
For tools that benefit justice-involved individuals, bias can come in the form of denial of opportunity — for example, socioeconomically disadvantaged groups who don’t own smartphones or live in areas with reliable cellular service won’t be able to reap the benefits of court reminder apps. In those cases, we need to think of additional ways to reach these audiences. For example, this might include mail or postcard campaigns, e-mail as a backup to text messaging, or using contacts with parole and probation officers.
Before launch, we can develop a data-driven rollout plan to reduce the risk of failure, track impact, and inform future iterations. Rollouts can be tricky in the criminal justice context — traditional tech experimentation frameworks, like A/B testing, can become ethically questionable when people’s lives, freedoms, and futures are at stake.
At Recidiviz, we use pre-determined “backstop” metrics during our launches to determine when changes should be rolled back (essentially the opposite of success metrics). For example, if a new feature for our line staff tools were to disproportionately lead to fewer positive outcomes for Black individuals on supervision, our metrics would show that as it started to happen, giving us the chance to stop the feature roll-out and make changes before trying again.
Ultimately, every tool will have benefits and drawbacks. Surveillance technologies that use facial recognition can help law enforcement quickly wade through footage and identify potential suspects, but can also violate the privacy and security of those being monitored. If these costs can’t be mitigated, or if the tangible benefits can’t be realized, then it may be time to consider another type of tool, or none at all.
This is an evolving piece. We’d love to hear your feedback and suggestions on the Playbook, especially if you have experience working in this space. Reach out to us at this link.
Recidiviz is proud to be a GLG Social Impact Fellow. We would like to thank GLG for, among other things, connecting us to some of the experts who provided valuable insights showcased in this post.