The first problem with this is that if you build a 100% honest model of re-offending probability, what you're building is a model of your own system, not of the person. For example: If you lock someone in prison for a few years, offering them no training or rehabilitation, and then upon release have various penalties on them which basically prevent them from getting a job -- ranging from the simple "nobody wants to hire someone with a conviction record" to "explicit legal bars to their getting certain kinds of job, living in certain areas, etc" -- then it will probably not surprise you that this person is significantly more likely to turn to a professional life of crime. A model which correctly recognized and predicted this would therefore conclude that the only solution is to lock this person up for life, since at any point after they're released, they're simply likely to become criminals again.
This highlights the deeper problem in such a model, of course, which is that its basic design parameters, where the only variable it controls is "imprison more" or "imprison less," create a false dichotomy: rather than asking "which course of action is most likely to lead to the person no longer engaging in crime," it only considers one possible course of action, and that action (again, by the design of the system) most often increases the probability of future crime.
The criticism of this system that it will end up encoding implicit racial biases is only sort-of correct. This model will definitely end up having a strong racial component; even if you eliminate race as an input, your race is so strongly correlated to other things like where you live that the system will end up modeling your race, and basing its decisions upon that, one way or the other. And that will, indeed, end up increasing sentences for Black and Latino offenders, for all the reasons specified above.
But in this case, the racial biases which the system would acquire are simply one manifestation of the even deeper and more profound problem that this model is simply designed to optimize for throwing people into prison.
If you want a variation on this which actually works, give the model access to a wide range of possible consequences, and ask it which of those will minimize the odds of re-offense, presumably balanced against various costs. You'll almost certainly find that rehabilitation, training, and treatment overwhelmingly work best to minimize that. (And in the cases where they don't, your best bet is likely to simply take them out and shoot them)
I would actually quite strongly favor such a project, because it would require its creators to make very explicit the thing for which they are trying to optimize. You can't lie to a computer about what you want it to do; if you want to minimize the chance of re-offense, you have to tell it to do so. If you instead want to optimize the system for retribution, or to cow a broad population into submission, or to maximize revenue, the model will absolutely be able to do that as well -- but you would have to tell it explicitly to do so, and it's very hard to lie to yourself about that.
h/t over on Twitter for prompting me to actually write about this one.