老熟女午夜福利

In the past year or so, an increasing number of police departments appear to be allowing their officers to use AI large language models (LLMs) to generate a first draft of their police reports based on body camera audio recordings of an incident. As we laid out in a white paper last year, there are a number of reasons the ACLU opposes that practice, but one of them is the unreliability of AI and of the human beings supposedly overseeing it. Thinking further about these deployments of the technology, it becomes clear that AI is actually a trap for the police officers that use it, and for many other workers as well.

As anyone who has spent significant time using LLMs knows, these models are quirky and unreliable. They are prone, for example, to make unpredictable errors or 鈥渉allucinations,鈥� in which they simply make up facts. In an ideal world, officers would review the draft police reports that AI generates for them and correct any errors. But we don鈥檛 live in an ideal world, and it鈥檚 just a fact that many officers aren鈥檛 going to review the drafts that AI generates for them. The output of LLMs intuitively appears to be very smart, but it鈥檚 unreliable just infrequently enough to lull people into complacency, especially those without enough experience to understand its limitations.

The AI trapdoor
Vendors and proponents of these systems place a lot of stress on the fact that it鈥檚 the responsibility of police officers to double-check AI output. The dominant provider of AI-assisted police report products, Axon, makes sure to emphasize that the burden is on police departments and officers to avert problems caused by AI unreliability. 鈥淎s you consider your AI policies, it is important to consider the following things,鈥� , followed by this bullet:

Always ensure the user is in the loop: AI should not replace human decision making, so officers need to be trained and reminded that they are responsible for making the final decisions. You need to understand the strengths and weaknesses of AI, and make sure you leverage its strengths.

Yesterday I wrote about another Axon AI product, 鈥淧olicy Chat,鈥� which uses LLMs to allow officers to search their departments鈥� policy manuals; the company that officers can be provided with 鈥淧olicy guidance in seconds鈥� and 鈥淚nstant answers, confident decisions.鈥� But at the same time, Axon that

Policies may be updated or changed. While efforts are made to keep the information up-to-date, there may be delays or omissions. Always verify with the official policy source. Consult the official policy documents, or your supervisor for the most accurate and up-to-date information.

This is such a trap. The whole promise of their product is that officers won鈥檛 have to go paging through their departments鈥� policy manual, parsing the language like a lawyer, but can just rely upon the LLM to summarize it for them. But at the same time, they鈥檙e trying to disclaim any responsibility for errors the LLMs may make, washing their hands of responsibility for any problems if officers do not abandon the very promise of the product by doing precisely that 鈥� acting like lawyers to 鈥渧erify with the official policy source.鈥�

The average human being in the flow of a busy day is just not going to do that. Axon is promising the world, but throwing the burden on officers should anything go wrong.

Axon has gestured toward solving this problem for its Draft One product by offering purported safeguards such as the intentional inclusion of nonsense language to force officers to review AI output. That鈥檚 at least an admission that there is a problem, but as we in our white paper, the company鈥檚 safeguards are of dubious utility and often aren鈥檛 being used.

B-17 bombers and 鈥渉uman factors engineering鈥�
Experts have long realized that you need to design technology not around some presumed perfect and rational person, but around how human beings actually behave. There is an entire field called 鈥渉uman factors engineering鈥� aka 鈥溾�� that studies this in contexts like , urban planning, and traffic safety.

One example is 鈥渟ocial paths鈥� or 鈥�,鈥� a well-known phenomenon in urban planning. While designers of public space might prioritize things like aesthetics, property lines, or traffic flow, , and they frequently carve their own informal paths. The result is the common appearance of dirt paths across lawns and the like, where people spurn even slightly longer paved routes. (A professional Dutch photographer has created a nice of such paths.) It can be very hard to stop people from taking shortcuts, and urban planners know that signs, fences, and barriers that attempt to steer people along the 鈥減roper鈥� route are routinely evaded and often repeatedly torn down.

Social paths are an example of what happens when designers fight patterns of human cognition and behavior rather than working with them. In the aviation industry, it鈥檚 long been recognized that the design of cockpits and aircraft controls aren鈥檛 safe unless they鈥檙e designed around human limitations. During WW II, for example, over 400 U.S. B-17 鈥淔lying Fortress鈥� bombers were lost in nearly identical crash landings. No malfunctions could be found and the crashes were blamed on pilots. But a psychologist investigating the problem for the Army two separate levers 鈥� one to control the landing gear, and another to control the wing flaps 鈥� were placed near each other and looked identical, and exhausted pilots returning from lengthy and stressful missions were flipping the wrong switch as they were landing. The problem was almost completely eliminated when the two handles were given much different designs (triangular vs. circular) to help unreliable human brains avoid that mistake (a strategy engineers now call 鈥渟hape coding鈥�).

One pithily captures the paradoxical duality of the situation: 鈥淭he pilots were well aware of which lever to pull. It was 鈥榟uman error鈥� that caused the mistake. But laying the blame on the pilots wasn鈥檛 ever going to solve the problem.鈥�

The individual vs. the system
This 鈥渟ystem vs. individual blame鈥� paradox lies within many technological systems 鈥� including AI. Humans want the shortcuts that LLMs provide, but in any serious context the advantages of those shortcuts is substantially undermined by having to laboriously check the work of the AI. A significant proportion of the time, humans are going to skip the due diligence. Like pedestrians in parks who don鈥檛 want to walk three extra yards, we need to base policy not on an assumption that stubborn patterns of human behavior can be reformed, but on a realistic recognition that because of deeply ingrained tendencies we all have, a certain proportion of people 鈥� perhaps a very high proportion 鈥� are going to take those shortcuts.

Of course in some contexts, such as in a nature preserve, physical shortcuts off established paths can be an anti-social behavior that just needs to be stopped. But policymakers still need to be aware of patterns of human behavior and account for them. In a nature preserve, that would mean being aware of just how much work it might take to stop people from taking a shortcut.

One useful concept from psychology that human factors engineers is what鈥檚 called 鈥溾��: the idea that people are guided by two different systems, one that is rational, logical, and deliberative, and another that is intuitive. In an aircraft, the rational mind would remember there are two different switches; the intuitive mind goes by the rough shape and placement of the desired switch. And as the transportation planning historian Peter Norton , 鈥淚n safe systems, intuitive behavior is not deadly.鈥� In many respects, the behavior of LLMs seem smart enough that our intuitive brains trust them, even if our rational brains know that we shouldn鈥檛.

This dynamic is of course not at all limited to policing; it extends anywhere AI is being experimented with or pushed on employees, including such fields as healthcare, financial services, and law. The implications can vary for civil liberties and the public good, but the same 鈥渟ystem vs. individual鈥� paradox lies within many of them. In the legal profession, where people are very much paid to submit briefs that are rational and deliberate rather than intuitive, many were shocked by not only by the of lawyers being caught submitting briefs with hallucinated AI citations, but also with the of that .

Speeding up the line
That paradox dovetails with a parallel way in which AI is a trap for workers: as an old-fashioned labor . Historically that has meant factory workers forced to deal with an assembly line to increase owner profits, while at the same time being ordered to work accurately and safely. Or, more recently, delivery drivers squeezed between pressure to make more deliveries while at the same time being exhorted to drive safely in order to deflect criticism and liability for crashes from the employers.

Right now there is an enormous weight of expectations being placed on AI to deliver fantastic gains in efficiency and profits across a wide variety of fields, and enormous sums of money being poured into the chase for those gains. 鈥淔ew dispute that the legal industry is under great pressure to use AI,鈥� one reporter . For workers, that may mean growing expectations from management or clients for a certain pace of work that doesn鈥檛 take account of the need for due diligence. That can pressure all workers not to spend the time doing such checking 鈥� until disaster happens, at which point, like the B-17 pilots, the blame is placed on them.

Workers need to recognize this trap 鈥� and so do policymakers
If you're a worker who is being directed to use LLMs as part of your job, you may be getting set up for failure 鈥� but in contexts where your job is important for society, you're going to have to recognize that fact and raise the issue in your workplace or otherwise do your best to prevent any harm. Desire paths are often cited as an example of how it's better to go with the (literal) flow of human behavior and cognition than against it 鈥� but again, sometimes, such as in a nature preserve, people simply must be blocked from going with their flow.

For police officers, we cannot stress this enough: If your department has you using this technology, you really, really need to check its work. Even though 鈥� or especially because 鈥� that may not come naturally. You need to do that not only to keep yourself out of potential trouble on the witness stand or elsewhere, but also to fulfill your responsibility not to treat suspects unfairly. As a police officer, you have enough power that your failures can lead to innocent people experiencing terrible things including being locked up for life.

But as an officer you should also recognize that you鈥檙e being put in a difficult situation by the corporate vendors who sell this technology and the command staff who deploy it 鈥� set up with unreasonable expectations around human behavior or the reliability of AI. Unlike the B-17鈥檚 designers, whose design flaw was unintentional, those who implement AI today should by now be fully aware of the AI trapdoor and how it threatens to ensnare workers.

If you鈥檙e a police department, you shouldn鈥檛 be putting your officers in this position or endangering the public by doing so.

At the end of the day, the paradox here is that individual users of AI need to be vigilant and in some situations can rightly be blamed for failing to do so. But at the same time, much of the fault for problems lies with those who create techno-social systems that breed human failure. We can debate the moral complexities of individual responsibility, but policymakers need to be realistic that lapses are inevitable and take steps to limit certain uses of AI. To take just one example, that means not allowing AI-assisted police reports. For employees, it鈥檚 a trap. For the public good, it鈥檚 a threat.