July 24, 20264 min read

Keep Humans in Control: The Ironies of Automating a Sales Process

Bainbridge's forty-year-old result on why automating the easy parts makes the human job harder — and what that implies for automated follow-up.

Automation
Human Factors
Reliability

The classic result in human-factors engineering is that automation does not remove the human from the loop. It removes them from the easy parts of the loop, leaves them the hardest parts, and simultaneously destroys the practice that made them good at those parts.

Of the four principles on Ariadne's site, "keep humans in control" is the one that sounds like a values statement. It is not. It is a design constraint derived from a literature that has been consistent for forty years.

The ironies

Lisanne Bainbridge's 1983 paper in Automatica is four pages long and has aged remarkably well Bainbridge 1983. Its central observation:

The designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate.

The consequences compound. If automation handles the routine cases, the human is left with the exceptions — which are, by construction, the hardest cases. And because the routine cases no longer reach them, they lose the everyday practice that built the judgement the exceptions require.

Bainbridge also states the hard limit that anyone designing a monitoring role must respect:

It is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour.

Barry Strauch, a former NTSB investigator, revisited Bainbridge thirty-five years on and concluded her paper showed "prescience in predicting automation-related concerns that have led to incidents and accidents, and relevance to issues that are manifested to this day" Strauch 2018. The ironies did not get solved. They got deployed more widely.

Figure 1What automating the routine cases does to the human's remaining work. Volume falls; average difficulty rises; practice on easy cases — which is how judgement is maintained — disappears.Diagram by the author.

Four ways it goes wrong, with names

Parasuraman and Riley gave the field its vocabulary Parasuraman & Riley 1997:

Use

Whether operators voluntarily engage or disengage the automation — driven by trust, workload and perceived risk, with large individual differences that make behaviour hard to predict from the design alone.

Misuse

Over-reliance. Failures of monitoring and decision bias: the operator accepts the automation's output without the check they would have applied to their own reasoning.

Disuse

Rejection. Most often driven by false alarms. Their base-rate analysis is the sharp part: when the target condition is rare, even a highly sensitive alarm yields a low posterior probability that any given alarm is real — so operators learn, correctly, to ignore it.

Abuse

Automating a function because it is technically possible, without regard for what it does to the human's remaining role. This is a designer failure, and it is the one I am most often asked to commit.

The disuse mechanism deserves the arithmetic, because it is why your alerts get ignored. With base rate $P(T)$ , hit rate $P(A \mid T)$ and false-alarm rate $P(A \mid \neg T)$ , Bayes gives:

P(T \mid A) \;=\; \frac{P(A \mid T)\,P(T)}{P(A \mid T)\,P(T) + P(A \mid \neg T)\bigl(1 - P(T)\bigr)}

Take a lead-scoring alert with a 95% hit rate and a 10% false-alarm rate, firing on a condition that occurs in 2% of leads:

P(T \mid A) \;=\; \frac{0.95 \times 0.02}{0.95 \times 0.02 + 0.10 \times 0.98} \;=\; \frac{0.019}{0.117} \;\approx\; 0.162

A 95%-accurate alert is right 16% of the time. Your team is not being lazy when they stop responding to it. They are being Bayesian.

~30 min

Limit of effective human attention on an uneventful display

Bainbridge, Automatica 1983

16%

Posterior probability an alert is real: 95% sensitivity, 10% FAR, 2% base rate

Bayes' rule

Failure modes named: use, misuse, disuse, abuse

Parasuraman & Riley, Human Factors 1997

The mitigations mostly do not work

The natural response is training and awareness. Parasuraman and Manzey reviewed the evidence and found complacency and automation bias emerge specifically "under conditions of multiple-task load, when manual tasks compete with the automated task for the operator's attention" — precisely the condition of a small service business — and that the standard countermeasures do not reliably fix it Parasuraman & Manzey 2010.

So the design has to carry the load, not the training.

What this means for automated follow-up

Concretely, the rules I build to:

Automate timing, never judgement. The system decides when a follow-up happens and who it goes to. What is said to a customer in a non-routine situation stays human. This keeps the human in the easy cases too, which is what preserves their judgement for the hard ones.

Make exceptions arrive, not accumulate. An exception queue that a human is expected to check is a monitoring task, and monitoring tasks fail at thirty minutes. Exceptions must be pushed to a named person with a deadline.

Tune alerts against the base rate, not against sensitivity. Compute the posterior before shipping the alert. An alert that is right 16% of the time will be disabled within a month, and it will take the useful alerts with it.

Preserve reversibility. Every automated action a customer can see should be visible to your team and undoable. This is what "keep humans in control" means operationally: not that a human approves everything, but that a human can always find out what happened and change it.

Leave the keys. Which is the fourth principle, the subject of a later post, and the reason all of the above has to be readable code you own rather than a configuration inside somebody else's platform.

References

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779. https://doi.org/10.1016/0005-1098(83)90046-8
Strauch, B. (2018). Ironies of automation: Still unresolved after all these years. IEEE Transactions on Human-Machine Systems, 48(5), 419–433. https://doi.org/10.1109/THMS.2017.2732506
Parasuraman, R., & Riley, V. (1997). Humans and automation: Use, misuse, disuse, abuse. Human Factors, 39(2), 230–253. https://doi.org/10.1518/001872097778543886
Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381–410. https://doi.org/10.1177/0018720810376055

Next: stage 09, and the arithmetic of a loop that pays for itself.

Sourena Khanzadeh

Founder & Growth Engineer, Ariadne Growth Systems

Toronto, Canada

Ariadne Growth Systems Growth System Audit support@ariadne.fyi