Responsible AI
Responsible AI — who pays when it is wrong
When an AI gets it wrong, where does the consequence land?
Responsible AI judges a system by what happens when it is wrong rather than by how well it scores. An error rate in an evaluation and the loss borne by the person on the receiving end are not the same size, and the question is who absorbs the difference.
The gap between evaluation and consequence
95% accuracy reads as a good number. But if the remaining 5% is not spread evenly and lands on particular people, then for those people the system is not 95% accurate. Aggregate metrics hide this.
What to ask at design time
When this feature is wrong, can the user tell? If they can tell, can they undo it? If they cannot undo it, was a person meant to confirm it first? Automation with no answer to those three is pushing responsibility onto the user regardless of how well it performs.