The mil-std-882e standard specifies levels of software control, i.e. how dangerous the software can be based on what it is responsible for. Although the standard is a little more complicated, we can simplify to essentially four levels:

1. The most alarming case is when the software has direct control of something that can be immediately dangerous if the software does the wrong thing.
2. Still dangerous, but slightly less so is either (a) when the software has direct control, but there is a delay between when it does the wrong thing and when it becomes dangerous; or (b) when the software is not directly in control, but a human must immediately react to software signals and perform an action to prevent danger.1¹ E.g. the software commands a reactor shutdown when there are only seconds remaining until the reactor blows up.
3. Yet less dangerous is when the software is not in direct control, and there is time to verify its suggestion against independent methods to make sure the action recommended by the software is indeed appropriate.
4. The least dangerous is when software only has an auxiliary use and is not involved in controlling something serious.

I thought this was a neat way to look at things, and particularly salient now that llms and computer vision have blown open new opportunities for injecting software into processes in which software were previously subservient to humans.