No Feedback ∴ No Good
The title should be prounounced no feedback, therefore no good – but it is meant in the sense that there’s no way of knowing if you’re any good without feedback.
I once interviewed wtih a person who had worked as a security advisory consultant. This meant their job used to be advising large organisations on the status of the softer parts of their security systems: people, processes, information barriers, etc.
This was something I’d always been curious about, so I had to ask:
Beyond the basics, how did you know if you were doing a good job?
Their response was candid:
I didn’t. I’m not sure how I would.
The problem is, of course, that security breaches of the kind they were trying to protect against are rare, and if they happen, they only tell you what failed, not all the things that worked. This advisory role has very few feedback paths and they are all low-quality. The advisor simply cannot tell whether what they are doing is working or not, and much less if it’s an effective use of money.
These professions are not altogether uncommon: large parts of of social science, political analysis, career counseling, macroeconomics, education policy, and advisory consultancy fall prey to this problem. When the feedback paths are missing, success is measured through peer opinion, i.e. the practitioners that have a good reputation among their peers are considered skilled, regardless of what their real world impact is.
I don’t know exactly what the point of this is – and I am sure Nassim Nicholas Taleb has rambled about it far more than I could ever dream of – but I recommend asking that question more often of people you admire. How do you know if you’re doing a good job? can yield fascinating answers from skilled people.
Sometimes the answer sounds sensible but are essentially a veiled “when other people tell me I did a good job.” These are not the types of professions I would be good at, but I know a lot of people who are very successful at them.
Just after writing this article, I was coincidentally reminded of where I got this question from in the first place. In Working Minds1 Working Minds: A Practitioner’s Guide to Cognitive Task Analysis; Crandall, Klein, Hoffman; MIT Press; 2006., the authors give some advice on how to find an expert to subject to cognitive task analysis. But they caution that in some fields, it might be hard to find an actual expert. They list three signs of such a field:
- High turnover (few people work long enough to build expertise);
- Recent technological revolution (expertise requirements changed); and
- Lack of clear feedback.
The third point is what this article is about, and how do you know if you’re doing a good job is one of the first questions one should ask of experts when learning to navigate their field.2 Another good question is Was that outcome your intention? – which, to my delight – my son is asking a lot.