This article is part two of a five-part series that explains the Leadership Operating System I developed earlier this year as a post-doc project.
The research aimed to codify leadership into learnable practices. From it, I created an assessment tool that gives you a forensic reading of how you lead. Give it a go.
On 12 July 2011, Netflix ended the plan that gave customers DVDs by post and unlimited streaming for $9.99 a month. Anyone who wanted both would now pay $15.98, a rise of 60 per cent.
On 18 September, Reed Hastings emailed his customers. It opened with “I messed up. I owe you an explanation.” He kept the price rise, and announced that the DVD service would become a separate business called Qwikster, with its own website.
On 10 October, Netflix dropped Qwikster (the name lasted three weeks). “It is clear that for many of our members two websites would make things more difficult,” Hastings wrote.
Two weeks later, Netflix reported that it had lost 800,000 US subscribers in the quarter. By then its shares had lost more than half their value since July.
And the bet underneath all of it, that streaming would replace the DVD in the post, turned out to be right.
So was that bad judgement, or good judgement badly carried out? And how could anyone have told before the results came in?
Understanding judgement
When Don Moore and Paul Healy reviewed the evidence on overconfidence, one finding stood out. People asked for a range they were 90 per cent sure contained the right answer and were only right 73.1 per cent of the time. Roughly one answer in four that they were sure of fell outside the range.
Barbara Mellers and her colleagues ran a related experiment. They tracked between 2,200 and 3,900 forecasters a year making predictions about world events, and found a group who scored 96 per cent on how well their forecasts separated events that happened from events that did not, against 75 per cent for everyone else. The paper’s title is about identifying those people and teaching the same habits in others.
Both studies could measure judgement because people said, in advance, what they expected and how sure they were. Judgement can only be checked if you state what you expect, and how sure you are, before the result arrives.
In my Leadership Operating System, Judgement is the second of five domains. It covers how well you reason and choose when the information is incomplete, the priorities compete, and the consequences take time to show. It has three disciplines:
Diagnosis and sensemaking, which is working out what the problem is;
Strategic and systemic reasoning, which is working out what each option will cost; and
Decision discipline, which is deciding when, and how sure to be.
My sense is that capable people are judged on results for so long that they start to treat results as the measure of their judgement. Results measure judgement and luck together. As you move up, fewer decisions have an answer you can look up, and more of the job is choosing between defensible options with half the facts.
If you keep one thing from this piece, keep this:
Nobody can check your judgement until you write it down, including you.
By the end of this piece, you will have a five-line note to write before any decision that matters, and a working list of the sixteen behaviours behind it.
The five-line decision note
Five lines, written before you decide, for any decision that would be costly to get wrong or hard to reverse.
The problem underneath is…
What we do not know, and are assuming…
The options we considered…
What we are giving up, and why…
How sure I am, and what would reopen it…
The first two lines are diagnosis. They stop you solving the problem as it was presented, and they put your assumptions where someone can see them.
The middle two are reasoning. A note with only one option in line three is a recommendation looking for approval. Line four is the one I see left blank most often, because the cost is the part nobody wants to say out loud.
The last line is discipline. A number makes your confidence something you can be wrong about, which is how you find out whether your nine-in-ten calls come true nine times in ten.
A finished note can be this short:
Problem: the north region is missing its target because two big accounts moved their buying to head office.
Assuming: the accounts will not move back this year.
Options: chase the central buyers from here; restructure the north team; wait two quarters.
Giving up: six months of pressure on the north team’s numbers, which I will explain to them.
Sure: 70%. Reopen if either account starts buying regionally again.
Each line draws on one of the three disciplines, and between them they hold sixteen behaviours.
Diagnosis: judging what the problem is
Problems usually arrive with a solution already attached. Diagnosis is the discipline of separating the two before anyone spends money on the fix.
Identify the underlying problem. Ask why the problem exists before anyone proposes a fix, ask again of the answer, and write the cause down in one sentence.
Ask for the information that changes the decision. Say what you would decide now, then name the one fact that would change it, and ask for that alone.
State what is not known and what is assumed. Give every decision paper, however short, two headings: what we are assuming, and what we do not know.
Spot which problems need people to change. Ask who has to behave differently for this to be fixed, and whether they want to. If it is a named group who would rather not, a process fix will fail.
Notice early signals of change. Keep a running list of small things that do not fit, such as a customer who went quiet, and review it every fortnight.
The second behaviour has a useful test in it. If you cannot name the fact that would change your mind, you already have enough information and are delaying. Diagnosis will get its own article in the discipline series.
Strategic reasoning: judging what it will cost
When John Graham, Campbell Harvey and Shiva Rajgopal surveyed more than 400 executives, 78 per cent admitted they would give up long-term value to smooth earnings. The cost of a decision is often paid in a later year, or by a different unit. This discipline has six behaviours, and most of them are about finding that cost before you commit.
Generate more than one option. Never bring one option to a decision. If only one exists, the second is doing nothing, with its consequences stated honestly.
Name the trade-offs. Say what you are giving up in the same sentence as the decision.
Anticipate how a decision plays out over time. Write one paragraph on the situation a year on, then one on how the people affected will have responded by then.
Weigh this period’s results against future capability. When you protect this period’s number, write down what it spends: the training cancelled, the hire deferred.
Treat cost, risk, capability and time as one whole. Put all four on one page before recommending. If one is blank, you do not yet have a decision.
See how your unit affects the wider organisation. For each of your main goals, ask which other unit finds it harder to succeed because you are succeeding, and go and check with them.
For the second, the sentence carries its own cost:
“We are putting the team on the launch, which means the upgrade slips by a quarter.”
Strategic and systemic reasoning will get its own article.
Decision discipline: judging when, and how sure
Good reasoning still has to end in a decision, made at the right time and held with the right amount of confidence. This discipline is about the timing and the confidence, and about knowing which decisions deserve slow thought.
Decide in proportion to the stakes. Ask how costly the decision is to get wrong and how hard it is to reverse. If it is cheap and reversible, decide in minutes.
Decide in good time. Set the decision date when the discussion opens, and on the day, decide with what you have.
Express confidence that matches the evidence. Put a number on it, with the reason: “Seven times in ten, because of the renewal rate.”
Commit once decided. Reopen a decision only for new information that would have changed it, had you known it at the time.
Change course when the evidence warrants, and say so. Set the review date and the reversal trigger at the moment you decide, so that reversing is the plan working.
The first behaviour is what keeps the note from becoming a burden. Most decisions are cheap and reversible, and deserve minutes. Decision discipline will get its own article.
Four ways capable people get this wrong
The single option. The first idea that sounds right becomes the only idea, and the decision meeting turns into a request for approval. We do this most when we are busy and the idea is ours.
The confident guess. Certainty in the voice, with nothing behind it. The team stops checking, because you sound sure.
The reopened decision. A decision revisited because someone senior frowned or a meeting went badly, with no new information. After two or three of these, the team waits a week before acting on anything you decide.
The comfortable wait. One more report, one more month, because deciding feels riskier than waiting. Waiting has costs too, and nobody writes them down: the option that closes, and the people who cannot act until you do.
Netflix, played back
Run the note over the summer of 2011, using only what was said in public.
The problem underneath was real. Netflix said the price rise reflected the cost of acquiring and delivering films, and Hastings argued that DVDs and streaming were different businesses with different costs. As a diagnosis of where the company was heading, it held up.
What was Netflix assuming? That customers would accept paying 60 per cent more, and then accept two websites. I can find neither assumption stated in what the company announced. The two assumptions nobody stated were the two that customers rejected.
From outside, we cannot see which options were considered, or what Netflix expected to give up. We can see the last line working. On 10 October, Hastings reversed the Qwikster split publicly within three weeks of announcing it.
You might be thinking, “I don’t have time to write a note for every decision.”
You do not need to. The note is for decisions that are costly to get wrong or hard to reverse, and there are fewer of those than it feels. For the rest, the first behaviour of decision discipline applies: decide in minutes, and say it was a quick call.
Nor will the note make you right. Some decisions will go badly whatever you write. What it gives you is a record of what you expected, so that when the result comes in, you can tell bad luck from a bad assumption.
Write the note for a decision you have open
The evidence that people are overconfident, and that calibration can be improved, is reasonable: Moore and Healy’s figures are from controlled studies, and Mellers and her colleagues tracked thousands of forecasters over several years. The five-line note is my arrangement of practices that are widely used and consistent with that research.
Pick one decision you have open now that would be costly to get wrong, and write the five lines.
Then put a date in your diary to read the note again once the result is known.
Take this away
The tool: The five-line decision note, written before any decision that is costly or hard to reverse.
Ask: “How sure am I, in a number, and what would make me reopen this?”
Do this week: Write the note for one open decision, and book the date to read it back.
And then go to Leaders of the People and assess your own Judgement.