In 2002 Donald Rumsfeld made this statement about weapons of mass destruction in Iraq
Reports that say that something hasn’t happened are always interesting to me, because as we know, there are known knowns; there are things we know we know. We also know there are known unknowns; that is to say we know there are some things we do not know. But there are also unknown unknowns—the ones we don’t know we don’t know. And if one looks throughout the history of our country and other free countries, it is the latter category that tends to be the difficult ones.
He got a lot of flack for this. Most people could not understand what he was talking about, and they made fun of him for it. However, over time, many people started to realize that what he was saying actually was quite insightful. I was going to create a table to illustrate this, but then I saw that the wikipedia page already had a nice one, so I am just reproducing it here:
| Aware | Not aware | |
|---|---|---|
| Understand | Known knowns: Things we are aware of and understand | Unknown knowns: Things we are not aware of but do understand or know implicitly |
| Don’t understand | Known unknowns: Things we are aware of but don’t understand | Unknown unknowns: Things we are neither aware of nor understand |
Sometimes people like to think that we can prepare for all possible scenarios when planning a project, but I think that is unreasonable. It does not take the unknown unknowns into account. There are some types of work which have many unknown unknowns, while other types have fewer. For example, scheduling delivery of packages, for example like UPS or DHL, is quite predictable. There may be quite a few known unknowns, e.g. we know that there are sometimes traffic jams or engine problems which can cause delays, but I would say that there are relatively few unknown unknowns. Research and engineering tends to have many more unknown unknowns, especially when it comes to scaling. There are many physical forces in the world which have essentially no effect at one scale, yet a big effect at a different scale, and there is no way to find out what these are without actually trying to scale. The only thing we know in these cases is that scaling is hard.
This week at work I was reading a project plan from a colleague for rebuilding a new search index, which included the following
If everything is fine (doubt it) move on with the rest …
I think the comment of “doubt it” is very astute. I see it as being aware of the unknown unknowns. I see some engineers who spend a lot of time trying to build and test software which takes every edge case into account, which can be very time consuming. Then there are those who only test the most common use cases and just assume that everything will be okay. I prefer a middle ground approach – yes, try to build something robust from the start, but realize that for many new projects, especially ones which involve scaling something to millions of documents, requests, etc. it is impossible to know what will break until you have actually tried to build the whole thing. Expecting that it will work on the first try is overly optimistic. Spending lots of time trying to account for all edge cases is a waste of time. Instead, come up with a good plan, try it out and monitor, and be ready to stop processes and revert code if necessary. Then you can spend some time analyzing logs to see what went wrong, and try again, until you get a solution that actually works.