The starting point for knowledge is mystery. Everything we now know started as a mystery in which we couldn’t even discern the variables that mattered, and therefore had no capacity to understand cause and effect. Think of how the world was baffled, for example, in the very early days of the AIDS crisis. We didn’t know how to think about this new and horrible condition.
But in due course, as is the case in many domains of knowledge, AIDS became less of a mystery. With hard work and study we advanced to a heuristic — that is, we started to understand what variables mattered and developed a sense of the cause and effect. We came to the conclusion that it is an acquired autoimmune disease transmitted primarily through sexual contact. This enabled researchers to focus on the relevant variables and better understand cause-and-effect relationships — for example, the relationship between unprotected sex and transmission…
AIDS researchers and every other scientist since Aristotle have attempted to ferret out cause and effect because they want to explain how the world works. They want to drive knowledge toward an algorithm like E=MC2 with all the subtlety gone.
The question is: How do they do it? How do they eliminate the subtlety between cause and effect in order to drive knowledge toward algorithm? Typically, the approach is to tackle cause and effect (dynamic complexity) by reducing the number of variables considered (detail complexity).
My own clan — the economists — is particularly inclined in this direction. There are a thousand economists working on partial equilibrium problems for every one working on a general equilibrium problem. This is despite the fact that no one would contest that general equilibrium clarity is the most valuable knowledge by far. Why? Because it is really difficult to specify any general equilibrium cause-and-effect relationships.
Instead, most of the guns deployed in modern knowledge advancement are aimed at narrow problems for which the cause-and-effect relationship is specified with the famous “all other things being equal” proviso. Each narrow knowledge domain develops analytical tool-sets that deepen the narrow knowledge domain. Each narrow domain develops ever more algorithmic knowledge, and those developing the knowledge are extremely confident that they are right because they are so specialized within their own domain. The liver expert is completely confident that he or she is correct even if it is the interaction with another condition that threatens your health most.
This approach has created another kind of complexity: inter-domain complexity. Every field is segmented into multiple domains, each with deep algorithmic knowledge, specialized tools, and experts in the domain who think they are absolutely right. And they are indeed right, as long as we ignore the reality of detail complexity.
However, the real world we live in, and have always lived in, is a world of detail complexity. So when we sacrifice dealing with detail complexity to focus on dynamic complexity, the solutions don’t produce the outcomes that we really want. For all their great work, it is unclear that economists have actually helped government officials manage the complex task of managing a national economy any better than they ever have. And despite massive advances in narrow domains of medical knowledge, actual health outcomes have been difficult to improve, especially in errors of high detail complexity.
This is, I believe, what makes it feel that complexity has increased. I absolutely do not believe that the subtlety between cause and effect has increased at all in the world. But the negative manifestations of the largely unaddressed inter-domain complexity make it feel like we have massive un-addressable complexity overwhelming us.
In other words, we are bedeviled by manufactured complexity — complexity that could have been avoided but has instead been amplified by the pursuit of narrow knowledge in a broad world.
It is vital, therefore, to our ability to make progress against large-scale problems that we figure out how to tackle inter-domain complexity.
