Curiosity Skilled the Cat
The underrated trait missing in most competency matrices
Over the course of two decades, I have had the joy of working with hundreds of engineers from all walks of life. The best of the lot are blessed with coveted hard skills like programming, cloud infrastructure, devops and security, side-by-side with soft skills like accountability, drive, and communication.
Looking back at some of the most productive and impactful engineers I had the joy of working with, I find that besides being highly independent, accountable, and self-motivated, what binds them all is the common thread of curiosity.
Curiosity is Human
Curiosity is core to what makes us human. We are able to understand and control our environment more than any other being because we have been curious.
Curiosity helped us discover fire and control it. It helped us move from foraging and hunting to agriculture. It invoked us to create new tools that made us productive. Curiosity about what lies beyond the horizon enabled us to settle in remote corners of this planet. Today, curiosity about what lies above pushes us toward distant planets and invites us to peer back in time through massive space-bound telescopes.
When it comes to building great software, the same trait is indispensable. Let’s take a closer look at how curiosity helps make those engineers I mentioned earlier, great at what they do.
Sustainable Solutions
When something really needs fixing, the curious engineer will probe and question until they get to the bottom of what caused the problem in the first place. As opposed to simply sticking duct tape to hold things together, they will uncover the root cause and come up with solutions that drive at the fundamentals of what holds the system together.
Let’s look at an example inspired by something I personally witnessed.
Failed by rate limits
One fine November evening like today, our observability system triggers alerts for high error rates - requests are not being fulfilled and many apps running on our platform have stopped to function reliably. Traffic on one set of services is abnormally high. We instinctively fear a DDOS attack, although the scale is not scary enough to bring down the platform. We soon find out that one of the cloud telephony apps running on our platform is the primary source of this traffic spike. We immediately disable the app to win temporary respite, and reach out to the app developers.
Based on an analysis of endpoints being hit and a quick review of the app’s source code, we realize that the app was being rate limited by design but wasn’t respecting the HTTP 429 response, and kept retrying indefinitely. Many engineers would be happy with this conclusion, parking the problem on the app developer, and moving on with life.
A couple of engineers however were not satisfied with this. They wanted to understand -
Why did the app need to make so many requests in the first place?
Why did our platform not manage to scale out in response instead of starting to error out?
To cut a long story short, here is what they eventually learned.
The app relied on middleware that had stopped responding, leading to increased retries to the middleware that in turn triggered more requests to our platform.
One of our Rails services had a default IP-based rate limit that kicked in, leading to 429 errors our own services started encountering, cascading further out to other apps.
This enabled us to fix, among other things,
Additional rigor in our app review process for apps that rely on middleware.
Automation to allow us to quickly disable “rogue apps”.
Recalibration of limits and how we handled them internally within our platform.
We clearly came a long way from just fixing the app in question.
Resilient Systems
Curiosity is never satisfied by what is visible on the surface. It drives forth with the itch to scratch and peek under the abstraction. In an age where a typical engineer builds on layers and layers of abstraction, often blind to the quirks hidden behind the interface of a framework or service, our curious engineer has already revealed a few secrets from this realm through root-cause analysis excursions of the past. These secrets help them visualize the layers as complex terrains rather than a neat, flat surface.
As a result, they can envision where landmines exist under the abstraction, and design and develop a system that navigates this terrain safely in terms of predictability and reliability. If you care about these qualities and would rather not learn through surprises after the system is deployed, unleash your curious engineers on these problems.
Innovating with Purpose
Unless you are working toward inventing technology that does not exist, it is very likely many of the problems you are trying to solve have been solved before. Understanding the difference between table stakes and innovation is critical for a business in order to spend its time and energy wisely.
Our curious engineer is always wondering if a problem has already been solved elsewhere. Very likely, their curiosity has already had them poring through tech blogs studying how other teams have solved certain problems. When a new problem is posed to them, one part of their consciousness is on the lookout for proven solutions. Once a proven solution is mapped to the business problem at hand, they compute any delta and work on a solution for this delta that the business can differentiate upon.
Critically, their curious selves have also exposed them to emerging trends and new-fangled ways of solving old problems, which enables them to innovate for differentiation.
Let’s look at another example.
Look ma! No servers!
When our biggest customers wanted to run automations against change events our products generated, our curious engineers found that -
Competitors solve this through webhooks but expect customers to run webhook consumers with custom code
This was 2017 and AWS Lambda was threatening to go mainstream to solve this very problem
Delivering change events reliably was table-stakes and a solved problem. Running custom code on behalf of a customer on Lambda functions we manage was innovation. This enabled us to channel our energy toward this differentiation to stand out as one of the early serverless platforms in the industry.
Harness Curiosity
Like with the Schroedinger’s cat thought experiment, we always have the urge to unravel the quantum superposition just to reveal the truth behind what binds the universe. Leaders can harness curiosity by parking time in the moment to help engineers get to the root of a problem. If this tendency is found to be lacking, champion the minority who try to be curious despite the pressures and demands of business, and task them with analysing your recurring headaches and sharing what they have learned.
Organizations and teams that retain curiosity learn faster than others. Isn’t the purpose of a business to learn quickly what is needed to become successful and then to stay there?