Software in the Age of the Internet
Accepting a new set of expectations laid upon your team today
In a previous post, we observed the stark differences between the team I was part of when I began my career in software and the teams I find myself on today - both eventually building enterprise-grade B2B software. The two primary differences I want to focus on today are -
The diversity of specialized roles on the team went from 7 (QA, DB/Search expert, DevOps, Infrastructure, Front-end, Back-end, Systems) to 3 (QA, Front-end, Back-end).
From being heralded as revolutionary for releasing once a quarter, it is now expected that we deploy on demand.
When you distil it down, the common theme we find is the Internet and its economies of scale. Time for a closer look.
Economies of scale
The mind-boggling scale of the Internet today is second only to the mind-blowing scale of the Observable Universe. Some numbers might help in boggling your mind -
An estimated 5.18B users can access the Internet today.
Over 15B devices are estimated to be connected over the Internet.
Over 69M messages are sent over messaging channels in a given minute
500 hours of video content is uploaded to YouTube alone in a given minute
Over 1 trillion MB of data is added every day to the Internet.
All these numbers have only been growing since the dawn of the Internet.
This exploding demand fostered a vicious technological cycle, that I have tried to represent using the barely scientific flywheel below.
The most important of these technological revolutions has probably been Virtualization.
Now, virtualization wasn’t new to an engineer back in the 2000s. We had already begun running Linux on Virtual Machines via VMWare’s ESX family of products. What changed was that everything began to be connected to the Internet, and everything started to become virtualized, building abstractions over everything I learned in school in my Operating Systems, Computer Architecture and Networking courses. This led to the birth of the so-called “Public Cloud” where servers, networks, storage were all assembled at huge scale and offered via subscriptions to anyone with a credit card connected to the Internet.
The picture below tries to distil the journey of these abstractions over the decades, using Amazon Web Services (AWS) as the anchor, and tries to predict where it is headed from the software engineer’s perspective. The likes of Vercel, Netlify, Render, have already started to define the APaaS (Application Platform as a Service) category, bringing this future closer to reality.
The main takeaway is that as a developer you find yourself working at a higher and higher level of abstraction as time progresses, with the abstraction hiding away all the hard problems that have already been solved, at scale, for you. Gone are the days where the Operating System was our abstraction and one of the smartest System Engineers on my first team built a blocking queue service and associated framework from scratch. There are few reasons for someone to build, host and run a Queue Service today, let alone to host, manage and administer a Database.
The Internet as the Great Leveller
While a formal education in Computer Science never hurts, the abstractions we build upon today have made building web applications within reach of anyone who can write some code (or if you choose the right problem, without writing code). Simultaneously, these skills have become highly accessible as educational content is easily available in various forms, with communities growing like weeds around this content.
As the internet continues to become more accessible, developers start to show up from all corners of the planet. These same developers not only start to build new applications to drive their local economy, but also develop content in local languages to make everything even more accessible to more people. You see where this is going - another flywheel!
In short, the need for highly specialized skills is rarer when it comes to building a web application, and the skills you do need are highly accessible.
Internet (and the browser) as the most powerful distribution channel, ever
Any successful business trying to sell more of its product will attest to the definitive importance of efficient distribution channels being key to its success. A distribution channel is what helps you take your end product to your customers, hopefully at scale.
With those 5B users connected to the Internet accounting for over 60% of the global population, there has never been a bigger distribution channel. You don’t need to lay roads or run cables to reach your end users.
From a software engineering perspective, this distribution channel enables us to make new product versions instantly available to everyone, anywhere, as long as they are connected to the internet. There is no need to ship a build anymore - something we used to do using DVDs (remember them?) and UPS. There is additionally no need to run through a set of cryptic instructions to install the software to access the new version. Builds are deployed to your infrastructure in the Public Cloud, and a browser page-load starts accessing the new version instantly.
We are now used to seeing updates, bug fixes, cool new features, every time we logon to our favorite social app or try to place an order online via e-commerce. These expectations of the average user have spilled over into all walks of software - we are a restless lot today. New versions and changes cannot take days or weeks, let alone months.
A New Generation
Software and the engineers building them are held to a very different set of standards today. Remember, we haven’t even looked at the growing influence of Generative AI in all of this. You aren’t expected to be a rare specialist in one particular domain, but our tolerance for outages and glitches in software is a lot lower than it ever was, meaning the bar is higher in another dimension - quality and performance.
This does not mean there is no place for specialists. As we will see in a following post, most organizations end up requiring specialized SREs, Internal Platform teams, and DevSecOps engineers once they reach a particular scale. These however end up belonging to centralized teams and aren’t luxuriously available for every team that is shipping software. Your team is most likely to have only a couple, or at best, 3 different roles in software.
If you begin to accept this reality as a young engineer yourself, you will want to start seeing yourself as a more fully rounded engineer expected to play multiple roles that were traditionally mapped to different individuals in the past. If you would like to understand not just why this is expected but also how you can prepare for this, stay tuned!