1.3 Choosing the correct playbook
It’s time for that talk. Why do some people rave about SaaS and ridicule custom software development, nitpicking the sources they cite? Why do others rave about custom software development and refuse to touch anything premade? Why do some people support the agile methodology while others think it’s just a fad? What on earth is going on? How can people disagree so much?
The first piece of the puzzle is understanding these things as playbooks — overarching strategies and approaches to delivering solutions. Each playbook comes with its own philosophy, methodologies, and trade-offs. You can’t discuss any of them without having a deep enough understanding of what they stand for, so let’s try to define the most common ones.
Building solutions from scratch, also known as custom software development, offers a unique level of control over every aspect of the design, functionality, and user experience. This control empowers you to align your solution perfectly with your unique business processes. However, it’s important to note that this level of control requires a high level of capability and skills. While you enjoy the freedom from vendor lock-in, you also bear the responsibility of maintaining most of the technology. The time to market is relatively long, and significant capital expenditures are required, but the sense of control is unparalleled.
Considering the risk of failure (statistically, 2/3 of software development projects fail) and that most companies have finite high-end capabilities, building from scratch is most typically selected only for essential mission-critical systems. Heavy use of open-source frameworks, libraries, and such, as well as low-code or no-code platforms, reduces related risks, making building from scratch more palatable for business managers, but they come with vendor lock-in trade-offs.
The second standard playbook is commercial off-the-shelf software. Commercial off-the-shelf might be something you buy, install, and configure or a SaaS service. Since you’re purchasing premade, the time to market is simply superior. The capital expenditures are lower than in building from scratch, but this often comes with the trade-off that the operational spending will be higher. It is also likely that the product is well-tested and mature because there have already been major users, and the product will receive automatic updates. The significant downside of commercial off-the-shelf is that you can’t customize it freely, and there will be stronger vendor lock-in.
Since the lack of customization options has limited the use of commercial off-the-shelf software, some vendors have opted for their strategy to sell development platforms instead. Think of something like Salesforce, which has ready functionality packages available but was built from the ground to be extendable. A whole ecosystem of 3rd parties offers modules and customization services on top of Salesforce. Again, the major downside of this approach is capabilities - managing a network of vendors and components and governing the development can prove demanding.
Never attempt to customize a commercial off-the-shelf product that wasn’t purpose-built from the ground to the top to be an extensible platform! Nothing else will create a more substantial vendor and technology lock-in. Nothing else will blow up the development budget as developing on a platform that isn’t proven efficient. Vendor-specific black boxes may create skills and requirements that are nearly impossible to fulfill, and your negotiation position will be inferior. You might also have a fragile and costly time maintaining the Frankenstein System.
Agile development is an iterative approach to development that focuses on delivering small, functional pieces over time. It’s about continuous feedback loops, pivoting mid-project when requirements change, and delivering results fast. The ability to evaluate results earlier reduces the risk of colossal failure. While it’s true that many companies struggle with agile, it’s important to remember that when done right, agile can lead to significant success. It’s not just a methodology, it’s a mindset that, when embraced with discipline and skilled people running the show, can lead to transformative results.
Waterfall development is a linear approach where the development team defines requirements, designs, builds, and tests sequentially. Waterfall development can, in principle, succeed if the requirements are well known. It is possible often with technical back-end software. When humans use the software, they hate being forced to adhere to set processes and frequently change their minds after seeing the first version of the software. The late discovery of defects leads to costly fixes; overall, waterfall models are inflexible, and nothing gets delivered before everything is ready. Overall, waterfall development works well for software that serves machines.
There are also low-code and no-code platforms. They are a paradigm of visual development using drag-and-drop interfaces. Their main benefits are blazing development speed, lower reliance on high-end development skills, and the ability to develop software closer to the business needs. Sometimes, enlightened business users develop using low-code or no-code platforms, and seasoned software developers mostly just assist with more advanced features. The downsides of such platforms are vendor lock-in, which can limit your flexibility in the future, and shadow IT-related issues, which can arise when business units or individuals use these platforms without the knowledge or approval of the IT department, leading to potential security and compliance issues.
In the real world, you will seldom see these playbooks applied in pure form. Most companies tend to go for hybrid approaches. A standard, basic model is to categorize all systems into four segments: 1) Managed research & development, 2) Strategic initiatives, 3) Commodity systems (such as email or payroll), and 4) Systems that are being sunset. The first category can be anything; strategic initiatives are where you put all your best personnel resources in for custom software development, the commodity systems are SaaS, and the sun-setting is about finding replacement solutions using the first three.
This previous kind of thinking is fundamental. If the question were whether you could build everything custom, the answer would be, of course, you can. But should you pour all your talent and capabilities into customizing everything to the maximum? No. Custom software development is hard. It should be reserved for the most challenging and important systems. If you try to customize everything, your resources will be misallocated, and your most important initiatives will be more likely to fail.
What will also certainly fail is trying to change the project’s playbook on the fly. It’s not a technical challenge. It’s about capabilities and skills, sometimes contracts, and sometimes intellectual property rights. If you switch the playbook, the reasoning behind choosing the playbook changes. That should never be possible without restarting entirely from scratch. Hard reboots are hard for a reason.
What about artificial intelligence, then? Think of the generative artificial intelligence as a force multiplier here. It empowers developers to accomplish more. A high-impact team using generative AI will have an even higher impact, whereas a mediocre team will still be mediocre. What will change is how much your high-impact teams can bite on. They could take on one more extra project or make one system slightly more extensive using generative AI. Other than that, the generative AI shouldn’t change your calculus.
If you have ever read about a company exiting the cloud and saving millions, you should know this. Companies use the cloud for agility and the availability of talent. Going on-premises and optimizing your platform takes serious and rare skills. The point here is skills. Skills. Skills. A skilled engineer can do so much more and lower overall costs anywhere. Also, homogeneous environments are more straightforward to optimize. Larger companies tend to have more heterogeneous environments, requiring agility from the platform. Cloud, when used correctly, is yet another force multiplier.
Your job as an architect is not to pick and force a single playbook down everyone’s throats. It is to discuss and debate the playbooks with the others in your environment and help your company identify the right combination of playbooks for the correct reasons and projects. Ensure everyone knows these models’ pros and cons and where they lead. You will likely never find the perfect combination, but you can help your company navigate the trade-offs and understand why they are building the stuff and how they are doing it.