You need a big upfront design
You’ve probably heard that creating extensive designs of your system upfront is a waste of time and effort. This idea is borne from the theory that in a world of agile iteration, you can develop your designs as you go and evolve an emergent architecture based on real-world feedback. I’m here to tell you: this thinking is naive and will do considerable harm to your delivery of enterprise systems.
The Popular Opinion:
There are plenty of articles that declare a big upfront design an ineffective and antiquated practise only done by the relics of the waterfall era. Much of this rhetoric is recited by organisations trying to embrace an agile approach to delivery. It generally boils down to the following reasons:
Requirements can’t be fully understood upfront.
By the time you’ve completed the design, it will be out of date.
You can’t know enough upfront to design everything.
Without doing, there is no learning.
The scope of your design can grow out of control and be hard to finish.
Technology engineering is not like civil engineering.
And on the face of it, these don’t sound too unreasonable, do they?
The original pioneers of this idea tend to mean that the design is fully detailed before any build work starts and then set in stone until final delivery. However, this ideology is more often used as a justification to avoid any meaningful upfront design and leave it to be done ‘just-in-time’ while building the solution. This is the interpretation we’ll be addressing here since it is the most common and problematic.
Even so, it isn’t difficult to understand why this is a common way of thinking. There is something very gratifying about getting on and building things, which fits nicely with delivery managers wanting to show clear and rapid tangible progress to the business. So it is hard to resist this confluence of desires and ideology. However, this can easily lead the inexperienced down a very tricky path.
There are, however, good reasons why this thinking has come about that we should not dismiss. Some organisations, and certainly far too many historically, have failed to deliver projects successfully because the design was both overly time consuming and ultimately not fit for purpose. Architects can become detached from the reality of the business (strategic goals, financial constraints, economic factors, user needs, etc.) as well as from the practicalities of the delivery process.
All in all, it is easy to sympathise: The organisation is not delivering because the design process blocks them. So it is not unreasonable to challenge upfront designs and to suggest evolving the design incrementally and collaboratively.
Why this is wrong
Unfortunately, this ‘anti-upfront-design’ approach can be a dangerous one; mainly because the interpretation of this ‘sound-bite strategy’ leads organisations to avoid upfront design & architecture altogether (or at least minimise it to a level of uselessness). Teams tend to avoid things they’re either not so good at or connected with, so they can focus on what they do best (a developer-focused team will naturally want to write code and build things).
Many organisations, therefore, have minimised the architect’s role and empowered developers to “just go deliver things”. Initial results are usually exciting — something tangible is ready sooner, and the business stakeholder feels more engaged and empowered too. All in all, everyone feels like this is a productive and effective approach…
Until we realise that our plans aren’t working out in the long term, the delivery has slowed down, maybe even stopped. We find it hard to meet the project requirements. Making changes is difficult. And it is hard to reason about the system or communicate how it works to key stakeholders (e.g. security) because the design documentation is missing or inadequate, and the team only have the ghosted outlines of past whiteboard sessions around the office, along with the lingering scent of hopes and prayers and a recycling bin full of good intentions.
Worse still: the solution, when implemented by multiple teams collaborating without this upfront cross-cutting design, starts to resemble a slightly demented and sickly chimaera crossbreed of competing ideas and styles, with inconsistencies and incompatibilities that would put the Winchester Mystery House to shame.
The Winchester Mystery House is a sprawling, architectural oddity, with a lot of complexity and inconsistencies: Doors open to the outside from the first and second floors with 20–30ft drops, stairs that lead up to blank walls, buckets full of keys, rooms that are totally walled in (and only recently discovered), and much more — all as if it had been built without any real forethought or design.
Creating the design as you go means:
You will not have enough understanding of the design to be able to plan the delivery sequence of complex, shared, or high dependency capabilities, which will attack your critical path at every opportunity.
You will not see significant pitfalls of a chosen approach (technical or otherwise) that only become apparent when you work through the way the entire design interacts with itself and others.
You will not find opportunities to simplify the build work — e.g. identifying repeating patterns very early will enable a re-usable component or capability to be created. Doing this as you go will mean continual refactoring whilst on the critical path for delivery and without a coherent/broad view of requirements — which will lead to a lot of disruptive breaking changes.
You will not be able to partition the system into cohesive yet minimally coupled deliverables so that teams can work together with minimal dependencies (which create delivery complexity).
You will not be able to bring the team together under a shared understanding of the system and where/how they fit into realising it.
You will not be able to plan your delivery with any kind of meaningful results: dates, finances, risk management are even more unreliable without adequately understanding what it is you will be building.
You will build the first design that solves the first challenge you experience. Rarely is your first attempt at anything the correct answer. Designs that have had several (constructive) review cycles are much more durable than those that haven’t. (Even better if they are practically tested/validated)
Note: When we talk about ‘design’ here, we are not talking about adding a new widget to a web page, adding a new FAQ section, or changing your colour scheme; we are talking about full solution or system design.
Unfortunately, there can be a tendency to treat systems as just a set of independent features. Whilst this is part of the equation, it is not all of it — we tend to ignore the most effective way to make building those features easier and cost-effective: good solid architecture. It is this total system design view that we fail to deliver when it is needed: upfront.
For my book Mastering Digital: How Technology Leaders, Architects & Engineers build Digital Enterprises of the Future, I interviewed CTOs, CIOs, Chief Architects, and other senior technology leaders to understand what challenges they experienced in leading change in technology organisations. A common theme was the lack of a coherent big picture view of systems architecture (whether at the enterprise or solution level), and the problems that this created throughout the entire delivery process and beyond.
And again, in the Mastering Digital: State of Software Engineering Report, over 50 Architects and Engineers from around the world working in different types of organisations provided feedback on the issues they experienced day-to-day, and they found that the lack of coherent design and delivery-guiding architecture, as well as the bias toward features over foundations, has led to a lot of compromises and therefore technical debt.
I don’t believe there are any silver bullets for getting things right, but there are many poisoned chalices that will ruin even the best intentioned teams. Too many take the easier route of avoiding any real design at all and just write code, and many others get bogged down trying to wrangle the designs amidst the red fog of delivery.
For example, this ugly reality results in a project that should take 6–9 months and less than £2m, taking over three years and more than £20m, still with no coherent design or production-ready system. True story. Seriously. Though not unique: My consulting company provided a strategy review to an organisation for an in-flight multi-year programme to determine if they were a) on target to deliver their defined technical strategy and b) were going to deliver on time successfully. The answer to both of these, sadly, was ‘no’. There were, of course, multiple contributing factors, a lot of them to do with people, culture, and overall experience of the team. However, the lack of a robust architecture and upfront design made it almost impossible to meet their two primary goals (1: reduce operational systems costs and 2: create a technology platform for delivering re-usable service-based systems) — putting the delivery of a regulatory compliance system required by millions of people in jeopardy.
So, if doing everything upfront is problematic, and doing everything as you go is problematic — what do you do?
The Unpopular Opinion:
There’s no escaping the need to create a proper design if you want to create any kind of reasonably complex system that will have longevity. This design cannot be an afterthought, and it needs a lot of work upfront — it is far cheaper to redesign on paper than it is to write code (despite rhetoric to the contrary). A painful reality of systems development is that it is not just about writing a bit of code for that one feature. As an industry, we still lack the necessary maturity and discipline — we are a bit like a 5yr old football team chasing the ball around the field.
So the unpopular opinion I put forward to you is this: invest in an upfront design before you commit to building your solution. But, it is easy to abuse sound-bite strategies, so let me explain this a little better in the following sections, highlighting these key points:
You need to invest in effective foundational and flexible design upfront.
You need correctly qualified and experienced people to do it.
You need to adequately fund it and ensure it has the time & resources it needs.
You need to make this design the centre of your delivery work.
You need to improve the design based on real-world feedback iteratively.
Our role as architects of enterprise systems is to take control of the complexity and adapt the world to serve our needs better while preserving the integrity of the ecosystem. This requires a lot of understanding and thinking. It requires us to test and validate hypotheses and experiment without excessive cost, but also to integrate the many aspects of the complexity into a coherent whole. This is not simple, and designing things on the fly as you go is disastrously expensive in the long term — it just (unfortunately) looks productive in the short term.
Any fool can build software — but it takes a special kind of fool to accept an industry’s long term failures and technical debt as a successful business model.
Let's look at it this way: It is a bit like planning a trip to the other side of the continent, and simply walking outside your door and hoping for the best. At first, you are making positive progress — you are going somewhere. But you don’t know the journey or paths you could (or should) take — perhaps one is more direct, maybe one is less direct but faster. Perhaps it will rain or snow on some days, and you need to have found shelter by then (will there be any at that point?). Maybe you could have spent time upfront building a vehicle or just buying one, getting you there faster despite taking time to get started. Without forethought and planning, you will miss all of these things. However, do expect every design and plan to change when it hits the ground running — you will have to adapt to circumstances as they unfold. The better the design, the better it will cope with these changes. The absence of design does not help you here, but neither does a bad design. This is why you can’t use inexperienced people to formulate it. It requires the insight of experience, as well as several hard-earned skills and proven mental models.
Why this improves your enterprise systems
When you have a good, well-thought-out, and flexible design prepared upfront, you will begin to experience the following improvements:
Your plans will be easier to create, simpler to adapt, and more accurate.
Your systems will require fewer breaking changes.
Adding new features will be simpler and faster.
Changing the architecture over time will be easier and more predictable.
There will be a shared vision and understanding.
Putting this into practice
It takes hard work to deliver enterprise systems effectively. Creating that big design up-front is only one part of this and is not as ‘singular’ a task as it might first seem. Below are recommendations on how to make a design really work.
Create an Upfront Big Picture Architecture: This is your foundation for almost every other part of your delivery. It should be cohesive and comprehensive, including interactions with other systems and stakeholders. It should identify lower-level components and aspects for further design work, with enough detail to be able to define the deliverables for an implementation/delivery plan, requirements for technology, possibly a business case. Solution context is essential. It is about coherence, not completeness.
Create an Upfront Design Plan: Create a plan upfront of how you will develop the design, and at what stage of the process. This should include a design artefact map and a phased view of its maturation — either by adding lower-level design details (e.g. physical component designs) or improving a draft / speculative design with feedback from doing something real (e.g. PoCs).
Create Candidate Component Designs & Identify PoCs: It is crucial to start to flesh out how your big picture design will work in practice. Like how a specific component handles the processing you expect it to do and how it technically interacts with other elements. You need to investigate things that are not obvious or well understood. You don’t need to detail everything at this stage — you’ll come back later and refine. Do just enough. Where you are not sure about something, define a PoC (Proof of Concept) to test the idea.
Coherence vs Completeness: Detail and refine designs incrementally over time — your big picture architecture should describe enough of its characteristics to explain how it is structured and how things will interact with each other consistently - creating coherence in the solution. Then you can plan a series of lower-level designs that can iteratively build upon this macro perspective: first, get a rough design done to validate the solution context expectations, and then detail to the implementation level just in time for development.
Design for Change: You must consider as much of the solution context (inside and outside) as is possible to understand constraints, drivers, and both short term and long term goals for your design (even things out of scope right now). You can use this understanding to create a design that meets the bigger picture needs, whilst still being practical (and incrementally deliverable) enough for the initial scope of work. Change will always happen, but your design can survive and adapt better if you plan for it. Be mindful of dependencies, interfaces, and separation of concerns here.
Avoid Early Technology/Tool Coupling: The design should avoid too many tight couplings with technology/tools until you need to understand the technology’s constraints to make the implementation physical; otherwise, it will become brittle and difficult to change. You should consider the capabilities and constraints of applicable technologies/tools to help avoid surprises, but not as the bulk of the design. Use a solution/enterprise architect, not a technology specialist at this stage. Don’t select your tools before you design the big picture architecture.
Have a Plan and Plan for Change: Delivery plans will never be entirely accurate. That’s OK. The initial goal is to roughly shape resource requirements (team, budget, schedule, suppliers, etc.). Generally, it should describe your route, timetable, and status so that you can see if you are on course and align with other teams dependent on your plan. This kind of plan is not about daily tasks at this stage — that’s for lower-level planning, e.g. sprint planning. Continually improve it based on increasing knowledge of the work and performance. The plan should not be static. Always build iteration, feedback, and improvement time into the plan. This can work with a SCRUM agile approach or a simpler Kanban workflow, so there is no need to forego the value of these methodologies whilst still having a plan that goes beyond two weeks.
Continuous Governance & Continuous Feedback: Essential to the ongoing development of the design & solution is a close relationship between the architect and the delivery team. This comes in the form of support and guidance to clarify and explain the design, as well as a demonstration of the implementation to meet that design. Don’t forget the architect is a key stakeholder with sign-off authority. If the delivery does not match the design there is either a failure in communication or a delinquent team. This process needs to be done regularly and often to avoid last-minute surprises. The architect must listen to the feedback of the delivery team to understand why they can't implement the design or why the proposed solution is better (delivery teams have great ideas too). In practice, a weak architect will fail to make this work, as will a delinquent delivery team.
Work in Parallel Streams: The design stream has a lot of work upfront, which gradually tails off as the implementation becomes more concrete. The implementation stream should not start until the big picture architecture is done and you have a few candidate component designs, but once it starts with a few PoCs it can begin to ramp-up to full capacity. The big picture design scopes the ‘elements’ that need delivering and is used to form the plan’s sequence of design deep dives, PoCs, and development/build/test/etc.
Platform First: An often overlooked, but very effective way of delivering quickly and iteratively to a consistent, well thought out design over your projects with minimal per project overhead, is to create a platform first, then implement your projects on it. If you start every project from scratch with new ideas, designs, tools, frameworks, methods, techniques, patterns, etc., then you will have an ongoing battle in every single project to figure all of that complexity out. You will either have to wait for the (much bigger) big design upfront or feel the pain of designing as you go. Both will be undesirably expensive and time-consuming. Don’t do that. Build a platform or framework or service that has done all that difficult thinking work upfront, and produced a ‘product’-like solution that you can adopt just as you would install a tool from a commercial vendor, except this is precisely tailored to what you need for your environment. This needs to be properly productised though — don’t attempt to throw some code together and call it a product: you need a path to live, you need build and test automation, you need designer/developer/operator/support manuals and documentation, and even a service wrapper. It needs to be truly ready to use, and for all the use cases needed for your projects. Don’t be tempted to do this as part of the first project. It is a project in its own right or, more accurately, it will be an ongoing concern for a team for a long time — new features, maintenance, support, etc. — so create a dedicated organisation to manage it. Think of them as an internal supplier to other teams/projects.
A few final thoughts
Sound-bite strategies: they all ‘sound’ great, but even if there are some nuggets of wisdom that went into creating them, they usually hide the messy, complicated reality. This leads the inexperienced down difficult paths. Be careful in how you use them: be inspired but don’t apply until you have worked out the practical implications for yourself.
Practical theory: Despite valuing the designing and strategising sides of my skillset I also appreciate my pragmatic side and believe the practical reality to be a strong creative driver for the theoretical, as well as the theoretical being an essential shaper of the possible realities. Don’t underestimate either.
Bridging the gap: There is a disconnect between the thinkers and the doers, enterprise architects and delivery teams, dreamers and pragmatists, ‘business people’ and ‘techies’. These gaps appear everywhere: within teams and between them. Bridging these gaps is key to making enterprise systems coherent (and your organisation efficient). It is not easy, and joined-up thinking is rare. Most fail at the first hurdle. Without being joined-up, communication breaks down, and teams (and the systems they design, build, run) start to diverge and begin the deadly spiral into complexity.
Note: There are people, cultural, and behavioural challenges to address here too, though this is a topic for another day.
I will wrap up with a few observations:
Balance thinking vs doing: too much or too little of either is bad
Taking time for preparation is an essential strategy for success.
We are still very immature as an industry.
There’s a lack of joined-up architecture: we are siloed at every turn.
We need to balance the structured vs the creative natures of our work.
Ultimately, it is hard to build digital enterprises, but taking the time upfront to properly think through the approach and how all the bits fit together will make a significant difference to the success and quality of what you deliver.