In The Loop - Chapter 28: Overhauling a Technical System

24 min read

If you are a non-technical CEO (or senior executive) of a company that must migrate from a technical monolith to cloud-based microservices, this chapter is for you.

Given your role, a solid command of the rationale, resource implications and requirements of this technical migration is critical. Without it, you will underestimate what it takes. That means you’ll fail to prepare your board. You’ll fail to properly align the executive team. You’ll fail to provide enough support during the migration journey. You’ll fail to understand the organizational and cultural implications of migration. You’ll fail to coach your technical leaders as they encounter challenges along the way, and you’ll fail to detect when the migration train is veering off the rails. In short, you’ll fail.

The following is a monolith migration framework that will help you succeed (not fail).

If yours is a global 2000 public company that’s been around for more than fifteen years, some or all of your technical systems are likely comprised of aging, pre-cloud, monolithic technology. Even if yours is a rising startup, you too might be suffering under the constraints of a monolith — because the pressures of scaling may well have denied you sufficient time to refactor your technical systems into microservices.

You know you need to change. The reasons for change are many. Perhaps your competitors have built superior digital leverage, and your market leadership is threatened. Their digital capabilities, powered by microservices architected systems featuring a modern, cloud-based data infrastructure enable them to evolve more quickly than you — at a lower cost. You watch in frustration as they deploy advanced technologies such as IoT, AI and machine learning — running circles around you.

A monolithic technical system exhibits strong coupling between components. This means that an engineer can’t just update features at will. She must think through the impact of the change on everything else in the system. This fact forces technical teams into monolithic structures and waterfall (versus agile) software development methods. The monolith wages an incessant war against enterprise’s need to move quickly, act on data, leverage emerging technology and continuously adapt products and processes. The outcome is universal frustration and rising competitive risk. Here’s the important point: living with a monolith is not a technical problem. It’s an enterprise problem. A monolith gets in the way of becoming a fit systems enterprise.

But a monolith migration is not a decision taken lightly. Digital transformation is risky — and the bigger the required change, the riskier it is. For all too many enterprises, the overhaul of legacy technical systems has been a road to perdition. Despite investing multiple years, untold person-hours and tens (sometimes even hundreds) of millions of dollars, many large enterprises find their technical systems inadequately improved — still brittle, hard to change, incapable of supporting fast data applications, and expensive to run.

In Chapter 11, the case for digital transformation was made. In this chapter, you’ll learn the right way to approach digital transformation.

The chapter is organized into three sections:

  • The organizational and cultural implications of migration
  • The skill requirements of technical leaders
  • How to execute a migration

Organizational and Cultural Implications

Before you take on a monolith migration, you must come to terms with its impact on organization and culture. Monoliths create organizations and cultures that look and act very different than the organizations and cultures of companies with microservices based technical architectures.

When a company has lived with a monolith for a long time, its people have learned to adapt. Since changes to the monolith must be thoroughly thought through, digital product change occurs slowly. Innovation is incremental. Power flows to those who can control the pace of change, and who can ensure nothing breaks. Leaders constantly confront a mismatch between the business need for digital solutions and the monolith’s capacity. This forces tough prioritization choices, arrived at via heated debates that must often be resolved at the top.

Many monoliths feature centralized SQL databases, managed on premises. They lack the storage capacity and processing speed to support fast data, such as IoT sensor data or streaming data. These databases can’t handle semistructured data. As a result, it’s likely that other databases have arisen as appendages to the monolith. But since these databases don’t reside inside encapsulated services, specialized for each service’s unique needs, the capacity to bring data to every type of decision throughout the enterprise — to build a data-driven culture — is constrained.

This impacts decision making. In a data-rich environment, most decisions are informed by data — no matter who brings the data to the table. Facts democratize. But when facts are scarce, opinion reigns. Opinion-based decisions inevitably flow up to the highest paid person in the room. So the more data driven an organization is, the more likely it is that the company will have a high-empowerment culture. By comparison, with the monolith, there’s less data — and therefore (all else being equal) it’s a more top-down culture.

In monolith-burdened companies, innovation is eyed with caution. Leaders know they need to innovate, but they also know their capacity constraints. Innovation teams are sure to add new priorities to a long list of digital features waiting to wiggle through the monolith’s narrow development aperture. Because the prioritization of feature development is so impacted, the hurdle threshold is high. Innovation teams must present formal business plans with hard budgets. The budgeted return on investment must be high. These aren’t projections; they’re commitments. Since it’s so important to know the impact of the plan on development resources, the product vision, product requirements document and product road map must be detailed. This requires engineers and product managers to agree on scope and timelines in a “contract”: a classic waterfall approach to development. The high level of predictive accuracy this approach requires has the effect of making innovation incremental. No leader will sign on for a hard budget commitment to an unproven, “breakthrough” product concept. Such concepts are best developed within the product discovery system, where hypotheses can be freely tested and iterated in search of product /market fit. They don’t lend themselves to timeline predictions, and are best kept small (below the investment hurdle rate) until product / market fit is proven.

In Engineering, the monolith reinforces waterfall methods. The Product function is held at arm’s length; the Engineering function controls the means of production through formal approval processes. A few long-term senior engineers are the “keepers of all knowledge”; they know the intricate dependencies inside the monolith. They are key bridge players between Dev and Ops, ensuring that monolith changes are built properly so that the updates don’t break the system.

Companies with technical monoliths are usually characterized by top-down decision making. The decision process is formal, slow and careful. Knowledge of all dependencies is a first order problem. In such a decision culture, power flows to leaders who are effective at controlling the pace of change, or who possess a unique understanding of the monolith’s hidden secrets, or whose opinions have proven to be generally right in the past. Companies with technical monoliths are also behind the innovation curve. They can’t ideate, build, test and iterate at innovation speed.

For all these reasons, a monolith-to-microservices migration means shaking up the enterprise power structure, as follows:

Your top technical leaders, who have maintained their positions because they have been uniquely capable of enforcing control over the pace of technical change and reducing the incidence of monolith breakdowns, may be threatened by the rise of microservices and the autonomous teams and agile methods they enable. So too with senior executives who have grown accustomed to approving the product road map in gory detail. So too with leaders who have led by opinion — as data driven decision making takes hold, they might sense their authority being threatened.

Functional leaders may resist the rise of technical domain teams. As product managers and engineers work together with significant autonomy to tackle business outcome objectives, these functional leaders may feel they’ve been pushed to the sidelines. As the monolith decomposes into microservices and domain teams proliferate, functional leaders will find themselves shifting from controller to coach. Some won’t like the change.

And as to the rank and file, who have lived for so long in a top-down, control-based culture, the rise of innovation, risk-taking, and team-based structures will bring new accountability. Having lived in a “it’s their problem” world, this personal accountability will be frightening and disorienting.

All of this is that much more challenging, because it can’t happen overnight. Depending on the company’s (and monolith’s) size and complexity, this shift may require from two to five years to execute. As a result, leaders must “live in two worlds” for a sustained period of time.

The point is that digital transformation is enterprise transformation. You aren’t just changing the technology — you are adopting a flatter, more agile, more data driven, more democratized, more teams-based, more cross-functional, more innovative organization and culture. It must occur at the pace of monolith migration. If you execute well the monolith will decompose, and your organizational and cultural transformation will unfold, steadily but slowly.

As CEO, it’s on you to come to terms with these organizational and cultural implications. You will need to prepare your organization — from your most senior executives on down — for the migration journey, and then provide continuous support and encouragement along the way. Some won’t get on board; when a key player proves he can’t change, you’ll need to make the change.

Top Team Strategic Alignment

It is a non-trivial challenge to bring your top team along with you as you envision the future and embark on digital transformation. So it’s best to involve them early. Create a project team at the top. Engage them in early design. This will help you build change agents at the top who share your commitment to the migration. As time goes on, you will observe how each senior executive responds to transformation. Eventually, you may determine that some senior executives are too wedded to the current power structure. Perhaps they have become impediments to change. If so, you will need to replace them.

Engineering Organization and Culture

Conway’s Law posits a link between the architecture of your technical systems and the architecture of your organization. Monolithic technology will mean a monolithic engineering organization. As you refactor the monolith, this will prompt the refactoring of your engineering function. Microservices enable you to create technical domain teams assigned a business purpose, with the autonomy to act in pursuit of business outcome objectives. Since digital transformation is often a multi-year migration, you will soon find yourself with two very different engineering groupings, both of which must be maintained for a sustained period of time. One large grouping of engineers will be responsible for the monolith. This group will be constrained to maintaining and updating the code base with extreme care, probably via a waterfall method — in order to identify all dependencies before updating anything. Meanwhile, the other grouping of engineers will be organized into autonomous domain teams, with freedom to update code and add new capabilities inside their assigned domains at will.

These teams will look and act completely different from each other. Those working on the monolith will be sensitive to the rise of these autonomous development teams, which enjoy freedoms they don’t have. The most senior engineers who work on the monolith will know that their power (which has come from knowing all the complex dependencies inside the monolith) is in decline. It will take smart leadership and delicate career pathing to ensure most engineers, especially in the monolith group, feel valued and find themselves doing interesting work.

Product Organization and Culture

Product leaders will experience a big adjustment as the migration progresses. Whereas in a monolith-centric world the focus is on centralized product roadmap development and feature prioritization, in a microservices and domain team world the focus is on establishing the proper domain boundaries, defining business outcome objectives for each domain, stocking domain teams with 10X product managers, and getting out of the way. Of course there is coaching and support. But product leaders must make the shift from a control paradigm to a coaching paradigm.

Skill Requirements of Technical Leaders

If you don’t have top echelon technical leaders experienced in monolith migration, your first priority will be to put them in place. These leaders must appreciate the reasons for migration and the implications of migration on organization and culture. They need the technical competency and experience to design a smart migration path. They must possess effective leadership skills, so as to manage the people who will execute the migration while maintaining existing systems. And perhaps most important, they need to possess sound judgment: to balance the business’ continuous needs for new digital features with the simultaneous need to make migration progress.

Executing a monolith migration involves a series of incremental steps that occur over a sustained period of time. The leadership challenge is that the business benefit of these migration steps tends to be back-end loaded. Meanwhile, the business marches on: a chief revenue officer needs a new product feature to fend off a competitive threat. The CFO needs a new data feed implemented, connecting the product platform to the accounting system. The digital feature demands will just keep coming.

Because of this conflict — the long term work on monolith migration, at odds with the short term work on immediate business needs — the risk of rising organizational resistance is high. Many will resent the diversion of resources into migration work. They will perceive that enormous investments have been poured into an initiative yielding skimpy fruits, while critical features lie waiting to be built. It takes smart, wise, resilient, high-EQ leaders to properly resolve these priority conflicts.

Here are the technical roles that will be at the center of a monolith migration initiative:

  • Chief Product Officer / VP Product
  • Chief Technology Officer / VP Engineering
  • Chief / VP Security Operations
  • Chief / VP Data and Privacy
  • Chief / senior / principal architects

Product and Engineering

Two roles are of particular importance in monolith migration: the head of Product, and the head of Engineering. The job of the Product function is to ascertain and prioritize the business’ digital needs, and then move them into and through the development process. The job of the Engineering function is to execute product development that delivers the intended feature / functionality, while building, improving and maintaining infrastructure that is secure, mutable, resilient, elastic and cost-optimized.

It is easy for these two roles to become at loggerheads. Product, representing the business, pushes for the fastest path to a new feature — even if that means an update to the monolith to get it done. Engineering, responsible for infrastructure, pushes back — because such a step would delay and complicate monolith migration.

For these reasons, heads of Product and Engineering should possess similar backgrounds. Each should have some experience having lived in the other’s function. The head of Product needs to have deep technical competency. The head of Engineering should have experience in (or working closely with) Product. Both must have experience with domain driven design. Both need to understand reactive microservices architecture, modern cloud-based data and networking infrastructure, modern technical capabilities, security and permissions issues, disciplined agile delivery methods and the power of self-organized domain teams. Both must know how to lead self-organized domain teams, and how to increase the density of 10Xers within them. And of course they will need to have past experience in executing a monolith migration.

Both must know how to juggle and ride a bike at the same time. The monolith will remain in place throughout the migration, so they must lead the monolith teams with a control-based approach, while leading the rising microservices teams with a high-autonomy approach. It’s not an easy task.

Because of all the tradeoffs at play in a migration, it’s important to align Product and Engineering leader incentives. It’s hard enough to keep these functions aligned under the best of circumstances: if the incentives are out of sync, it becomes nigh on impossible.

SecOps, Data and Privacy

Leaders responsible for digital security, data and privacy are critical supporting actors in the monolith migration. Working with the heads of Product and Engineering, they ensure the migration is executed such that at each step both the monolith and the new microservice are secure and compliant with the data permissions hierarchy. No leaky buckets.

The Architects

Senior architects are critical to effectively planning the migration path and executing monolith decomposition. As such, you need true 10Xers in these roles. They must possess a clear understanding of the desired microservices end state. Because of all the dependencies inside the monolith, they will also require a deep understanding of the monolith’s current state. They will need to think through the dependencies between microservices and across distributed systems, so that as the decomposition progresses all required dependencies continue to be fulfilled. And of course they must possess an authoritative understanding of optimal methods in monolith decomposition.

How to Execute a Migration

As discussed in Chapter 11, digital transformation doesn’t stand alone. You migrate your monolith because it’s holding back the business. So migration must begin at a strategic level — by assessing the enterprise’s current state in comparison to its generative and adaptive imperatives. An outward-in, customer-centric, future-to-present approach ensures that digital transformation is grounded in the overall purpose of the enterprise.

An Evolving Domain Model

It starts with you as CEO, and by extension your top team. At the outset of a digital transformation initiative, you will gather a handful of your top technical and business leaders and launch an executive level project. Choose your most trusted lieutenant as team leader, and give the team a written charter (objectives, leader, members, timeline and member commitment expectations).

The team’s first step will be to devise the business strategy — as discussed in Chapter 11.

The next step will be to extract from that work the critical technology implications. What is your enterprise’s position in the ecosystem compared to competitors? What capabilities must you develop to win? What does winning require, in terms of speed of digital development, data ubiquity, talent, culture and other factors?

Then the team will need to map the systems and domains inside your enterprise, and the stocks, flows and feedback loops that exist within them — as they should be. What design of the company’s systems and domains would best position you to win in the ecosystem? What systems and domains are most critical to your future? How might these systems and domains become more aligned with the business opportunities and challenges you face? Be sure the team is looking across people, workflows, technology and money flows. Challenge the team to let go of function-centric view of the enterprise, and adopt a systems view. Functions sit inside systems.

To map a system, you need to know its purpose. Ask the team to define system purpose in the context of company vision and ecosystem reality. And if you double click within each system, each domain has a purpose. Ensure these domain purposes are in alignment with each other. With purposes clear, the team can then map the people, workflows, technology and money flows inside the systems and domains — as they would look in a redesigned future, and as they are now. These two maps are not yet at the technical level — they are just a high level conceptual maps of the business workflow.

To aid in this effort, you may want to review the reference architectures for the eight operating systems shared earlier in this book. While they will need to be customized to your reality, you might find them a good starting point. The mapping at this stage should be of the enterprise’s operating systems, not its meta systems. As a reminder, here’s the overall enterprise schema:

And just by example, here is the reference architecture for the accounting system.

Similarly, here is the reference architecture for the revenue engine system.

Reference architectures for other operating systems can be found in Chapters 13 and 14. They are a good place to start, but you will need to customize them to fit your own reality.

As the team progresses, you may detect resistance from some members of the executive team, or even project team members. They may realize the future state map will influence the company’s future power structure. Some might initially propose future-state designs quite similar to current state, or that advance the interests of one executive over another. Be careful. These ideas might be a power play or a defensive maneuver — or they could be well thought through, appropriate inputs. You, the project team leader and other executives will need to exercise strong discernment and leadership, especially in the early stages of the work, to guide the team towards a future-oriented, systemic, enterprise-level view.

The senior project team must then identify the places on the map where future state is most at variance with current state. Is it in certain product features? Is it inside the revenue engine system, such as onboarding? Is it in dynamic pricing? These points of variance will hint at where you will need to focus first. How does the monolith support current state workflows inside these parts of the map? To what degree do the monolith’s limitations keep these parts of the map from progressing towards future state?

Towards that end, the team will need to conceive how new technical capabilities (IoT, AI, big data analysis, blockchain, etc.) could be applied inside redesigned systems and domains to elevate the company’s generative and adaptive imperatives. Could we make better products? Could we become more resilient, more scalable and more efficient? By thinking through new technology, the future state map will evolve. This is normal and good — the map is meant to be an iterating, evolving document.

Then ask the technical leaders to identify all the technical barriers to this envisioned future. How does today’s monolith impair speed of software development? Can features be worked on independently, or do changes to a feature require an upgrade to the entire monolith? Can the monolith deliver real-time decision analytics drawn from streaming data? What is the monolith’s data processing capacity? What would be the cost of storage and compute if we were to implement IoT solutions and create a high-volume stream of incoming data?

Notice that this top team project begins at the beginning: by defining how the enterprise must change to meet its threats and opportunities. By reconceiving of the enterprise in systemic terms, in light of your vision and emerging ecosystem realities, you start digital transformation the right way. The team must validate its future state map with solid analysis. It can then define the gaps between current and future state with confidence. This exercise will inform the migration path. It should also help make the case for change, important for bringing along your board and mid management executives.

This top team project is likely to require multiple months of work (depending, of course, on enterprise size and context complexity).

Remember that no model is reality. It’s just your best approximation of reality at any given point in time. So the model this top team creates will evolve. The same will be true of the models built at every level of abstraction, from this initial high-level enterprise-wide model all the way down to the detailed technical models. Every model at every level of abstraction will go through many iterations, and even when they feel “done” they won’t be done. Models are never done, because the context is always in flux.

The high-level system and domain maps and senior project team conclusions become the foundation for a domain driven redesign of your enterprise’s technical systems. If the project team has been successful, team members will have deepened their commitment to the migration. And they will better appreciate the time and resources it will take. These team members, especially the senior executives on the team, will become critical partners in making the case for change: up to the board, across the executive team and down into mid-management and beyond.

Conceptual Domain Design Teams

With the enterprise now expressed in terms of its systems and domains (current state and future state), it is time to dig deeper. At this stage, we are not yet dealing with technical-level requirements. The next objective is to zone in on the domains that the high level map has identified as most critical to the future. We now want to model more detailed business logic of current state versus future state inside these domains.

The gaps between the current state model and the future state model will help technical leaders and other top executives prioritize which domains would benefit most from technical refactoring — and which are hardest to change due to technical complexities. Let’s say that across the enterprise, two systems and six domains (three in each system) have been targeted as top priority problem areas. To take the next step, a project team is needed for each domain — comprised of a product manager, one or more technical architects, some senior engineers, and one or more business managers who work inside the domain.

On each project team, the domain owners and technical experts must define the problem domain and its subdomains with rigor. As Eric Evans argued in his book, Domain Driven Design, this involves defining the subdomains inside the problem domain, their boundaries (the bounded contexts), the transformations that must occur within the bounded contexts, and the dependencies between bounded contexts. (Suggestion: ask domain team members to read Evans’ book.)

Domains are made up of core subdomains (responsible for a primary function), support subdomains (which provide specific support to the core subdomain), and generic subdomains (which provide a type of support to the core domain that is replicated elsewhere in the domain or in other domains). The boundary of a subdomain is called its bounded context. Each subdomain should do one thing and one thing well. Within the subdomain, there is an input, a transformation and an output. Transformations invoke changes in data state. Rules that define triggers to a data state change should be explicitly defined.

At this stage of mapping, we are primarily focused on the core subdomains. Support and generic subdomains can be called out, but don’t need to be modeled in detail.

Wherever dependencies exist between subdomains, it’s best to design their dependent steps into the beginning or end of each subdomain’s assigned transformation. The dependencies between subdomains are captured in a context map. As you model these dependencies, you should iterate on the design so as to maximize isolation of work inside a subdomain. The idea is to minimize external dependencies. This domain driven design approach will set the stage for development — and enable a reactive microservices architecture.

To ensure close collaboration between non-technical business owners and technical teams, language matters. The language used by business owners should be consistent with the language used by engineers. This “ubiquitous language” facilitates shared understanding and alignment.

Retaining shared meaning at every level of abstraction is critical to domain driven design. Business owners will be surprised by how much time must be spent to define domain entity terms and domain business methods. Ubiquitous language keeps everyone on the same page. As engineers start writing code, they will actually use these “ubiquitous language” terms (for entities and methods) in the code itself.

Remember that each team is working within one problem domain. After teams define the future state domain-level model, they do the same thing with current state. The current state map will stipulate the transformations that are executed within the monolith, and how these transformations impact people, workflows and money flows.

Domain Data Implications

In modeling how the monolith operates inside a domain, teams will start with the data. Data is always tricky. The Dimension Research study found the following:

“Almost all large companies (98%) report that they are facing challenges with their data with the most frequently reported problem being a need to manage multiple data silos (57%). Other issues reported include the high cost of database management software (45%), concerns about data security (39%), performance issues (33%), inability to distribute data where it is needed (33%), lack of access to real-time data (32%), scalability (28%), and more¹.”

Kurt Cagle addressed the problem in the Forbes article “Why Most Digital Transformations will Fail”:

“A key aspect of digital transformation is metadata management. This means not only determining what resources you are interested in, but also what databases contain relevant information about those resources. The central goal of digital transformation is to make your data findable and addressable. This is actually a pretty critical function — there are tools that allow you to track APIs, but these usually do not give you a context for saying ‘if I want to find information about customers, who has that information, how is it addressable and what keys do I need to use to get it²?’”

As the domain teams identify data sources and uses, they will record the information in an enterprise-wide knowledge graph. A knowledge graph defines the relationship between data entities relevant to the domains in question. It is often presented in the form of an ontology.

There are actually two knowledge graphs — current state and future state. These knowledge graphs sync up with their corresponding context maps, which define the dependencies between domains (current state and future state). A knowledge graph doesn’t need to be exhaustive, but should capture the most active data dependencies (think 80 / 20 rule). Separately, teams can work together to build a data catalog (a taxonomy) that identifies where domain-relevant enterprise metadata can be found. The context maps, knowledge graphs and data catalog will come in handy when the technical migration stage begins.

Domain Team First Stage Deliverables

The domain teams will have completed their initial projects when current state and future state have been defined in terms of the people, workflows and technology. As to the technology, the domains and subdomains will be defined at the conceptual level (current state and future state) for the application layer, database layer, integration layer and infrastructure layer. Context maps, knowledge graphs and a data catalog will have been created. The team will have identified the technical realities of the monolith as it pertains to these domains — its attributes, issues and constraints.

This work could take the assigned domain teams many months to a year or more to complete, depending on scope and complexity.

Determining the First Microservices and Decomposition Sequence

By now, the technical domain teams have built up enough knowledge to determine how to plot the attack on the monolith. The principle is to start small, create a quick win, and expand from there.

This “low hanging fruit” approach is important, because you want to maximize the likelihood of early success. For your first project, you may not refactor anything in the monolith at all. As a first step, you may just build a new feature that sits outside of the monolith, designed via reactive microservices architecture and resident on the cloud.

In this approach, your goal is to mobilize one technical team and solve one meaningful problem. Assign the business owner who operationally runs the domain to the team. As Chris Richardson advocated in his article “Refactoring a Monolith into Microservices”, the first step might look like this³:

The glue code comprises the anti-corruption layer, which is necessary to translate between the monolith and the spotless new microservice you have created. Anti-corruption layers are hard to build, but they are critical. The new service will have been executed well if updates to it can be built and deployed, and the service scaled, independent of the monolith.

Soon you will need to refactor some transformation that is currently executed by the monolith. For your first refactoring project, choose a domain and set of subdomains that would deliver high impact if refactored, but which aren’t very complex to refactor.

This brings us to a key point. When refactoring a monolith, avoid — at all costs — the “big bang” approach. With this approach, you would rebuild the entire monolith from scratch using modern microservices and cloud-based methods, and then “switch over”. That’s a fool’s errand. There’s way too much complexity to take that approach. A better approach is the strangler method. First popularized by Martin Fowler in a 2004 post, “Strangler Fig Application,” the strangler method takes an incremental approach⁴. Component by component, you refactor the highest impact / lowest complexity projects first. As you gain confidence and prove success, you can expand the number of teams executing refactoring work and can increase the complexity of the projects. Step by step you strangle the monolith, until it slowly withers away.

Complexity comes in many forms. It may be that the monolith exhibits strong coupling between the database and the application layers, or between the application and integration layers. Or perhaps, as with Zuora, you need to support backward version compatibility because of pre-existing customer integrations. Or perhaps there are indirect side effects of existing monolith transformations that are not yet fully understood. That’s why it’s important to start simply and take on more complex projects only as you increase knowledge and confidence.

After the first couple of projects (such as the building of your first new feature outside the monolith, and the first refactoring of a monolith component), you may be able to begin breaking the monolith into mini-monoliths. This will advance your decomposition efforts. In his “Refactoring a Monolith into Microservices” article, Richardson proposes a natural “seam” that exists in most monoliths:

“A strategy that shrinks the monolithic application is to split the presentation layer from the business logic and data access layers… There is usually a clean separation between the presentation logic on one side and the business and data‑access logic on the other. The business tier has a coarse‑grained API consisting of one or more facades, which encapsulate business‑logic components. This API is a natural seam along which you can split the monolith into two smaller applications⁵.”

He presents the concept graphically as follows:

Richardson, Chris. “Refactoring a Monolith into Microservices.” NGINX, 29 July 2019, https://www.nginx.com/blog/refactoring-a-monolith-into-microservices/.

As the migration progresses, you will begin to take on tougher decomposition projects — where the complexity level is higher. By that time, you will hopefully have multiple engineers experienced in the refactoring journey, knowledgeable about the monolith and ready to take on the task.

Project Teams

So what about these project teams?

Every team needs an operational business owner. This may or may not be a product manager. The business owner plays a critical bridge role in digital transformation. This person is responsible for helping the technical members of the team understand the business context. But she also communicates with other operational workers inside the domain. Just like a bidirectional API, this person makes sure that both teams — operational and technical — are kept in alignment.

As to the rest of the team, membership depends on the work to be done.

The data layer is comprised of event data, process data, transactional data, rules data, configuration data, reference data and data used for analytics and reporting. Each is managed inside a subdomain. Data architecture rules apply to how data is created, accessed, modified, and deleted within each of these data subdomains. For this you need a data architect.

The application layer is comprised of channel components, process components, activity components and task components. Each of these component types is considered an application subdomain. Application architecture rules apply to the selection of frameworks used in building and maintaining each component type. Channel components are specific to the technology used to collect data from users, sensors and other source devices. Process components are channel unaware, allowing for the implementation of omni-channel workflows. Activity components are process-unaware and are used to coordinate access to task-level components. Definition of these rules is the responsibility of an application architect.

When application systems are split into separate front-end and back-end run-time environments, this separation usually occurs between the process and activity subdomains and their corresponding services. Whenever this occurs, an Application Programming Interface (API) is required. When application systems are split into separate front-end and back-end run-time environments that span legal entities, the API is extended to form an External API (XAPI), which often involves mapping payloads to industry-standard code sets. For loosely-coupled distributed systems architecture to work effectively, application logic must be kept out of the integration layer. This too is the responsibility of an application architect.

The integration domain is comprised of local integration, enterprise integration and cross-enterprise integration subdomains. In loosely-coupled distributed system architectures, integration subdomains are the key to balancing agility and complexity. Definition of the rules that govern the integration subdomains is the responsibility of an integration architect.

The infrastructure layer is comprised of network services, security services, storage devices, servers and shared services. Each of these infrastructure types is considered an infrastructure subdomain. Infrastructure architecture rules apply to the design and implementation of managed services that define a given hosting environment. Definition of these rules is the responsibility of an infrastructure architect. The most common form of managed services offerings are cloud hosting environments. Generally speaking, the separation of concerns between application and infrastructure domains is well-established, although things like load-balancing tend to creep into the application domain, adding unwarranted complexity to the application system. For this work you need an infrastructure architect.

Leveraging disciplined agile delivery methods, engineers work with these architects to write the code and build and deploy the software.

Rinse and Repeat

A large monolith may consist of hundreds of modules. In a perfect world, all would migrate to the cloud, reconstituted as reactive microservices. But of course that’s a multi-year endeavor — and may not even make sense. If the monolith does the job without excessive cost, then let it keep doing the job. The guiding factor is always pragmatic: is the pain of staying inside the monolith greater than the pain of refactoring?

You will refactor the most important modules first, especially the ones that must change most frequently or that need to leverage new technical capabilities such as streaming data. You also want to move modules out of the monolith that have high data storage or compute requirements. These can be operated much more efficiently on the cloud as microservices. You’ll also consider the complexity of extraction. A module is easier to extract from the monolith if its data dependencies have been managed with asynchronous messaging instead of tightly coupled, synchronous messaging. These factors will influence your refactoring sequence.

Of course, your technical teams will evolve and learn. Over time, a growing number of architects, engineers and product managers will gain experience in monolith module migration. As the understanding of the monolith and refactoring experience grows, you can move on to more complex extractions.

Summary

As you progress on the journey of digital transformation, your organization and culture will transform. Decision making will become more team-based, data driven and agile. Your organization will become more domain team-based (both functional and cross-functional). Knowledge will become more distributed and data more ubiquitous. You will spend more time clarifying domain business outcome objectives, and less time micromanaging product decisions. Leaders will shift from controllers to coaches and talent optimizers. You’ll increase your focus on innovation.

With technical systems increasingly built via reactive microservices architecture on modern cloud-based infrastructure, you will be able to embrace emerging technologies such as big data, AI, machine learning, IoT and blockchain. You now have the technical capacity to deal with streaming data and spiky compute loads.

As your migration progresses, the most complex technical problems will not be inside the microservices — or even in the migration itself. Increasingly the greatest technical challenge will be in managing the connections between services. Issues such as service discovery, failover, security, replication, coordination, metadata management and integration with other systems will predominate. These integration challenges will exist between subdomains, domains, systems and across the entire enterprise. You will assign your most advanced engineers and architects to these types of integration and data issues.

But if you do it well, it will all be worth it. Because you will have built the technical foundation for a fit systems enterprise.


To view all chapters go here.

If you would like more CEO insights into scaling your revenue engine and building a high-growth tech company, please visit us at CEOQuest.com, and follow us on LinkedIn, Twitter, and YouTube.


Notes

  1. “New Research Report: 2019 IT Architecture Modernization Trends.” Datastax, https://www.datastax.com/resources/report/new-research-report-2019-it-architecture-modernization-trends.
  2. Cagle, Kurt. “Why Most Digital Transformations Will Fail.” Forbes, Forbes Magazine, 7 Jan. 2019, https://www.forbes.com/sites/cognitiveworld/2019/01/07/why-most-digital-transformations-will-fail/#2cce53e520ee.
  3. Richardson, Chris. “Refactoring a Monolith into Microservices.” NGINX, 29 July 2019, https://www.nginx.com/blog/refactoring-a-monolith-into-microservices/.
  4. Fowler, Martin. “StranglerFigApplication.” martinfowler.com, June 29, 2004. https://martinfowler.com/bliki/StranglerFigApplication.html.
  5. Richardson, Chris. “Refactoring a Monolith into Microservices.” NGINX, 29 July 2019, https://www.nginx.com/blog/refactoring-a-monolith-into-microservices/.
Tom Mohr

Leave a Reply

Your email address will not be published. Required fields are marked *