In previous articles about ESB solutions, KT.team experts explained why such solutions are needed (see article”A tool that will help keep your business young →”), which of them are popular on the Russian market (see article”Comparison of ESB solutions that are popular on the Russian market. Part 2: system features→”) and what steps will be taken when implementing ESB (see article”From spaghetti to flexible architecture in 5 steps: stages of implementing an ESB layer→”. Let's say you have already decided that the introduction of a service tire will be the best solution to your problems. The next step is to understand how much effort and resources it will require.
This article is intended for executives who are responsible for implementing ESB, controlling the budget, the choice of the technology stack, and the implementation process itself. It will help you take into account the main costs that are required for budgeting when implementing yourself, and draw attention to the key aspects of negotiating with a contractor if you need his help.
We will help you deal with frequently asked questions.
- What are the steps to implementing ESB and how long does it take?
- What are the costs of implementing and owning ESB solutions?
- What specialists are needed to implement and support ESB? What do they usually do on their own, and why do they hire contractors more often?
- What are the most common mistakes when implementing ESB and how can you avoid them?
Reasons for implementing ESB
As a rule, companies start implementing a middleware layer after working on one of two typical scenarios.
The first scenario: For many years, the company's IT architecture has grown around a monolithic all-in-one system. In the first years of the company's life, it covered all the business needs in terms of IT, then the team completed and finalized it... until eventually abandoning any part of this system became too complicated and expensive. At the same time, the IT department (and other company departments) is well aware that the market has much more flexible, convenient and easy-to-maintain systems for specific company business processes. And now you are faced with the task of replacing the heavy monolith with separate services and microservices.
Scenario two: The IT architecture already consists of at least a dozen systems connected by point-to-point integrations. Adding new systems to such a circuit is expensive and time-consuming, while maintaining existing integrations is expensive. At the same time, data from old systems is uploaded in complex and inconvenient formats. You need a middleware layer to untangle integrations and make the architecture more flexible and easy to maintain and upgrade.
At the same time, other departments constantly contact the IT department with recurring problems and tasks:
- marketing cannot freely develop channels to attract users, since traffic from successful advertising campaigns creates the risk of the system crashing;
- the sales department cannot implement a promising product or integrate, as there will be a flow of messages between applications that your system will not be able to cope with;
- the group of companies acquires new enterprises, but cannot fully benefit from integrating their data stored in heterogeneous IT systems (for example, aggregating data in real time and creating unique automated products or analytical tools).
Implementing ESB is a logical step. But without experience in implementing similar projects, it is quite difficult to predict how long this process will take, how much it will cost at the implementation and ownership stages, and what its economic effect will be.
First things first. First, let's look at the sequence of actions when implementing an ESB layer.
Pre-project survey
At this stage, a team of experts studies existing data flows, builds a scheme for the future architecture, and calculates the cost of implementation.
Not all IT integrators who specialize in implementing ESB systems are ready to provide analytical services separately from project implementation. Therefore, in order to isolate this stage from the total cost and duration, we recommend collecting and considering several commercial offers.
As for KT.team's practice, pre-project analytics is a separate service in our company. As a rule, the analytical phase takes from two to four weeks (depending on the scale of the existing IT architecture and the number and complexity of data flows) and costs approximately 1.4 million rubles.
The stage seems optional, since the IT director usually knows the IT outline of his company very well. However, KT.team's experience has shown that if we neglect the pre-project study, the tire implementation process will slow down and the cost of implementation will increase.
Thus, on projects of comparable scale, the implementation of middleware without a preliminary analytical stage took 1200—1500 hours of development, edits and improvements, and with pre-project research, 600 hours.
ESB implementation stages
We identify five stages of implementing ESB into the enterprise's IT architecture.
We will tell you more about these stages and their duration in the article”From spaghetti to flexible architecture in five steps: stages of implementing an ESB layer”, here we will describe them briefly.
The first step is choosing where to store your ESB layer. We suggest starting this stage before choosing a technology stack, as existing rules and legal requirements may limit your choice of the main system and additional components. For example, make is only suitable if you agree to work via the cloud. And WSO2 can also be installed on the company's dedicated servers.
The second stage is choosing an integration development environment. Throughput, scalability, compatibility with logging and monitoring systems, financial model, etc. — dozens of key parameters are taken into account. In KT.team's practice, this stage is included in pre-project analytics: we provide detailed information on three systems, which allows us to make a more balanced decision.
The third step is to select the ESB component stack. Here you have to choose (on your own or with the help of an integrator): message storage (for example, a message broker or database); logging and monitoring systems; hosting for deploying other components — in-house or cloud-based (IaaS provider).
There are specialized SaaS solutions for building ESB that provide hosting and a stack of ready-made components. Most of these SaaS are foreign.
The fourth step is to deploy and configure ESB components. It usually lasts about a month and a half before the MVP launches. Once the components are deployed and configured, the IT team begins to generate the first data streams between applications. At this stage, the bus coexists with familiar architecture and integrations. As a rule, a team starts with the most complex flow and then creates the rest based on it.
The duration and, accordingly, the cost of work can vary greatly, but, in our experience, on average, a project to implement ESB to the MVP level costs 1.5—3 million rubles.
The fifth stage is documenting the implemented ESB. All actions and rules for creating integrations are recorded and described by the development team. Properly written documentation will allow engineers unfamiliar with your ESB to configure new integrations in your existing architecture according to accepted approaches. The documentation process takes about 40 hours.
ESB implementation and ownership costs
When implementing and operating an ESB, the following costs are inevitable.
- Acquisition of licenses for components that are not free software. For example, ESB Mule is an open source product, but some of its components (Anypoint) are available only as part of a paid annual subscription.
- IT infrastructure organization: buying and depreciating equipment, renting a server rack in a data center, or purchasing servers, IaaS, SaaS, etc.
The cost of owning licenses and maintaining the infrastructure for a medium-sized project is 100—300 thousand rubles per month.
The optimal team for implementing ESB
If you plan to implement ESB with your own team, you will definitely need the following specialists:
- a senior engineer or IT architect who will audit IT systems and integrated applications, monitor the choice of the technology stack and develop a bus implementation strategy;
- a middle engineer who implements integration (you can involve additional engineers in the project and work on several tasks in parallel, thereby speeding up the overall process).
An important caveat: the team should be familiar with the low-code approach, or better yet, directly with ESB. Experience in creating the right integrations through middleware will help you avoid “beginner mistakes” and prevent the project from becoming more expensive.
But since businesses do not constantly need these competencies, companies often do not employ engineers with such specialization and experience in implementing ESB. In this case, the relevant work has to be outsourced.
The optimal command for operating the ESB
For the regular operation of the introduced tire, at least one citizen developer specialist is required to perform the following tasks:
- analysis of incidents and resolution of emerging problems;
- making changes to existing integrations;
- creating new integration flows.
If the ESB is well documented at the implementation stage, and the logging and monitoring systems are working properly, one person will be enough to operate the tire.
In our experience, companies usually employ full-time employees to perform such tasks and only turn to IT integrators who implemented and prepared documentation for complex issues.
ESB implementation cases from KT.team's experience
The technology stack, the timing of implementation, and the team that will implement it depend on a variety of details that are unique to each case. We will give as an example two cases that clearly demonstrate the specifics of different ways to implement ESB. For the purposes of this article, we are unable to disclose the details of our clients' situations. Here is only general information that gives an idea of the actual cost of implementing a tire.
Case one: a Russian company in the heavy engineering industry
Company size: more than 10,000 people, more than 10 industries.
Reason for implementing ESB: New manufacturing enterprises are joining the group of companies, as a result, it is necessary to integrate their IT systems, processes and data into the IT infrastructure of the entire group.
Implementation period: two months from signing a contract to launching an MVP.
Implementation phases:
- analyzing data and studying the current architecture (two weeks);
- development and configuration of a pilot project (four weeks);
- commissioning and scaling preparation (two weeks).
The specialists involved in the customer's staff: project manager, data methodologist, IT teams for new industries and the group's management company.
The specialists involved in the contractor's staff are: project manager, tech lead, ESB integration developer.
Hosting: own servers (24-core processor, 30 GB of RAM, 480 GB SSD drive).
Technology stack for ESB implementation:
- connectors and visual integration development environment — WSO2;
- message broker — Apache Kafka;
- logging and monitoring — ELK Stack;
- dashboards and notifications — Grafana
Paid licenses: not required, all components are free software.
The cost of integrator services: 3 million rubles — pilot project.
Case two: production and sale of Chinese goods in Russia through its own e-commerce platform, retail chain and marketplaces
Company size: more than 300 people.
Load volume: ERP support for more than 10 sales channels, 50 categories and 300 product subcategories.
Reason for implementing ESB: the need to organize a stable upload of data to an ever-growing number of sales channels (including affiliate channels), automatically taking into account their requirements for the information provided.
Implementation period: four months from signing a contract to launching a project.
Implementation phases:
- analyzing data and studying sales channel requirements (four weeks);
- designing, configuring and testing integrations on pieces of data (two weeks);
- scaling integrations to other data (eight weeks);
- transfer into operation (two weeks).
The specialists involved in the customer's staff: project manager, ERP programmer.
The specialists involved in the contractor's staff are: project manager, tech lead, ESB integration developer.
Hosting: provided by a cloud provider.
Technology stack for ESB implementation:
- connectors and visual integration development environment — WSO2;
- message broker — RabbitMQ;
- logging and monitoring — ELK Stack;
- dashboards and notifications — Grafana
Paid licenses: not required, all components are free software.
The cost of integrator services: 4.5 million rubles — pilot project.
Infrastructure cost: 100 thousand rubles per month.
The most common mistakes when implementing ESB
The first mistake is integrating applications without logging and monitoring subsystems. Ultimately, a company that has implemented such a stripped-down ESB layer will learn about incidents reactively. Imagine a situation: logisticians must send the paid item to the customer, but they do not find it in stock and report that the actual inventory balances do not match the data in WMS.
The second common mistake is the implementation of complex integration mechanisms in ESB. Let's take an online store as an example. To send a product card to the marketplace, you need data on balances, prices and positions that are stored in three different systems. Inventory information is updated every 15 seconds (about 40,000 times a week), price information is updated once per hour (168 times a week), and position information is updated once a week.
- If, every time the information in the item card is changed, a request is sent to all three systems, excessive data exchange will occur. In most cases, only balances change, which generates 40,000 hits to each of the three systems every week, for a total of 120,000. At the same time, the PIM product information system processes 40,000 times more requests than is really necessary!
- If you access each system separately, only when the data within it changes, you will get approximately the same 40,000 messages per week in total — three times fewer. With this implementation, databases that store information about prices and positions no longer have to withstand an unreasonable number of calls, and the tire is loaded three times less.
Even the graphical display of both schemes clearly demonstrates how the first option is more confusing, less obvious and prone to errors.
Excessive data exchange is easy to express in real money losses by assessing the cost of infrastructure that supports an additional load. Therefore, when implementing ESB, it is necessary to take into account the business logic of the information system.
Another pitfall that is easy to stumble over when implementing a service bus is the use of pay-per-request SaaS, which charge you per request instead of a month. It is necessary to calculate future costs based on the expected load. A well-designed to-be architecture scheme will help with this, which makes it possible to estimate the approximate number of calls to each system and the approximate number of information packets transferred. If their estimated value already exceeds the cost of virtual servers for the same unit of time, it is obvious that the latter should be chosen.
