Weak connectivity→ — the basis of a flexible IT infrastructure. Moreover, the mere fact that there are integrations in a circuit is not an indicator of a monolith or its absence. The main thing is how these integrations have been implemented. One method can lead to a pile of connections that are difficult to maintain and, as a result, difficult business scaling. The other is to a contour that can be quickly changed and modified on the go. In this article, we'll explain how integration approaches affect connectivity and what method to choose to keep the company's outline loosely connected.
We have previously discussed approaches to integrations into one of our articles→, today we will look at this topic precisely from the standpoint of building a loosely connected architecture and business flexibility.
What does a loosely connected architecture with effective integrations look like?
Before choosing an approach or tool, you should set a benchmark and see how things should ideally work. Below are five main criteria for the ideal state of integrations:
- any integration/service in the circuit can be easily and quickly improved, and this will not affect the operation of the circuit in any way;
- this integration can withstand peak loads and works without delays;
- a new data stream can be launched without improving services;
- an error in one service does not cause problems in others;
- monitoring is easy and can be delegated to technical support (instead of the development team).
The effectiveness of integration depends on the connections between components. Let's consider the key components.
The main components of any integration
Regardless of the type of integration, any of them has:
- source system;
- the system is the recipient of information
- ETL layer (extract, transform, load — extract, transform and load)
- data storage is the space in which the transmitted information is recorded.
This list does not reflect the nuances and logic of each particular integration. But it will definitely make it easier to understand the pros and cons of each approach, since the properties of a particular integration depend on where (relative to systems) ETL functions and storage are located.
Point-to-point
The simplest and most straightforward type of integration. And he is the one most often characterizes monolithic systems.
Imagine: there is one service, there is a second service, and they share information through direct integrations with each other. There may be several such integrations in both directions. For example, after one integration with 1C goods are uploaded to the site, and through the second one from the 1C site, new orders are uploaded to the site from 1C, updated order statuses arrive to the site, after the fourth... You can continue indefinitely.
ETL and storage in this integration are within the systems themselves. The source system extracts information from its databases and converts it into a form that is convenient for consumption. The receiving system stores information for further processing. No middlemen, no more storage, or no overall context.
The advantages of the point-to-point approach
- The fastest type of integration in terms of development: boxed APIs and minor improvements are enough. Teams developing various services agree on their logic of use.
- This integration is set up quickly, especially with a small number of services within the circuit.
- Creating a direct integration with simple logic is a low-cost task.
Disadvantages of the point-to-point approach
There are more disadvantages to this integration.
- Strong interdependence of systems. For example, in order for a user to place an order, an online store must receive information from 1C about the status in the loyalty program and the user's address. As soon as 1C “falls” or slows down due to a high load, critical information will stop flowing to the online store. Customers will not be able to place orders until 1C recovers.
To protect businesses from such risks, the development team will constantly invent fuses: setting flags, inventing queues, and reinventing sending messages between systems asynchronously. But then the relatively low cost of such integration can be removed from the list of advantages at some point.
- One service “knows” the context of another service. In what format and at what speed should data be transferred, what data is redundant or insufficient? When there is no general context, services have to “think” for themselves.
- Integrations overload the source system. This slows down the system's response speed and leads to data errors. In the case of a large number of integrations, this approach makes the IT circuit opaque and cumbersome. One of our clients came to us with a similar problem→ — a company with more than 200 retail outlets, each of which was integrated with the head office system on a point-to-point basis.
- A change in any system triggers a cascade of improvements in all related integrations and systems. Moreover, they can be minimal: adding even one new parameter to the product card will change the logic in all systems that collect this card.
- This approach to integration makes it difficult for companies to scale. This point is a consequence of all the problems above. A case in point is our client case→, teams, an application for taxi fleets and fleets that automates the receipt of waybills. The direct connection worked adequately until only three fleets were connected to the system, with one database for each. But when their number grew, the system's resources were clearly insufficient.
What about connectivity?
Ease and speed of refining integrations/services:
- Doesn't match. A change in one system requires changes in related systems, which complicates and slows down improvements.
Maintaining peak loads and working without delays:
- Partly compliant. Initially, integrations are fast, but as the number of requests increases, the source system may become overloaded, leading to delays.
Launching a new data stream without improving services:
- Doesn't match. The launch of new data streams requires changes in each of the integrated systems.
No problems in other services if there is an error in one:
- Doesn't match. An error in one system can cause problems in others because they are closely interrelated.
Easy monitoring and the ability to delegate technical support:
- Doesn't match. Monitoring is difficult due to the high interconnectedness of the systems, which requires the involvement of the development team.
The result: 0.5 points of weak connectivity out of 5.
If highly connected monoliths are not your goal, point-to-point integrations are best avoided.
Broker integration
In such integration, in addition to the source system and the receiver system, an intermediate layer appears — a message broker. The systems don't interact with each other — only with Apache Kafka, RabbitMQ, or any similar service. At the same time, IT systems are still the initiators of sending and receiving information.
In this type of integration, the information store is moved beyond finite systems: if the information has already reached the broker, the receiving system will retrieve it from there at the agreed time. ETL logic still remains on the side of the integration systems: the responsibility for extraction, transformation, and loading is shared between the source system and the receiving system.
Benefits of integration through a broker
- Data sharing is becoming more controlled. In the broker, you can see what messages were given, which ones were delivered, view logs, and see a retrospective of events.
- Source systems don't need to find out if the receiver system is available at the time the message is sent. Their area of responsibility is only the logic of forming their own message queue.
- Message brokers usually support contracts, that is, they check messages for basic rules (for example, the presence of a specific field in the required format).
- This integration makes development more standardized. Almost every service has tools for connecting a broker. This means that you don't need to develop from scratch; you don't have to invent crutches and invent the logic of integrations every time.
- Like point-to-point, the broker supports high data transfer speeds between services.
Disadvantages of integration through a broker
- The broker overloads the source system with unnecessary actions. If the broker was unavailable for a while, no new messages were added to the queue during this period. The broker himself “does not know” what and what system tried to send to him. Accordingly, the source system has to check which message was last in the queue and re-send everything that has accumulated during the inaccessibility.
- The broker overloads the recipient system with unnecessary data. The broker can't choose which messages to send; he gives the whole queue. For example, all inventory information or data from WMS on inventory balances, even if most of it is not needed by the receiving system. But the latter still keeps their databases and, due to internal logic, separates 5% of the required data from 95% of the extra ones.
- Using a broker results in ”cementing” the contract. All participants in the process must agree on the format in which the message will be transmitted. This deprives us of flexibility and leads to an increase in labor costs to support a single format.
- Another disadvantage is the lack of a convenient and/or detailed retrospective. For example, the Kafka broker may store historical data, but if something breaks, you will have to load the entire message queue, from the historical first entry to the last entry. It is impossible to make any selections on current statuses. The problem is most acute when you need to share a story with another team or migrate.
- Slower data reading. On the one hand, systems quickly exchange the entire flow of messages, but on the other hand, it is difficult to figure out what is relevant and eliminate duplicates.
- There is also an unobvious one managerial disadvantage. Each system has its own product owner, and the broker is usually owned by a technical department. For other departments, this is a plus (less responsibility), but in the end, none of the departments concerned have a sense of control over the infrastructure.
What about connectivity?
Ease and speed of refining integrations/services:
- Partly compliant. Data sharing is standardized, but the need to maintain a common data format can limit flexibility.
Maintaining peak loads and working without delays:
- Partly compliant. The integration supports high data transfer rates, but the source system may be overloaded due to redundant actions and data.
Launching a new data stream without improving services:
- Doesn't match. The new data stream requires verification and possible improvements to comply with the broker's contracts.
No problems in other services if there is an error in one:
- Partly compliant. The source system does not depend on the availability of the receiving system, but broker overload can cause delays.
Easy monitoring and the ability to delegate technical support:
- Partly compliant. The broker allows you to monitor data and logs, but the complexity of retrospecting and detailing events can make it difficult to fully delegate to technical support.
Total: 2 points of weak connectivity out of 5.
Bus integration (ESB)
ESB is a practical tool for implementing integrations in a loosely connected architecture.
Integration via the ESB layer includes:
- Systems are sources and consumers.
- A separate layer of ETL microservices that collect and give data, transform and normalize information. The data format, the transfer rate, the frequency of updates — all this logic is here.
- A separate storage circuit. It can be implemented in different ways: some buses support open source solutions like PostgreSQL or MySQL, while others use internal storage.
All three components are separated and perform their function: the system generates and consumes data, the ETL layer transfers it for its intended purpose and in the right form, and warehouses (which is logical) store data.
The advantages of using a tire
- The load on each system becomes controllable. The purpose of the service is to provide access to data via API (if it's a modern service), FTP, or any other available method. The ETL layer will take the data itself and transfer it to the repository. The service only “thinks” that all the information it is responsible for is current and complete at the time of transmission.
- You can quickly connect any systems to the circuit or modify them in any order. This aspect is the main condition for effective business scalability. Here is an example of how we solved the scalability problem by introducing an ESB layer in an electrical equipment manufacturer and integrated energy-efficient solutions.
- The bus takes care of filtering information. ESB sends the service only the data it really needs — just put this logic into the connector.
- Integrations are unified, and service logic is becoming simpler. The ETL layer uses common logic to receive and send data. The development of new connectors is reduced to reproducing already developed logic, adjusting for a specific type of data or system features.
- Development and monitoring can be delegated. This plus follows from the previous one. Development is simplified, and monitoring responsibilities can be removed from developers and delegated to technical support.
- Proper integration via ESB requires resiliency of the entire IT architecture. Services do not exchange data directly: the ETL layer saves it to special repositories and, if necessary, takes the current version from there.
Disadvantages of using a tire
Globally, there are two:
- Labor intensity at the start. For proper integrations, it is necessary to analyze the IT architecture: identify areas of responsibility and identify necessary and redundant information. On average, atthe weight of the ESB layer takes 15-20 months before the MVP launch.
- The starting cost is quite high. At the start, you will have to pay for business analysis, for thinking through the integration logic, and for implementing bus tools — ETL layer, DWH, etc. Here we described in detail the costs of implementing the ESB layer at each stage.
What about connectivity?
Ease and speed of refining integrations/services:
- Complies. Services can be easily developed or new ones added by separating the ETL layer and the data warehouse.
Maintaining peak loads and working without delays:
- Complies. ESB allows you to monitor the load on each system, ensuring stable operation during peak loads.
Launching a new data stream without improving services:
- Complies. A new data stream can be launched without improving the services, since the data transfer logic is managed in the ETL layer.
No problems in other services if there is an error in one:
- Complies. An error in one service does not affect others due to data isolation in the ETL layer and storage.
Easy monitoring and the ability to delegate technical support:
- Complies. Monitoring integrations can be delegated to technical support, as ESB provides the necessary monitoring and management tools.
Total: 5 points of weak connectivity out of 5.
Important: choosing the right integration tool is not a magic pill against a monolith
It is often believed that the introduction of a broker or ESB will solve all the problems of the IT circuit. But that's not true. Impossible build the right architecture without expanding areas of responsibility and without getting rid of connectivity.
Even after the bus is implemented, the architecture may remain monolithic if the service boundaries are incorrectly divided or the logic of ETL connectors is overcomplicated. As a result, the team generates code endlessly instead of making integrations simple and clear. Instead of weak connectivity and resiliency, the output is a hard-to-maintain architecture.
A misunderstanding of the approach or lack of experience in working with ESB can lead to the fact that the implementer may not build the integration correctly — for example, by transferring it to an intermediate layer unnecessary business logic. This will affect both the speed of development at the moment and the cost of further improvements.
To prevent this from happening, we need a competent implementation team: the entry threshold here is higher than in point-to-point integrations.
