There are a lot of materials on the Internet that talk about point-to-point integrations, brokers, and ESBs. As a rule, they focus either on the technical nuances of building integrations, or on the obvious pros and cons of each approach. In this article, KT.team experts suggest looking at all three types of integration from the point of view of the internal structure, which allows us to highlight not only the obvious but also the implicit advantages and disadvantages of each of them.
Is it more convenient to study the material in podcast format?
The basic structure of any integration
Regardless of the type of integration, any integration has:
- source system;
- the system is the recipient of information
- ETL layer (extract, transform, load, i.e. extract, transform, and load)
- data storage is a space where information transmitted through integrations is recorded.
This is a very top-level view that does not take into account the nuances of each particular integration and its logic. But it is precisely this fundamental structure that will help us understand the pros and cons of each type of integration. After all, integration properties depend on where (relative to systems) ETL functions and storage are located.
Point-to-point
The simplest and most understandable type. There is one system, there is a second system, and they exchange information. For example, through one integration with 1C, goods are uploaded to the site, and through the second integration from the site to 1C, new orders are uploaded. No middlemen, no more storage, or no overall context.
This means that in point-to-point integration, both the ETL function and the storage are within the systems themselves. The source system extracts information from its databases and converts it into a form that is convenient for consumption. The receiving system stores information in its internal repositories in order to process and use it later.
The advantages of point-to-point integration
Since integration does not involve any additional systems and bases, point-to-point integration is the fastest type of integration in terms of development and the easiest to manage.
Firstly, the boxed versions of systems usually already provide some API or connectors for connecting standard integrations.
Second, this type of integration does not require a manager or product owner to be involved in how the integration is built. The development teams for each system agree among themselves on the API scheme and the integration logic.
Thirdly, direct integrations ensure the rapid transfer of information between systems, precisely due to the absence of intermediaries.
Fourthly, the initial development of direct integration with simple logic looks like an inexpensive task.
And... In fact, the benefits end there.
Disadvantages of point-to-point integration
The first and biggest disadvantage of such integration is the strong interdependence of systems. Let's continue our example with 1C and an online store. For a user to place an order, the online store must receive information from 1C about the status in the loyalty program and the user's address. As soon as 1C “falls” or slows down due to a high load, critical information will stop flowing to the online store. Customers will not be able to place orders until 1C recovers.
To protect businesses from such risks, the development team will constantly invent some kind of fuses: setting flags, inventing queues, reinventing asynchronous sending messages between systems... As a result, “low-cost” direct integration will catch up with and surpass the cost of the “expensive” bus.
Minus number two: systems are starting to “know” too much about each other. In what format and at what speed should data be transferred, what data is redundant or insufficient? All this must be taken into account in the logic of integration.
The third disadvantage: integrations overload the source system, even if it initially works with a small amount of data. For example, CRM contains data about customers who are constantly accessed by an online store, delivery, marketing applications, etc. As a result, the load on CRM increases significantly due to constant calls from other systems.
The fourth disadvantage: a change in any system triggers a cascade of improvements in all related integrations and systems. If you add a new parameter to the product card, please change the logic in all systems that collect this card. And a change of logic is not always painless: in particularly critical cases, there is nothing else to do but simply roll back all the changes.
The fifth disadvantage is proxying. For example, information on balances should be obtained from the warehouse program by both 1C and the online store. In order not to overload the warehouse program and not transfer the same thing several times, developers can offer you an “elegant solution”: let WMS transfer everything to 1C only. And 1C, in turn, gives the online store not only its section of data on goods and orders, but also its balances.
Why is this approach bad? The relevance of the balance data that is displayed in the online store now depends on the performance of two systems at once. 1C (or any intermediary system) is overloaded with non-core data. In practice, we have also seen longer chains of data enrichment; this is not so rare. And if something goes wrong in the final system, we have to arrange an investigation: where the data transfer did not happen, for what reason, and how to avoid repetition.
The last, but in our opinion the most critical disadvantage is that point-to-point integrations block or significantly slow down IT system updates. If you want to replace the same CRM in three years, you'll have to re-develop and replicate the logic of dozens of integrations, not just those that are directly related to CRM.
At the same time, the company is becoming a hostage to developers who know a lot about the existing logic. It is almost impossible to get away from them or switch the team to other tasks. The developers themselves are also becoming hostages to their knowledge: they have to always be on their phones even on vacation, because you never know what could go wrong?
Broker integration
In such integration, in addition to the source system and the receiver system, an intermediate layer appears — a message broker. The systems don't interact with each other — only with Apache Kafka, RabbitMQ, or any similar service. At the same time, IT systems are still the initiators of sending and receiving information.
In this type of integration, the information store is moved beyond finite systems: if the information has already reached the broker, the receiving system will retrieve it from there at the agreed time. At the same time, ETL logic is still on the side of the integration systems. At different stages of data transfer, responsibility for retrieving, transforming and downloading is shared between the source system and the receiving system.
Benefits of integration through a broker
Data sharing becomes more controllable. You can see in the broker what messages were given, which ones were delivered, view logs, and see a retrospective of events.
Second: more source systems no need to find out if the receiver system is available at the time the message is sent. All that remains in their area of responsibility is the logic of forming their own message queue.
Brokers' advantages also include contract support. Contract — checking a message for some basic rules (the presence of a specific field in the required format).
And equally important in this approach is that development is becoming more standardized. Almost every service has tools for connecting a broker. This means that you don't need to develop from scratch, come up with crutches, and you don't have to invent the logic of integrations every time.
It is also worth noting that data transfer through a broker is fast, in this regard, the broker is almost as good as the fastest type of point-to-point integrations.
Disadvantages of integration through a broker
The first important drawback is that using a broker gives load on the service that sends the message. Let's imagine a situation: the broker has been unavailable for some time, during which time no new messages have been added to the queue. The broker himself “does not know” what and what system tried to send to him. Accordingly, the source system has to check which message was last in the queue and re-send everything that has accumulated during the inaccessibility.
This logic puts an additional burden on the system, so it's tempting to abandon it. But in this case, undelivered messages will sooner or later affect critical business processes.
The second minus is overload of the recipient system with unnecessary data. The broker can't choose which messages to send — he gives the whole queue. For example, all inventory information or all information from WMS on inventory balances.
Now let's imagine that your system doesn't need the full amount of data. For example, 1C, which operates at your branch in Ufa, does not need data on customers and orders from all your 20 branches. But the system cannot “ask” the broker for filtered information. Through integrations, it has to receive everything that CRM and OMS (Order Management System) give, save it to its databases and, using internal logic, separate 5% of the required data from 95% of the extra ones.
The third significant drawback is contract cementing. All participants in the process must agree on the format in which the message will be transmitted. This deprives us of flexibility and leads to an increase in labor costs to support a single format.
An unobvious disadvantage is related to lack of a convenient and/or detailed retrospective. For example, the Kafka broker may store historical data, but if something breaks, you will have to load the entire message queue, from the historical first entry to the last entry. It is impossible to make any selections on current statuses. The problem is most acute when you need to share a story with another team or migrate — getting this information out will be problematic.
This drawback leads to another significant drawback — the impossibility of high-quality verification. The systems load both old and new messages from the broker without making a difference between them.
When re-loading, there may be situations when a newer message is deleted: for example, instead of the “delivered” status, the order gets the “created” status. You can avoid such situations, but to do this, you will need to add an intermediate verification service or write the reconciliation logic in the recipient system.
This also affects speed. On the one hand, systems quickly exchange the entire flow of messages, but on the other hand, it is difficult to figure out what is relevant and eliminate duplicates. Therefore, reading data is much slower.
Using a broker can just as much lead to proxying as direct integrations. To prevent all systems in the circuit from receiving redundant data, it is “easier” to pass it through one system, normalize it there and transfer the already normalized data further. We described what this leads to above.
There is also an implicit one minus management procedures. Each system has its own product owner. But who owns the broker and related integrations? Most often, it becomes a technical department. For other departments, this is a clear plus (less responsibility), but in this case none of the departments concerned will have a sense of control over the infrastructure.
Integration via data bus (ESB layer)
Integration through the ESB layer at first glance looks more complicated than the previous types. It includes, firstly, systems — sources and consumers. Secondly, a separate layer of ETL microservices that collect or give data, as well as transform and normalize information. Thirdly, a storage circuit separated from both systems and ETL.
There are different ways to technically implement the tire approach. We will conceptually consider a more correct approach, when the bus controls the transfer/delivery of requests between services.
The advantages of using a tire
The basic structure of such integrations helps to understand the origins of the advantages of integrations through ESB. This is the only method of integration in which all three components are separated and perform their strict function: the system generates and consumes data, the ETL layer transfers it for its intended purpose and in the right form, and warehouses (which is logical) store data.
The result is, for example, reducing the load on services due to the lack of integration logic in them. The task of the service is to provide access to data via API (if it's a modern service), FTP, or any other exchange method available in the service. The ETL layer will take the data itself and transfer it to the repository. The service only “thinks” that all the information it is responsible for is current and complete.
The second plus follows from the first — the source system does not need to know about the technical requirements of the data consumer system. The data format, the transfer rate, the frequency of updates — all this logic is in the ETL layer.
The load on each system becomes controllable. In the ETL layer, you can adjust the system load and change settings: for example, do not load 1C during the day and update data only in the evening. Moreover, you can control the number of requests: how many packages to deliver to the service per unit of time.
The bus takes care of filtering information. In the previous section, we looked at an example from a 1C branch that has to collect 20 times more information than is necessary for work. This does not happen in integration via ESB. The bus sends the service only the data it really needs — just put this logic into the connector.
Proper integration via ESB requires resiliency of the entire IT architecture. Services do not exchange data directly: the ETL layer saves it to special repositories and, if necessary, takes the current version from there. For example, if your bonus management system is temporarily unavailable, this will not affect online ordering in any way. The online store will receive data on bonus points and personal discounts from storage.
When the bonus management system is online again, the connector will inform it about points deducted and new orders to recalculate personal discounts for regular customers.
In a global sense, the use of ESB provides flexibility and resistance to cascading changes. Imagine that you have decided to replace a foreign CRM with a Russian one. If you choose the point-to-point type of integrations, you will have to first re-develop point-to-point integrations between CRM and other systems, and secondly, make changes to the integrations of other systems that use CRM data. In the case of ESB, you only need to develop flows from CRM and to CRM. The flows of consumer systems will remain unchanged.
It might seem that using an intermediate layer reduces speed. But in practice the exchange rate is very high: Tens, hundreds, thousands of messages are delivered in 400-500 milliseconds!
A plus that will be more obvious to companies with multiple data streams: standardization of development. The ETL layer uses common logic to receive and send data. The development of new connectors, in fact, comes down to reproducing already developed logic, adjusting for a specific type of data or system features.
Disadvantages of using a tire
The main disadvantage is the complexity at the start. To do integrations right, you need to stop and think about IT architecture: identify the areas of responsibility of each system; understand what information it gets from and where it comes from, what it gives away, what is needed and what is redundant. It is important here not to complicate the overall approach and think strategically so that the chosen model shows its viability for many years to come.
Labor costs in preparing to launch a tire are also reflected in the external high cost of integrations. At the start, you will have to pay for business analysis, for thinking through the integration logic, and for implementing bus tools — ETL layer, DWH, etc. In our experience, this is approximately 90% of the cost of implementing and owning an ESB. The remaining 10% supports ready-made integrations.
A misunderstanding of the approach or lack of experience in working with ESB leads to the fact that the implementation officer may not properly build the integration — for example, by transferring it to an intermediate layer unnecessary business logic. This will affect both the speed of development at the moment and the cost of further improvements.
The worst option you can come up with without properly considering bus integrations is to turn a flexible and alienated architecture into a monolith. Instead of making it simple and transparent, the team writes a huge number of lines of code. Instead of weak connectivity and resiliency, the output is a hard-to-maintain architecture.
To prevent this from happening, we need a competent implementation team: the entry threshold here is higher than in point-to-point integrations.
