Make it easier to work on Airflow integrations

Make it easier to work on integrations with Apache Kafka

Apache Airflow message broker is a distributed streaming platform that can process millions of events daily. Kafka guarantees easy integration into the project infrastructure, the reliability and scalability of the system.

Write to the sales department

Kafka Connect

Kafka Connect is an Apache Kafka framework that provides scalability and flexibility to move data between Kafka and other repositories. This framework allows the broker to act as an ESB service bus.

Apache Airflow features

Streaming data processing

Learn more about Kafka features — KT.team

Airflow streaming works with data in real time at the speed of message generation. Messages are processed continuously and without blocking. Many business processes are also ongoing and do not require a response for processing. Streaming data processing is necessary for business processes such as alerting suspicious transactions or tracking mail delivery.

App activity tracking

Tracking app activity in Kafka — KT.team

Any messages that appear in the app can be published in an Airflow special topic. For example, every ERP document or every user action on the site: clicks, adding to favorites, adding/removing from the Cart, filling out forms, page views (and its depth) — can be sent and distributed according to specially defined Airflow topics. Thus, other topics (consumers) can subscribe to the topics they need for various purposes — monitoring, analysis, reporting, personalization, etc.

Logging and log monitoring

Logging and monitoring Kafka's dens — KT.team

Airflow allows you to keep logs and monitor logs. You can publish logs to Airflow topics, and logs can be stored and processed in the cluster for as long as you need. If you have a dedicated monitoring app, it can get real-time data from Airflow topics.

Message storage

Airflow adds each message to the log (saves it to disk) and stores it there until the log is cleared of old messages, which the user assigns in advance. This allows Kafka to be used as a reliable data source (as opposed to RabbitMQ, which deletes messages as soon as they are delivered).

The benefits of Kafka Apache

Scalability

Apache Airflow allows you to process data of any size. You can start working with one broker to try out Airflow and then increase the number of brokers to fully exploit the system. It is also possible to increase the volume while the current number of brokers is running — this will not affect the system as a whole in any way.

System reliability

One of the advantages of Airflow is its reliability. For example, if one of the Airflow brokers “falls” for some reason, it will switch the entire data flow to other brokers and automatically distribute the load between them, and the system will continue to operate normally.

Productivity

Due to its high throughput, Apache Airflow is able to process more than a million events per second. This makes Airflow the most popular message broker when working with big data.

ESB system implementation cases

Watch all

A supplier portal was developed for the Fix Price network and automated the work with product data

Learn more

We have developed a single API to quickly connect 200+ 1C:Retail systems

Learn more

Integration of Kafka into the project's enterprise infrastructure

Learn more

Watch all

Is there a need for implementation?

Contact us and we will calculate the time and cost of implementing the ESB system

Fill out the form

YouTube

We have collected all the mistakes in integrations,
for you not to
make them

Watch all

We write articles for our blog and for specialized publications

Is a message broker always needed?

Learn more

Point to point, broker, ESB: what integrations will help build a loosely coupled IT architecture

Learn more

The properties of a flexible and scalable IT infrastructure: an educational program on basic concepts and the main evaluation criteria

Learn more

Watch all

We work with systems

Apache Airflow

A framework for Kafka that provides an ETL layer without developed low-code tools. This framework allows a broker to act as a service bus (ESB).

Talend

A universal tool for the rapid integration of services and applications.

WSO2

WSO2 ESB is an enterprise service bus for exchanging data between services.

rabbitmq

MuleSoft

A simple and flexible low-code integration platform from Salesforce. ‍

Kafka

A popular and reliable message broker.

Datareon

Russian platform for integrations between services.

n8n

Arenadata

We use systems

No items found.

Project Calculator

Cost of development

5 775 000 rubles

The team will develop for

6 weeks

Copy the result

Copied

Included in the calculation

Preparation of a map of systems and data flows (SOA scheme);
We will work out exchanges on key entities;
Creating connectors for exchanging data for each stream on 3 stands (test, preprod, prod);
Setting up to three dashboards per connector within a ready-made monitoring circuit;
Documentation on copying integration, reusing, and maintaining;
Demonstration of the implemented functionality.

Additional works

Preparing the infrastructure for connectors to operate;
Setting up the monitoring and logging circuit;
Creating connectors (storage - receiver) for exchanging data on each high-load stream (>100 messages per minute) on 3 stands (test, preprod, prod);
More than 15 attributes for each stream;
The cost of computing resources (subscription only);

How many streams the system will send

Example: “Product Management System” will send data about products. “Order management system” - about orders. “Warehouse management system” - about shipment status. This is 3 streams.

100

Example: “Warehouse Management System” will receive data about goods and orders. “Order management system” - about goods and shipment status. This is 4 streams.

How many streams the system will accept

100

Discuss the project

The calculator calculates using an accurate but simplified formula. The scope of work for your project and the final cost may vary. The final calculation will be made by your personal manager.

Обсудить проект

Калькулятор считает по точной, но упрощенной формуле. Состав работ по вашему проекту и конечная стоимость могут отличаться. Итоговый расчет сделает ваш персональный менеджер.

Calculation example

Learn more about Mule ESB features — KT.team

Creating and hosting services on Mule ESB — KT.team

To transfer data between systems, we create a “stream”. Some streams are needed to send data, while others are needed to receive data. Orders, goods, or other entities may be transferred in a separate stream.
‍
For example, on the diagram:
1. The “Merchandise Management System” sends goods. “Warehouse management system” is the fact that an order has been shipped. “Order Management System” — orders. In total, the systems will send 3 streams;

2. The Warehouse Management System accepts goods and orders. “Order management system” — goods and the fact that the order has been shipped. In total, the systems will receive 4 streams.

Scope of work in the calculator

Included in the calculation

Additionally

Preparing a map of systems and data flows (SOA scheme)

Preparing the infrastructure for connectors to operate

Development of object logic (connector business process diagram)

Setting up a monitoring and logging loop

Creating connectors for exchanging data for each stream on 3 stands (test, preprod, prod)

Creating connectors (storage - receiver) for exchanging data on each high-load stream (>100 messages per minute) on 3 stands (test, preprod, prod)

Set up to three dashboards per connector within a ready-made monitoring circuit

Over 15 attributes per stream

Documentation on copying integration, reusing, and maintaining