Global experience in implementing Talend

Otto Group is a German online store, Europe's largest mail order store with more than 2.8 million items and 6,800 clothing brands for the whole family. It processes about 7 million online orders annually and up to ten new orders per second on peak days.

The existing IT architecture of the online store consisted of a Teradata database and a large number of data islands. The complexity of the architecture was also that some data islands used SQL, while others used PL/SQL, Oracle, and Perl scripts. Thus, to determine which advertising offer would be most suitable for customers, the company had to transform huge amounts of data, and the information system needed a single integration tool.
In addition, the project had a lot of integration scripts that were very difficult to maintain, as the developers who originally wrote the code were no longer on staff.

Otto decided to create a new data lake from scratch and use Talend as a bridge between a new data lake, which stores data from an online store and partner websites, and a 250 TB Exasol database. Talend also made it possible to use its own CI/CD process in development. The use of Talend made it possible to continuously integrate data sources, develop and deliver new updates and applications, and create test environments for launching new software. All this helped to increase the speed of development and the speed of fixing problems on the project and, in general, helped reduce the risks associated with new development and integrations. For example, if the system receives a request from a client to delete their data from the Otto system, this no longer takes much time, since there is no need for lengthy searches through all databases. Talend made it possible to quickly exchange data with their sources, making it easier to delete data from Exasol and the Cloudera data lake.

Bayer AG is a German chemical and pharmaceutical multinational corporation with more than 150 years of history and key competence in the field of healthcare in the world.

Bayer is constantly researching and testing new drugs to determine their efficacy and safety for further market launch. This process requires the accumulation, analysis and storage of a huge amount of clinical data from patients and volunteers. Data is also collected from laboratories and electronic devices. All information is automatically anonymized at the point of collection.
The old information system was outdated and could not cope with the load and the constantly growing data flow. Moreover, some processes were not automated, operations were processed manually, which often led to errors caused by human factors.

Instead of supporting the old system, Bayer decided to completely change the environment and infrastructure architecture. Bayer held a hackathon, during which it thoroughly tested Talend and came to the conclusion that it was best suited for continuous improvement of the system, given its principles of flexibility and the ability to support a microservice architecture.
The Talend ESB bus has become a middleware that connects repositories while being hidden from users' eyes. The data enters Oracle's local data lake, connects to the SAS system for advanced analytics and uses Oracle Life Sciences Data Hub to store data on clinical trials of drugs and drugs.
The Talend ESB service bus made it possible to quickly and in accordance with requirements add new functions, make the received data better and increase the speed of its processing. Another advantage of Talend for the company is the ability to track everything that happens to data, including where it comes from and where it is sent.