EDI as a basic system for electronic data exchange and integrations with partners

Development customers often think that EDI is a modern and universal standard for any company for data exchange and integration with any counterparties. Is this true? Let's take a look at what EDI is and why this technology has been so popular for over 40 years.

1. What is EDI. The history of origin and development. Standards

EDI (abbreviated from English electronic data interchange, Russian. “electronic data interchange”) is an automated data exchange technology that includes the transmission, flow of messages, document format and software used to interpret documents.

EDI originated in the 1970s and 1980s and is still popular today. Like many other early information technologies, EDI exchange was invented by the military and was originally used in logistics.

In the late 1940s, air traffic in Europe was accompanied by the transmission of large amounts of information about cargo, but the process of exchanging data was inconvenient, and there was an urgent need to develop new concepts and methods of exchange. One of the first integrated systems to use EDI was the London Cargo EDP Scheme (LACES) at Heathrow Airport (London, UK) in 1971. The introduction of the Direct Trader Data Entry (DTI) method has allowed freight forwarders to enter information directly into the customs clearance system. After that, the registration process became faster and easier.

The growth in maritime traffic and customs problems similar to those at Heathrow Airport led to the introduction of DTI systems in some ports and groups of ports in the 1980s.

The electronic transfer of data between companies, as well as between business and government in a single, common format, attracted more and more attention, and in 1987, the UN/EDIFACT standard (United Nations standard for electronic data interchange in administration, commerce and transport) was approved. The UN has taken on this important mission and still publishes updated directories every six months, investing considerable funds in this work. The directories contain standard messages for various industries; now there are more than two hundred messages.

The Technical Committee for Standardization No. 154 of the International Organization for Standardization (ISO, abbreviated from English) is also taking part in the development of the handbook. International Organization for Standardization).

The use of common EDI standards is convenient for business and beneficial to regulatory authorities: customs, tax services, etc. This makes messaging transparent and easy to manage. The UN even recommends using the UN/EDIFACT standard when exchanging messages within government agencies and between national governments.

Thus, the UN/EDIFACT standard can be used for integrations with foreign partners, and its subsets are common for exchanging data with companies within Russia, for example, SWIFT in banking and EANCOM in trade. There are also a number of other subsets for different industries.

In Russian e-commerce, the CommerceML format is common in the form of XML from 1C, as well as YML (abbreviated from English. Yandex Market Language) in XML form. But if you want integration with global corporations, this standard won't be familiar to them.

In North America, the American National Standards Institute (ANSI, abbreviated from English) standard is common. American National Standards Institute) — ASC X12 (also known as X12).

GS1 EDI sets standards in global supply chains. TRADACOMS is a standard common in UK retail. ODETTE, VDA are used in the automotive industry, and HL7 are used in healthcare.

As you can see, EDI can't be understood as a single universal standard — it's a multitude of standards for different regions and industries.

2. EDI exchange features

The concept of EDI is often confused with the concept of exchanging legally significant documents. The following areas can be distinguished on the Russian B2B electronic document management systems market:

• EDI as a technology for integration with partners' information systems;
  

• electronic document management (abbreviated. EDI) as a means of exchanging legally significant documents between counterparties;
  

• electronic reporting to the tax, customs services and other state regulatory authorities.
  

EDI or exchange of legally relevant documents?

The development customer needs to decide whether he needs the exchange of legally significant documents, reporting, or IS integration? These are different things.

Naturally, businesses have many legally significant documents that need to be exchanged with buyers, suppliers and the tax authorities (at least). The peculiarity of EDI in Russia is its variety of forms: there are several electronic reporting systems on the market (SKB Kontur, VLSI); the UPD format (short for “universal transfer document”) is advisory in nature; there are several digital signature operators (short for “single digital signature”), etc.

But the exchange is easier when all parties use the same format and rules. To make such an integration, electronic document management is needed. In this case, EDI means a set of standards (Legacy) related to electronic document management.

EDI as a means of reporting and exchanging legally significant documents is a broad topic, but in this article we'll talk about EDI as a technology for integrations: this is the part of the tasks we are dealing with in our work.

Which companies is EDI suitable for

Small organizations do not need such standards. However, on a global scale, some top 1 is not a very large organization that can easily develop its own internal standards and use more modern data transfer technology, such as API, instead of EDI.

But if we are talking about entire industries, the exchange of information between countries, and major projects in which thousands of developers are working on data exchange, then there can be only two ways:

• or use existing standards (UN/EDIFACT and its subsets);

• or develop your own EDI standards that will be supported by counterparties.
  

A simple XML schema won't do.

Developing our own EDI standards will require a lot of effort by analysts (and if we are talking about interstate cooperation, then politicians) on both sides. But the efforts spent on technical implementation will be insignificant in comparison with the approvals.

3. Expert opinion: how does an EDI integration customer evaluate this technology

To find out what big businesses think about EDI, we interviewed a representative of FM Logistic, an international logistics company. So, our interviewee is Jonathan Woloszczyk, Head of IS Build & Deploy, our longtime customer and partner.

— In our collaboration, we used the API to integrate with delivery services, and EDI as a means of integrating with WMS. We also know that FM Logistic has an EDI exchange with some partners. Why do you think the transport and logistics business and its contractors are still interested in EDI? What are its advantages?

— This has been the case historically. First, all major logistics systems and WMS were created 20—30 years ago, when only EDI existed. These WMS are now not just ancient, they're mega-ancient, and it's often impossible to create an API for them.

Secondly, logistics still have a very high level of trust in EDI file sharing. Logisticians are pragmatic people; they are more used to presenting a file as a physical object, an artifact. With EDI, a logistician can say “I sent you a file” and “I got your file”. Responsibility is transparent: each exchange participant can always prove what, when and to whom they sent them.

The API and transactional exchange are much harder to understand. It is difficult to conceptualize and trust the format for transferring data through the API. I think this is the main reason why EDI exchange is so popular. But many logistics companies are already making progress: the younger generation is coming with different views on EDI and API.

Third, writing a script and connecting it to the API is much more difficult than using EDI. Most IT solutions have a standard EDI specification at the input or output, and this is quite easy to use: when you want to connect one ERP client system to WMS or another IS, you just need to do mapping — you don't need to write scripts, methods, connect them to the API, etc.

— Now let's look at the Auchan example. They have EDI Orders and your WMS has Orders. Did you integrate through EDI? How do you get order data from Auchan?

— When we first started working, we integrated through EDI. The integration took 2—4 business days per stream, which is fast.

— Do you have the same fields in Orders?

— No, not only fields, but also the entire structure can be completely different, and the object itself, called Orders, may differ from the object called Orders here. We have to analyze the specifications and reformat their Orders to match ours. It's good that even someone who is far from programming can do such an analysis — not a techie, but any analyst who only needs to make a list in Excel and show which fields should be equalized. Then another person, the developer, implements it. The API is a bit more complicated: you need at least a basic understanding of what a method is and how an application can respond to its call.

— Broadly speaking, you could simply share the EDI interface with your e-commerce partners and thus integrate with them. So why did you choose an API for this?

— The e-commerce market is quite young, and our company is also young and modern. 50% of our customers have their own website that does not communicate via EDI, but through which they receive orders and send them to us as logisticians. The remaining 50% of customers still share XML files via EDI. It's not UN/EDIFACT, ANSI X12 or anything like that, but it's also EDI nonetheless. They send you files, you convert them and integrate them.

E-Commerce is very transactional, and many small orders, often piecemeal, go through us every day. Data exchange is fast and intensive, and there are a lot of documents. While in the B2B logistics business, we deal with large quantities of goods, and all data for the day can be reflected in one document.

A big advantage of the API is that after every data submission, you receive an “OK” or “not OK” response. In the case of EDI, you don't get feedback: you just send the file and wait, not knowing what happened to it next. Some time ago, EDI tried to flag files whether the sender needed a response, but failed. EDI has not yet confirmed that integration is correct, and although there are workarounds to achieve this, they are very rarely implemented. Even if a client receives a file, he may be “stuck” in the business process immediately after receiving it.

EDI was invented back in the 1970s to move away from paperwork: partners used to exchange real documents with orders printed in them, but now they share files using the same paradigm. The API works in a different way: we send a request and receive data. HTTP methods: GET, HEAD, POST, PUT, DELETE, etc. — they are also two-way, so I can send data to my partner to change something and get an answer that the changes have been made. Or I can send a request aimed at getting an answer to me with a guide. I'm referring to a method that returns part of the catalog to me and displays it in the UI and mobile app.

— So, EDI is so popular because there are still many offices that still exchange pieces of paper, and broadcasting these papers into electronic messages and exchanging them is now relevant for them. That's why EDI sells well, even though it's an outdated technology and an old format.

— Yes, but for these people who require EDI and don't want an API, this is the first step in digital transformation. It is psychologically important for them to go through this stage in order to later evaluate the benefits of the API.

Switching from paper to API right away is too radical a change of concept. “No, the API is not relevant to what I do. I'd rather take my paper and send it as a file via EDI.”

It is very easy to decide to replace paper with a file that contains the same data as paper. It looks pretty much the same. AND API? “No, I won't understand this technology of yours, wrap me up EDI, it's easier that way.”

— Why don't WMS developers want to implement the API into the system itself, because the speed of data exchange is important to them and, for example, ERP (in particular, SAP) are already quite actively moving towards the API?

— In fact, WMS is already looking towards the API, but it is very difficult for them to take specific steps, because all popular WMS are associated with EDI by a large number of algorithms. To attach an API to WMS, it takes a lot of work to conceptualize and remodel objects, functions, etc. At the moment, there are very few WMS on the market that can be managed by API, and those that do have a very limited catalog of actions. And for the most part, they just broadcast a single API, that is, when you make a request, they run it through an EDI import script and give the response and result synchronously. But creating a new method for them that will directly affect databases or share data into the core is an unrealistically time-consuming job.

Thanks to our partner for the interview: we got acquainted with the opinion of a representative of a large international company about EDI and the prospects for using this technology in the logistics business.

findings

The main task that EDI faces in the modern world is to replace paper-based document management with electronic document management and automate data exchange. And EDI is doing this successfully.

Let's take as an example a process that involves a bank and the tax service: payment of a payment order. The tax office creates a WSDL file that contains information about the recipient who is able to receive payment notifications, with a specific XML schema.

Once a banking IP receives this WSDL, it will not be able to automatically understand which particular service should send a notification. The bank and the tax office must agree in advance on the name of the services. And this cannot happen automatically without approval.

Another problem would arise if a foreign bank, for example, tries to send a payment to our tax notice, and the Russian Tax Service makes the “budget payment classification code” (FCA) element mandatory in the XML scheme. Or he will describe the payer's data in the form of three mandatory elements: last name, first name and patronymic. And in the information systems of a foreign bank, there will be neither the CSC nor the middle name anywhere.

All the necessary elements need to be negotiated and discussed, and only after that developers will be able to start sawing their WSDL, XSD, etc. It usually takes a lot of time and money to clarify all disagreements and create common requirements. And in what syntax the solution will be implemented: EDI, JSON, XML, or something else is a secondary question.

The value of EDI lies precisely in the fact that contractors (both business and government) can agree once that messages should be written in a certain way, certain data must be transferred to them in order to generate, correctly transmit and receive such data automatically in the future, without human intervention.