Introduction to the OpenESB monitoring framework

You can use OpenESB monitoring framework to extract, at the runtime, technical and business information from the OpenESB BPEL and publish them to create reports, generate alerts or event to monitor your processes and your business.

What is OpenESB monitoring framework?

At a higher level, an organization which wants to monitor its processes needs to extract two types of information

The first type is the technical information on how the processes technically run. It is used to evaluate the service availability, response time, quality of service, SLA. It could also be used to trace the activities at the runtime and understand “How the processes run”.

The second type gives information on the business itself. It is a kind of snapshots of the process whose content is defined at the design time and provides information for business analytics processes. It replies to the question “What my process ‘processes’” 

The OpenESB monitoring framework replies to these questions and provides two types of messages named “BPEL event” and “Business Message”.

Why an OpenESB monitoring framework?

Currently, OpenESB standalone edition (v3.1) inherited a monitoring system from the OpenESB legacy editions. This monitoring relies on a complex mapping process that stores the BPEL Java object in a database. Unfortunately, this process requires so many resources that very few users could use it. In a current configuration, when the legacy monitoring is on, BPEL response times can be increased by up to 10 times. Moreover, BPEL events and business messages can only be stored in a local relational database. It excludes the new persistence systems such as the NoSQL databases or the cloud data lakes.

The difficulty to get metrics from the running processes cuts the links between the business teams and the IT process implementations and due to the lack of feedback, stops the business processes improvement and often generates a loss of income to the companies.

The OpenESB monitoring framework has been designed and developed to solve these issues. It is a light and scalable framework that does not deeply affect the BPEL performances. Events and business messages sending is not limited to the relational databases but targets the latest big data tools such as Kafka, Geode and offer and access to the data lakes on the could.

Pymma relied on the following picture to design, develop and implement the OpenESB monitoring framework.

Our main idea was to create a virtual circle with the following steps:

• Business team: Defines the business processes and the metrics that will be used to evaluate the conformance of the process behaviour with the company’s requirements.
• IT team: Implements the business processes and the messages that will be used to generate the metrics.
• Business process: Generates events and business messages
• Analytic process: Collects and improves events and business messages
• Report and dashboard: Provides metrics for the business team

OpenESB monitoring framework installation

To install the OpenESB monitoring framework, you must deploy on OpenESB, the last version of the BPEL engine developed by Pymma and the publishers you need to use. Pymma’s BPEL is backward compatible with the OpenESB Standalone Edition. You don’t need to redesign the BPEL processes already deployed on OpenESB to access to the BPEL Events.

Business message definition is concurrent with the process implementation and does not impact it. To publish business messages, you need to add message mappings to the BPEL projects and then redeploy them. No other component is required to run the OpenESB monitoring framework.

BPEL Events

Three types of Events are defined, the BPEL event, the Activity event, and the Variable event.

Events contains raw information such as

• What is the event?
• When it happens
• Where it occurs

It is up to the analytic process to use this raw data and extract useful information for the users.

Using BPEL event does not impact the business process design, and no additional development is required.

Business messages

A business message is a makeup of variable embedded in the process context. At the design time, The OpenESB user defines concurrently to the BPEL development, the element they want to extract from the BPEL context and compose a new structure often defined by the business teams.

In a process, many business messages can be implemented and published concurrently. Generating business message does not impact the process development.

Event publisher

Once the event is issued, it is sent to an external analytic process not embedded in OpenESB.

The OpenESB architecture based on multi-instances deployment proved its capability to daily process billions of messages on production. Since OpenESB resources are dedicated to run and manage the business process, the external processes that will manage events and business messages must be delegated to an external system that releases OpenESB from this overload.

The event publishers are multithreaded and optimized to provide the best performances with few resources. Regarding the protocol used, they publish from hundreds to thousands of messages per second.

Business message publisher

The business message publisher has mainly the same features than the event publisher. Nevertheless, its design is often a bit more complex than the event publisher since it deals with messages without a predefined structure.

The business message publishers are multithreaded and optimized to provide the best performances with few resources. Regarding the protocol used, they publish from hundreds to thousands of messages per second.

External Process

The external process must not with the same resources the OpenESB. It could be designed, developed and deployed by the users or by Pymma on-demand. It could be any software such as alerts management, event processing, data lake, statistic or analytic tools, etc.

Consequently, the publisher acts as an “external process” client. Also, It transforms and adapts the event and the business messages to match the message format expected by the external system. It must not be used to execute a part of the external process and then affects BPEL performances. 

External process features

The external process must fulfil some critical requirements. It must be an asynchronous system with a high capability of ingestion since OpenESB monitoring framework relies on a “fire and forget” policy and does not wait for any acknowledgment from the External process to release its resources.

Some configurations working with multiple OpenESB instances generate up to billions of messages on a daily basis. It is up to the user to calibrate the capability of ingestion of the external process and keep safe and secure the messages published by OpenESB.

OpenESB monitoring framework scalability

The OpenESB monitoring framework has a powerful horizontal scalability. So, it matches the new architecture based on virtualization and cloud. Messages generated by the framework are stateless and atomic and allow many OpenESB instances to push messages to the same external system.

For better performances, the external module must scale horizontally as well.

Publishers implementation

Event publisher

The list of event publishers is not exhaustive.

• File event publisher
• MongoDB event publisher
• Elasticsearch event publisher
• Apache Geode event
• Apache Kafka event publisher
• RDB event publisher
• Google Big Query event publisher

On-demand, Pymma develops event publisher for its customer

Business message publisher

This list of business message publishers is not exhaustive.

• File business message publisher
• MongoDB business message publisher
• Elasticsearch business message publisher
• RDB business message publisher
• Google Big Query business message publisher

On-demand, Pymma develops event publisher for its customer

External process implementations

It is up to the OpenESB user to design and implement an external process that fulfills its requirements. As an example, we detail three cases that could be chosen as external process design implementation.

External process implementation 01

This implementation is dedicated to the configuration with a limited number of messages. 

OpenESB publishers put the events, and the business messages in a database which acts as a buffer and a container of data and the BI tools query the database to generate reports and dashboards.

If the configuration is simple and easy to deploy, but the OpenESB user must consider that the events and the business messages are recorded in a raw format and require a complex query to provide the right data for reports and dashboards. Database ingestion capability could be a point of contention since relational databases demonstrate low performance in high rate ingestion.

External process implementation 02

Cloud application performances can be used to generate reports and dashboard.

OpenESB publishers send the event and business messages to a cloud database. Then, online BI editors access the database to generate reports and dashboard. In this implementation, we rely on the cloud’s power to provide a powerful ingestion and execute complex requests. The cloud streaming databases such as Google big query are more suitable for this type of processes than the classical relational databases often dedicated to transaction and consistency.

Google Data Studio example

External process implementation 03

This implementation is made up of three elements:

• A buffer: Kafka receives the messages from OpenESB in an asynchronous mode
• An engine: Storm processes the raw events and messages and generates usable records
• A persistence system: MongoDB stores the record generated by the engine and made them available for any analytic tool

This implementation is powerful and scale horizontally and matches the most demanding configurations, but it requires a large knowledge to be deployed and maintained.

License and fees

Use of the OpenESB monitoring framework is subject to annual support subscription that gives you the right to use it in your configuration. The cost depends on the project, the number of instances and many other parameters. Please Feel free to contact us for any further information on this subscription. 

Pymma is organising two free half-day seminars in the USA to present OpenESB V 3.1. OpenESB: The Open Extended Service Bus has been dramatically improved in response to new integration challenges. The seminars are intended to present the new features developed by the OpenESB community and Pymma’s team.

OpenESB has widened its application scope and improved its capability on many topics.

Intelligent message routing: OpenESB message routing has been completely reviewed to provide a powerful flexibility. For each connection, you can define how OpenESB chooses the best service implementation. OpenESB considers the running context and applies predesigned routine algorithms (response time, number of failures, version) or your routing algorithms to route your messages to the best service implementation for your message.

Service composition: Our developers have achieved wonderful results with the OpenESB services Composition. Whatever the service-oriented application you develop (classical SOA or Microservices), you can always deploy your development with the granularity that fits your design. The OpenESB community implemented a new import-export feature for the service references to easen the communication and exchanges between teams.

Big Data: The OpenESB architecture based on multi-instances deployment proved its capability to process billions of messages on production on a daily basis. Connections to Big Data products have been added to access or to store this amount of information. OpenESB v3. 1 provides access to the Big Data world through many connectors such as the Apache Kafka Connector (https://kafka.apache.org) or the Apache Geode Connector (http://geode.apache.org).

Internet of Things: OpenESB v3.1 comes along with a new MQTT connector (http://mqtt.org). MQTT is the most used Machine to Machine protocol on the market. OpenESB V3.1 receives and processes IOT messages from any devices (Street Light, Container, Counter…) and embedded them in your business processes.

Better and Easier Scalability: Using stateless processes to improve scalability is one of the main rules of IT Architecture. OpenESB v3. 1 implements a new unique native and an effortless feature to share a business context between multiple instances and makes the stateful features such as the correlation useless. This new feature improves dramatically the scalability of the business processes running with OpenESB.

Monitoring: OpenESB offers a new powerful framework to monitor your business processes and to generate reporting and dashboard. OpenESB publishes technical logs and business traces to many persistence systems: file, relational database, Elasticsearch, MongoDB, Kafka, Apache Geode, Google Big Data, Microsoft Power BI…

The seminars will take place on 30th November 2017(14h-17h) in Manhattan New York and on 6th December 2017 (14h-17h) in San Francisco (CA).

NY booking: https://www.eventbrite.com/e/openesb-american-tour-2017-new-york-city-tickets-39687440202

SF booking: https://www.eventbrite.com/e/openesb-american-tour-2017-san-francisco-tickets-39688656841

Microservices and SOA: the antagonism is not technical but...

Figure 1: Microservices and SOA war

Recently, browsing on YouTube, I re-watched the lecture given by Martin Fowler in 2014 on the microservices. As usual, Martin was a wonderful speaker, clear and concise. But in the middle of the lecture when he talked about differences between Microservices and SOA, I found his speech a bit more confusing and his conclusion on that topic was unclear and vague. If you pay attention to his speech, before saying his conclusion on that chapter, he reported two witness statements: the first was from SOA people who developed successful SOA projects, surprised by the lack of innovation and freshness in the microservices definition and practices, and the second, from microservices people, mainly coming from big companies, fed up with SOA, who see SOA as ‘Enterprise Services Bus’ and ‘committee that wants to lay down standards’. Then, Martin concludes by an evasive definition that Microservices is a subset of the SOA.

However, he did not give any border line between the two domains, but said that the term, SOA, is a too broad and the microservices is a useful subset of SOA.

Pymma offers new exciting projects for interns in 2022! Due to the COVID crisis, all the projects can be done remotely. Consequently, we are looking for highly organised and motivated people to join our team. Good Java skill is required and an understanding of team tools such as Git, Jenkins is required. Integration background is a plus. 

Have a look at our internship page

Send us your CV and or contact us by email.

Good luck with your study in that hysteric time!

The Apache community informed us that a vulnerability has been found in the famous library Log4J. Please have a look here. We inform our users that OpenESB is not using LOG4J except for the distributed cache Geode Driver. If you use that plugin, please contact us ASAP.