OpenTelemetry (or ELK) integration for Connect:Direct statistics

Utilizing an infrastructure like Splunk and ELK, one can trivially create dashboards, event monitoring, web hooks etc. These help to understand and analyze the changing requirements, i.e. provide convincing information about your additional license needs and extra features necessary. One can easily graph the incoming and outgoing processes, grouped or filtered by netmap and other things for providing compelling insights into all your management reports.

Currently, the statistics of C:D are an useful tool for event logging and error detection for a technician. However, after he got an error reported, he needs to log into C:D on the correct node and read the statistics painfully within the UI (Requester/WebUI) or unofficially query the SQL database behind. However, if you got a non-trivial amount of nodes, net maps, or file transfers, this is error-prone and tedious. At least our business requires us to integrate into an existing central logging and monitoring infrastructures enabling the former potential benefits.

As a central logging is possibly a standard requirement for many enterprises, in our point of view it makes sense for you to create an official IBM/Sterling module/extractor which is aware of all the many scopes of the statistics and e.g. their text templates. It should be able to create/send nice (json-based) telemetry into an OpenTelemetry Connector (i.e. LogStash if you target the elastic stack only). The connector can then be configured to filter out or obfuscate any problematic data and further send the telemetry to your logging/observability infrastructure of choice.

Furthermore, this helps to connect the statistics with much more information about your infrastructure data like processors or telemetry of file producing applications.

Our research:

Probably, implementing this could solve the idea IZDS-I-31 as well which, however, concretizes on log file export.

As a "workaround" each enterprise could write a log extractor script on her own: e.g. https://www.ibm.com/support/pages/can-you-extract-information-directly-connectdirect-unix-logs or by utilizing the SQL database structure. However, this is error-prone, time consuming, and the very next update of C:D will probably break these scripts anyways. If this is written utilizing the database access it still needs to parse the data key/values which is clearly implementation dependent.