SFTP Client Adapter “ResponseTimeout” setting does not have the desired effect causing hung BP activity

SFTP Client Adapter “ResponseTimeout” setting does not have the desired effect causing hung BP activity

Documentation Reference Links for B2BI 5.2.x/6.0.3/6.1.0:

https://www.ibm.com/support/knowledgecenter/SS3JSW_5.2.0/com.ibm.help.svcs_adpts_m_z.doc/SFTP_Client_CD_svc.html

https://www.ibm.com/support/knowledgecenter/SS3JSW_6.0.3/integrating/integrating/integrator/SFTP_Client_CD_svc.html

https://www.ibm.com/support/knowledgecenter/SS3JSW_6.1.0/integrating/integrating/integrator/SFTP_Client_CD_svc.html

Based on the published documentation for the SFTP Client construct there is a default ResponseTimeout value defined within the SFTP Begin Session service by which this timeout length can be overridden during certain SFTP operations. In our Technical Support Case TS004564739 we specifically call out the issue around the Change Directory function, though the same "ResponseTimeout" facility is present in SFTP CD/LIST/PUT/GET/DELETE/CHMOD/etc. operations as well and is similarly affected/ignored.

The B2BI documentation states repeatedly, across multiple SFTP operations under the different B2BI versions:

ResponseTimeout The maximum number of seconds it can take for the trading partner system to respond before the session times out and terminates. If a number less than 30 is specified, 30 seconds will be used. Optional. Default is ResponseTimeout specified in SFTP Client Begin Session service.

PROBLEM SUMMARY:

The SFTP Client Adapter “ResponseTimeout” setting does not have the desired effect in most SFTP Client Adapter operations. Arguably this is a bug, however we have been instructed by IBM Support to file this as a Request For Enhancement (RFE) as well. This unexpected and undesirable behavior of mishandling of the ResponseTimeout causes business processes to hang and queue depth to grow for each missed timeout.

BACKGROUND DETAIL:

Based on the above description of "ResponseTimeout" it should be reasonable to deduce and expect that the “response” timeout would be experienced if the remote SFTP server was unable to initiate and/or complete the requested operation. Given an example of a short or quick operation like change directory, chmod, move, etc. one would normally expect a “response” timeout should be reflective of the actual operation completion within a reasonable amount of time. However, that is not the case. What was even more unexpected was that we were told by IBM support that the ResponseTimeout value does not exhibit this expected behavior.

Under certain conditions, network failures, or during other unusual circumstances, operations like change directory could potentially take much longer than the response timeout value, which would be undesirable. In our example documented within the technical support case, due to unexpected back-end network issues on a remote SFTP server, we were experiencing numerous instances of change directory operations that were holding up business processing for multiple hours at a time. We found the SFTP Client Adapter and its CD operation specifically are not honoring the “response timeout” setting. The remote server did not respond with “change directory complete” in a timely fashion, and the business processes became effectively hung at that point in time, and BP Queue Depth increased for every file being sent across the connection to the server with the issue.

One would think it should also be reasonable to assume that while an entire “PUT”, “GET” or “LIST” longer running operation (start to finish) where data/file is actively being transferred, may not complete within the response timeout timeframe but that the session would remain alive and active since regular responses are received as long as data is being transmitted and acknowledged and there should be data consistently being exchanged until the transfer/operation itself completed. The response timeout counter in these cases should be consistently reset while data is still flowing through the connection considering those as active server responses.

The SFTP Client Adapter ResponseTimeout value should be handled more appropriately.

See also: https://watsonsupplychain.ideas.aha.io/ideas/B2BI-I-803