We are running the ITX 10.1.2.0 Launcher Ubuntu image in multiple pods in our production batch environment. Each pod is running a single Launcher instance and each Launcher is currently only running a single system, although both of these configurations will likely change over time (i.e. more Launchers/systems within a single pod). All Launchers use common PVCs for data persistence during pod scaling. For maximum throughput we are scaled to run 6 pods. The main "Tenant Map Handler" application receives a "request" ZIP file in a landing folder on the PVC ("/tnc/landing/polling") for which the map has a source event defined for the inbound request ZIP file. The map unzips the request, processes the request to run the tenant's map, and produces a "response" ZIP file with the output from the tenant's map execution. What we are seeing intermittently is that the response ZIP file is being created with incomplete contents, and the Tenant Map Handler map is failing due to missing files during the re-ZIPping process. I feel the issue of missing files in the response ZIP file is occurring due to multiple pods and here is why: 1. When a request ZIP file arrives in the PVC “/tnc/landing/polling” folder that one or more Launchers are watching, the ZIP file is moved into a work folder created by the Launcher named as “/tnc/landing/polling/.WTX_<port>_<podname>” (ex: ” .WTX_7002_tnc-launcher-batch-prod-9fdc94”). This is called the private Trigger Directory and the arrival of a file in this folder causes the appropriate map to be triggered. 2. These Trigger Directory folders are supposed to be uniquely named so that only the Launcher that created it will be triggered by the arrival of a trigger file (ZIP, in this case). 3. However, there appears to be a 40-character limit for this Trigger Directory folder name being imposed by ITX when it creates them. The pod name is composed of the namespace, replica set name and a unique random 5-character alphanumeric ID (ex: "tnc-launcher-batch-prod-9fdc9454c-tvxl5", where "tnc-launcher-batch-prod" is the namespace, "9fdc9454c" is the replica set name, and "tvxl5" is the unique ID assigned to the pod instance). The 40-character limit is cutting off the pod name before the 5-character unique ID when it creates the folder. 4. So, if additional pods are started and use the same replicaset name (which they usually do), we end up with several Launchers sharing a Trigger Directory and then triggering off the arrival of a single trigger file in the one Trigger Directory. Most of the work file naming generated by the Tenant Map Handler map is threadsafe, because the map prepends a unique ID to all input and output files (ex. threadID-332462201314107821). One exception to this is the name used for the temporary response ZIP file while the map adds files to it (ex. “T_B2B_275BDEBAT_B0275B00000006362056_06082023153809_TEMP_OUT.zip”). This temporary file is renamed to the final filename once it has been built so another response ZIP file delivery app will “see” the file and process it. I believe the response ZIP files are incomplete because several instances of the Tenant Map Handler map are triggered by the same request ZIP file and processing it at the same time and are all stepping on each other while building that temporary output file, or while renaming it to the final response ZIP filename. We can add the threadID to this temporary response ZIP filename, which should take care of the incomplete response ZIP files. This is an easy fix. However, the much bigger issue is how to get the Launcher to create truly unique private Trigger Directory names so that only one map instance will occur per trigger file. There doesn’t appear to be an ITX configuration setting to control this.
Delivered in ITX 11.0. Expanded to 50 character limitation.