Recently we ran into an issue where our Adyen WebHook notification was failing to process a response which caused a backlog of Payment notifications and had an impact to our payment processing for an entire day’s worth of orders. While the significant issues that arose have more to do with process issue, we did need to understand the failure itself and why it happens.
Today we have a very simple webhook created where Adyen will post a JSON payment to IBM OMS which will store the message in a queue for async processing and respond that the webhook is successful. However on that day, there was some special JSON escaped character that caused XML serialization errors and the webhook was failing. Due to the failed response, this caused a backlog of payment events being sent to the OMS and the back log of confirming payment operations until the problems was reported and addressed (approximately 24 hours).
The webhook in question that Adyen calls in our platform is https://omshost/smcfs/restapi/executeFlow/AdyenNotifications?_informat=json
And for the case in question, the payload of the webhook had a special character (\f) that caused the issue such as
{
2 "live": "true",
3 "notificationItems": [
4 {
5 "NotificationRequestItem": {
6 "additionalData": {
7 "fieldName": "Unknown\f",
}
66 }
67 ]
68}
The current service has no custom logic and is using the standard infrastructure of OMS where it is receiving the JSON payload, using the OMS foundation that has an XML version of that payload and then serializing the XML to a queue. However in this case, the response was
{"errors":[{"ErrorDescription":"Error description not available","httpcode":400,"ErrorCode":"Error during serializing xml document to text: java.io.IOException"}]}
After some investigation, we realized this seems to be more inherit in the IBM framework itself and JSON handling is not adequately supported do still levering the XML 1.0 specification has not been upgraded to comply to the 1.1 standard introduced in 2006.
JSON itself is UTF-8 at its core and has special notations for form feed (\f), newline (\n), beep (\b) and other odd characters.
Backspace is replaced with \b.
Form feed is replaced with \f.
Newline is replaced with \n.
Carriage return is replaced with \r.
Tab is replaced with \t.
Double quote is replaced with \”
Backslash is replaced with \\
A lot of these characters are fine but the \f and \b are the characters in question. Because the XML 1.0 spec does not support form feeds, when it attempts to serialize the XML, it errors out.
Attached are java program using the IBM OMS functions to load JSON & XML documents that have form feed characters but when trying to write out the XML, it fails once it accounts an element that has the attribute with this character. When you update the JSON or XML payloads in the test to remove the form feed, the entire input is printed as expected.
While we are attempting to understand the source of the data and how that was input at all, due to the service disruption it occurred, we need a solution within IBM to handle this type of PayLoad and be able to strip the offending character out or upgrade the posting or leverage the 1.1 spec when writing and reading to help account for these character payloads. The webhook provider has a replay mechanism so we do not want to falsely response to these requests so it would be best on how to handle appropriately due to the fact the OOTB JSON support corrupts the payload already.
