IBM CoC Data Extract Tool - Date Range Attribute required for extracting the data for the particular time period

Hi Jebastine, I hope you have had a chance to review my previous response and the expected use of the Data Extract Tool. Based on further discussions with Engineering, we are not planning to add a date range for the Data Extract Tool and recommend following the guidelines specified above for extracting data.

Hi Jebastine, I have been discussing this with engineering. Generally speaking, the data extract tool is meant to extract data on an ongoing basis (https://www.ibm.com/docs/en/order-management?topic=overview-data-extract-agent). While it is possible to run ad-hoc extracts (https://www.ibm.com/docs/en/order-management?topic=deo-running-data-extract-service-ad-hoc-retrieval-requests), this is not meant to be a common occurrence. For smaller ad-hoc queries, the DB Query Tool can be used as mentioned earlier.

For your scenario - extracting data so that you can purge your DB tables - you should be able to simply run the extract starting at your oldest data point and use the MaxDaysExtractAtOnce setting to break up the extracts into small chunks to help process the large data set efficiently. You don't need to manually calculate chunks of data to extract. The first run of the process will take longer to execute, but if you set MaxDaysExtractAtOnce appropriately, it will automatically optimize itself to process the initial workload and then should be left running to continue to extract your data. After this first run, you will have the data you mentioned - 2016 through 2019, broken up into chunks, as well as data beyond 2019.

Hi Aaron,

Thank you for quick response.

Currently we have enabled the order purge process in PROD env. As part of purge, the corresponding transaction table records are purged and archived into History tables.

As part of DB performance review, IBM team has recommended that to remove the old archived records from history table so that it will increase the DB performance. So we are enabling the IBM CoC Data Extract tool for extracting the history table records and storing the data extract files to our local SFTP servers.

For examples, I am sharing the 1 year data for 2 history tables records and we have other tables also.

YFS_ORDER_HEADER_H having the 4821474 records for 2019 year.

YFS_ORDER_LINE_H having the 12584767 records for 2019 year.

In DB Query tool, it allow only 10k records for at a time but it will get the time out error. so we can't use the DB query tool for extracting the data for this scenario.

Based on IBM team recommendation, we are enabling the IBM CoC Data Extract tool. This is an automated process so there is no manual work is needed for pulling the records.

In this tool, currently there is no date range attributes to configure the particular time period (for example 01-01-2016 to 01-01-2019) for extracting the data.

If Data extract tool provides, the date range attribute then we can directly update the configuration and pull the records for particular time period.

Please let me know if you need any further details.

Thanks,
Jebastine

Hi Jebastine, thank you for submitting your idea. If I understand your scenario correctly, you are looking to extract data from a certain time period.

We do get this request occasionally from our customers, and our recommended option is to use the DB Query tool for ad hoc queries rather than a manipulation of the data extract process itself. The data extract process is geared toward ongoing extract of data for reporting purposes, but ad hoc usage can lead to unexpected load on DB resources that we want to avoid.

The DB query tool can be used to query for exactly the data you are looking for, which can also be saved.

Can you please review the DB Query tool and see if that meets your needs? If not, can you please provide additional context for your scenario that cannot be met by the DB Query Tool?

Thanks,

Aaron