IBM Sterling Ideas

Shape the future of IBM!

We invite you to shape the future of IBM, including product roadmaps, by submitting ideas that matter to you the most. Here's how it works:

Post your ideas

IBM is transforming its request for enhancement (RFE) process. The purpose of the transformation is to provide a more consistent experience for you to submit requests and to enable IBM product owners to respond to your requests more quickly. For more information click here.

Start by posting ideas and requests to enhance a product or service. Take a look at ideas others have posted and upvote them if they matter to you,
1. Post an idea
2. Upvote ideas that matter most to you
3. Get feedback from the IBM team to refine your idea

Help IBM prioritize your ideas and requests

The IBM team may need your help to refine the ideas so they may ask for more information or feedback. The offering manager team will then decide if they can begin working on your idea. If they can start during the next development cycle, they will put the idea on the priority list. Each team at IBM works on a different schedule, where some ideas can be implemented right away, others may be placed on a different schedule.

Receive notifications on the decision

Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.

IBM CoC Data Extract Tool - Date Range Attribute required for extracting the data for the particular time period

In CoC Data extract tool, currently there is no date range attribute to configure directly into the tool.

We need to manually calculate the days based on current time and required date. Based on that, we need to update the FirstRunExtractInDays and maxdaysextractatonce values.


  • Guest
  • Apr 27 2022
  • Needs more information
What is your industry? Retail
How will this idea be used?

If CoC data extract tool provides the date range attribute then we can directly update the particular date range details and it pulls the data extract from configured table.


  • Admin
    Aaron Cutter commented
    6 May 01:30pm

    Hi Jebastine, I have been discussing this with engineering. Generally speaking, the data extract tool is meant to extract data on an ongoing basis (https://www.ibm.com/docs/en/order-management?topic=overview-data-extract-agent). While it is possible to run ad-hoc extracts (https://www.ibm.com/docs/en/order-management?topic=deo-running-data-extract-service-ad-hoc-retrieval-requests), this is not meant to be a common occurrence. For smaller ad-hoc queries, the DB Query Tool can be used as mentioned earlier.

    For your scenario - extracting data so that you can purge your DB tables - you should be able to simply run the extract starting at your oldest data point and use the MaxDaysExtractAtOnce setting to break up the extracts into small chunks to help process the large data set efficiently. You don't need to manually calculate chunks of data to extract. The first run of the process will take longer to execute, but if you set MaxDaysExtractAtOnce appropriately, it will automatically optimize itself to process the initial workload and then should be left running to continue to extract your data. After this first run, you will have the data you mentioned - 2016 through 2019, broken up into chunks, as well as data beyond 2019.


  • Guest commented
    29 Apr 07:24pm

    Hi Aaron,

    Thank you for quick response.

    Currently we have enabled the order purge process in PROD env. As part of purge, the corresponding transaction table records are purged and archived into History tables.

    As part of DB performance review, IBM team has recommended that to remove the old archived records from history table so that it will increase the DB performance. So we are enabling the IBM CoC Data Extract tool for extracting the history table records and storing the data extract files to our local SFTP servers.

    For examples, I am sharing the 1 year data for 2 history tables records and we have other tables also.

    YFS_ORDER_HEADER_H having the 4821474 records for 2019 year.

    YFS_ORDER_LINE_H having the 12584767 records for 2019 year.

    In DB Query tool, it allow only 10k records for at a time but it will get the time out error. so we can't use the DB query tool for extracting the data for this scenario.

    Based on IBM team recommendation, we are enabling the IBM CoC Data Extract tool. This is an automated process so there is no manual work is needed for pulling the records.

    In this tool, currently there is no date range attributes to configure the particular time period (for example 01-01-2016 to 01-01-2019) for extracting the data.

    If Data extract tool provides, the date range attribute then we can directly update the configuration and pull the records for particular time period.

    Please let me know if you need any further details.

    Thanks,
    Jebastine

  • Admin
    Aaron Cutter commented
    29 Apr 03:15pm

    Hi Jebastine, thank you for submitting your idea. If I understand your scenario correctly, you are looking to extract data from a certain time period.

    We do get this request occasionally from our customers, and our recommended option is to use the DB Query tool for ad hoc queries rather than a manipulation of the data extract process itself. The data extract process is geared toward ongoing extract of data for reporting purposes, but ad hoc usage can lead to unexpected load on DB resources that we want to avoid.

    The DB query tool can be used to query for exactly the data you are looking for, which can also be saved.

    Can you please review the DB Query tool and see if that meets your needs? If not, can you please provide additional context for your scenario that cannot be met by the DB Query Tool?

    Thanks,

    Aaron

By clicking the "Post Comment" or "Submit Idea" button, you are agreeing to the IBM Ideas Portal Terms of Use.
Do not place IBM confidential, company confidential, or personal information into any field.