Idea ID: 2801487

More Parameters for Purge History SQL Scripts

Status : Waiting for Votes
over 1 year ago

The current PurgeHistory Stored Procedures are a great way to maintain the DB, however, I would like to suggest some additional parameters to allow for a more finegrain usage:

1) Parameter to toggle Flow Input/Output Deletion

The OO API to purge the Execution Summary has inputs to toggle the Deletion of Flow Inputs and Flow Outputs. The SQL scripts should have this option as well. Use Case: You want to delete only the detailed info in the step log but not the input/outputs of the flow due to retention policies.

2) Parameter to Purge Particular Runs

The SQL procedure should have a parameter that allows me to trigger the purging for given Run IDs, irregardless of the given timestamp. This option would probably not be used in a scheduled job but is very usefull for ad-hoc purging. Use Case: Due to an unforeseen error in the flow logic, a flow runs a very long loop very often, creating unnecessary data. I immediately want to purge this new data and not wait for our scheduled purge.

  • Hello community,

    we would like to perge certain flows by id as well.

    In our environment we poll several itsm tools (SM, ServiceNow, BMC) for open tasks / incidents that can be automated by OO. Because of the polling mechanism we have flow runs without a result. All these flow runs without a result we want to delete immeaditly.

    So it would be great to purge flows by id.

    Best regards,

     

    Michael

  • I'm also interested in deleting flow by RunID, of course this should be only possible by the SQL Stored Procedure and not by API.

  • Hi Vlad,

    thanks for your response. I appreciate your thoughts on this very much!

    Let me clarify what I mean by option 1: I am not looking for a toggle to enable the purging of the Flow Inputs but rather to disable it. So currently, when executing the Stored Procedure, we only have the option to purge all of the flow data including flow inputs/outputs. I'd like to keep the flow inputs/outputs to comply with several retention polices.

    Regarding the option 2: As the person executing the stored procedure already needs to have somewhat privileged access to the DB, they already have the opportunity to "cover their tracks", so to speak. Is there something I am missing here?

    Best regards,

    Felix

  • Hi,

    Option 1 is already in the scripts by default. In the purge flows/api this is a separate option in order to make the purge flow take less time by running them specifically targeting  only step log data, only step inputs/outputs, only execution summary or everything.

    For scripts this  is not needed since they can delete the same amount of data as flows in a fraction of the time. Therefore a complete purge history event on scripts will handle:

    -step log data,

    -flow inputs

    -flow outputs

    -executed flow graphs

    -execution summary (optional)

     

    Now for option 2 while I can see the usefulness of such an option there is the risk of having this option be used as a cover tool for malicious intent. The scenario is that someone who wants to create trouble can run a flow to delete VMs /falsify records and then can use this option to remove the execution from historical data making it look like nothing happened  and the failure occurred "naturally". Because of this the option itself poses a security risk since we would be providing an attacker with tools to cover up their tracks.

    While for general availability such a modification to the scripts is not likely to be created, for emergency use by support teams scripts to handle such data can be created as an on the fly solution.

    Regards,

    Vlad Moldovan

  • To my experience, suggestion #2 is a very important feature to get rid of lots of flow data trash related to a zombie flow.