I wish to do IDB maintenance in Data Protector 9.0

I wish to do IDB maintenance in DP 9.0, Which command to execute or please confirm if it is same as we do in DP 6/7 version.

Parents
  • Hi tabrez16,

    what's your goal with this? There is a manual procedure for old raima database in versions prior 8.0, that is no longer needed with new postgresql IDB. Its autovacuum procedure takes care of this 

  • I want to do purge and readdb and write DB.

    Is the process same for DP9 as we do in DP6/7 version.

    Or any other commands need to be executed to perform purge and read write.

    if it is done automatically, please confirm me which parameter need to be checked to verify it is done or not. is there any global parameter to be checked to verify whether it is happening automatically or not

  • AutoVacuum is not enabled

    If i remove the # and enable the Autovaccum, when will this Autovaccum be performed.

    What else need to ne enabled to have no impact to backups.

    As per below note it requires track_counts to also be on. Any thing else. Please share more info and bext way to enable AutoVacuum.

    # AUTOVACUUM PARAMETERS
    #------------------------------------------------------------------------------

    #autovacuum = on # Enable autovacuum subprocess? 'on'
    # requires track_counts to also be on.
    #log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and
    # their durations, > 0 logs only
    # actions running at least this number
    # of milliseconds.
    #autovacuum_max_workers = 3 # max number of autovacuum subprocesses
    # (change requires restart)
    #autovacuum_naptime = 1min # time between autovacuum runs
    #autovacuum_vacuum_threshold = 50 # min number of row updates before
    # vacuum
    #autovacuum_analyze_threshold = 50 # min number of row updates before
    # analyze
    #autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum
    #autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze
    #autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum
    # (change requires restart)
    #autovacuum_vacuum_cost_delay = 20ms # default vacuum cost delay for
    # autovacuum, in milliseconds;
    # -1 means use vacuum_cost_delay
    #autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for
    # autovacuum, -1 means use
    # vacuum_cost_limit


  • wrote:

    I want to do purge and readdb and write DB.


    Why? Cargo cult?

    The whole raison d'être for the IDB change between 7.0 and 8.0 was to get rid of the very reasons that made this procedure necessary before. The old IDB had a file name part that managed to reduce the total space necessary for storing every single path name in all of the backups, mainly by having every unique name stored just once and optimized further by chopping them up into path components of various lenghts. The disadvantage was that environments with a large influx of new (never seen before) path names and - as a result - a large rate of path names getting obsoleted when backups expired, would accrete a pile of filename junk in that part of the IDB. So much you had to regularly purge it. And that alone would only make things worse, as the resulting comb of a fragmented DB table, when filled again from the beginning, would turn into a tragic source of random read access on every following IDB check or backup, with times required for these operations exploding into multiple hours. So to get rid of the fragmentation caused by purge, you had to dump and restore the IDB in place, as this would compact it down to the remaining filename entries and make it more linear access, specifically when it started to grow again.

    Now all this is thankfully gone. The PostgreSQL part is just the client, object, session, device and media metadata and even with vast numbers of media, it is of a trivial size and doesn't need any significant maintenance. The filenames are now part of the DCBFs which has one drawback: They are now much larger. There is no more filename deduplication as we had it before. But this also means that filenames are coming (with backups) and going (with recycling of expired media) fully automatically as DCBFs are coming and going. There is no more explicit maintenance here, it's implicit in the DCBF lifecycle. A classic time-space-tradeoff in the end. And we're speaking of human time here, given how long a purge/writedb/readdb maintenance operation could take and occupy my full attention (lest I wreck the entire installation with a wrong spell). Now we have something else to find to do with this time. Luckily, there's enough other issues in DP8/9 to waste it with ;)

    HTH,
    Andre.

  • Guys,

    tabrez16 brings up a valid point.  This got me thinking, and I decided to check my postgresql.conf file.  I see the same thing: #autovacuum = on - which, according to the .conf file itself ("Comments are introduced with # ANYWHERE on a line" [emphasis is mine]), means this is commented out.

    So, HPE folks: Does this mean the autovacuum is running or not?  Is there another file that I need to look at  (other than \ProgramData\OmniBack\server\db80\pg\postgresql.conf)?

    Thanks

    Ken

  • Guys,

    tabrez16 brings up a valid point.  This got me thinking, and I decided to check my postgresql.conf file.  I see the same thing: #autovacuum = on - which, according to the .conf file itself ("Comments are introduced with # ANYWHERE on a line" [emphasis is mine]), means this is commented out.

    So, HPE folks: Does this mean the autovacuum is running or not?  Is there another file that I need to look at  (other than \ProgramData\OmniBack\server\db80\pg\postgresql.conf)?

    Thanks

    Ken

  • Hi Ken,

    I don't really know for sure, but I'd think that

    # The commented-out settings shown in this file represent the default values.

    would mean that all the values found in the autovacuum parameters section, while commented out, just state the compiled-in defaults. Clarification on that by HPE would be great, of course.

    HTH,
    Andre.

  • Dimitar was guiding us to the right direction already. Starting with Data Protector 8.0 we're using PostgreSQL 9.1 as the Internal Database. Even when autovacuum is commented out in postgresql.conf is will run by default. This can be seen much better on UNIX Cell Managers when looking at the ps command output.

    Daily maintenance will purge obsolete items from IDB and autovacuum take care of the rest.

    autovacuum (boolean)
    Controls whether the server should run the autovacuum launcher daemon.
    This is on by default; however, track_counts must also be enabled for autovacuum to work.
    This parameter can only be set in the postgresql.conf file or on the server command line.

    (https://www.postgresql.org/docs/9.1/static/runtime-config-autovacuum.html)

    Hope this helps!

    Regards,
    Sebastian Koehler

  • Sebastian,

    So, based on earlier replies and yours, I'm presuming that finding a line like

    #track_counts = on

    in the postgresql.conf file means the same thing, that track_counts is actually enabled and the autovacuum process is then doing its thing, happily chugging away sucking all the crud out of my >260 GB DCBF folder structure?

    Thanks

    Ken

  • Hi Ken,

    sorry to disappoint you, but autovaccum has nothing to do with the DCBF part. :)

    Autovaccum will take care of what is stored in the DatabaseFiles part of the IDB. Based on my experience the DatabaseFiles part is less than 1% of the whole IDB. You can check that with omnidbutil -info or a recent IDB backup (objects size) for the current size. With 260 GB DCBFs you should have 1-2 GB DatabaseFiles.

    The DCBFs part is maintained by daily mainteance usually running at 12:00 o'clock. For exmaple, if catalog protection of a cartridge expires the catalog data is purged from disk. If a media is overwritten the DCBF will be deleted and recreated. If a media is expired, the DCBF is deleted. Of course references to the DCBFs are stored in the DatabaseFiles part of the IDB. But the vast majority of the data (fnames, etc) are stored in the daily catalog binary files.

    And yes, track_counts is also on by default.

    track_counts (boolean)
    Enables collection of statistics on database activity. This parameter is on by default, because
    the autovacuum daemon needs the collected information. Only superusers can change this setting.

    Regards,
    Sebastian Koehler

     

  • Sebastian,

    Thanks for the clarification.  My database grew a lot early and as we import more servers from other cells it keeps growing, but it appears to have stabilized.  The only thing I'd say is make it easier to install DP to something OTHER than C:.  I don't like my C: drives getting real big, and we generally like to keep apps off the C: on our servers.

    I do think the integration with StoreOnce Catalyst has been a good thing in terms of media management and keeping the database size under control.  It has really reduced our media management workload.

    All previous comments about DP 9 aside, it has allowed me to (at this point) consolidate some 40 cells into one, and another dozen or so will soon join the consolidation.

    Thanks

    Ken

  • Sebastian,

    Thanks for the clarification.  My database grew a lot early and as we import more servers from other cells it keeps growing, but it appears to have stabilized.  The only thing I'd say is make it easier to install DP to something OTHER than C:.  I don't like my C: drives getting real big, and we generally like to keep apps off the C: on our servers.

    I do think the integration with StoreOnce Catalyst has been a good thing in terms of media management and keeping the database size under control.  It has really reduced our media management workload.

    All previous comments about DP 9 aside, it has allowed me to (at this point) consolidate some 40 cells into one, and another dozen or so will soon join the consolidation.

    Thanks

    Ken

Reply
  • Sebastian,

    Thanks for the clarification.  My database grew a lot early and as we import more servers from other cells it keeps growing, but it appears to have stabilized.  The only thing I'd say is make it easier to install DP to something OTHER than C:.  I don't like my C: drives getting real big, and we generally like to keep apps off the C: on our servers.

    I do think the integration with StoreOnce Catalyst has been a good thing in terms of media management and keeping the database size under control.  It has really reduced our media management workload.

    All previous comments about DP 9 aside, it has allowed me to (at this point) consolidate some 40 cells into one, and another dozen or so will soon join the consolidation.

    Thanks

    Ken

Children
No Data