Trusted Contributor.. T0mTran Trusted Contributor..
Trusted Contributor..
1843 views

What are "Origins" and "TRIM Bulk Loader"?

Jump to solution

Hi all,

 

I came across these two featuers in the Help file but couldn't find anymore information about them?

 

What are they? What are their purpose? And more importantly, Where in HP official documents can I find more info about them?

 

thank you,

 

Tom

 

 

1 Solution

Accepted Solutions
Established Member.. Jason Boswell
Established Member..

Re: What are "Origins" and "TRIM Bulk Loader"?

Jump to solution

 

Concepts

Whenever you wish to use BulkDataLoader to import records, you will need to use a TRIM SDK object called an Origin. This object encapsulates some defaults which can be useful in determining how new records are initialized and it also provides a way of identifying records in TRIM after an import run has completed. An Origin has an associated OriginHistory object which records details of each invocation of the Origin. In the TRIM user

interface, you can navigate to these runs by selecting the Origin (see

Tools Record Origins) and from there, navigate to all OriginHistory objects representing all the runs done for this Origin. This OriginHistory object is also used by the Event Processor to efficiently schedule content indexing, word indexing and retention trigger calculations.

How It Works

The BulkDataLoader provides a function called SubmitRecord(), which replaces the Record.Save() call you would normally use when creating new records. This function places the underlying TRIM database management layer into a "Bulk Loading" state, telling TRIM to write details of SQL INSERT transactions to temporary disk files, rather than having INSERT’s executed individually by the Database Engine.

A set, or batch, of records would be processed in this way, continually adding to the temporary files until all the records in the batch have been added. At this stage, the temporary disk files need to be passed up to the SQL engine for bulk loading. This is done by calling the ProcessAccumulatedData() function. The batch size can be any value (greater than zero), but the larger the batch size, the better the performance gain. For large import tasks in the millions of records, a batch size of 10,000 is reasonable. Any errors that occur during the loading of a single batch result in the whole batch being discarded, so this needs to be kept in mind if very large batch sizes are used.

View solution in original post

4 Replies
Highlighted
Grundy Acclaimed Contributor.
Acclaimed Contributor.

Re: What are "Origins" and "TRIM Bulk Loader"?

Jump to solution

Its covered in the helpfile briefly, in the TES helpfile specifically I think.

It has come up before in the forum and I explained what I could.

 

The new bulk loader is just a back-end import tool which bypasses many of the slow overheads of the TRIM interface.

One thing that is needed when bulk loading like this is some information that configures where the data is coming from and going, including the electronic documents. This is where Origins come in.

 

If you're not looking at bulk loading (Talking millions of documents here), then just ignore the bulk loading feature and 'Origins'.

 



::::::::::::::::::::::
NOT A HP EMPLOYEE
::::::::::::::::::::::

INFORMOTION.com.au
Trusted Contributor.. T0mTran Trusted Contributor..
Trusted Contributor..

Re: What are "Origins" and "TRIM Bulk Loader"?

Jump to solution

Thanks, Grundy.

 

Is the software included in HP TRIM Base license?

 

If not, is it worth it to buy Bulk Loader? Yes, we will be dealing with millions of documents, but we have TRIM Port. If TRIM Port can do the job, we probably don’t need Bulk Loader, right?

 

Which overheads are being bypassed? I’m concerned if Bulk Loader bypasses steps that are important to us, then we have to go through every record again after the migration – just defeat the purpose.

 

0 Likes
Established Member.. Jason Boswell
Established Member..

Re: What are "Origins" and "TRIM Bulk Loader"?

Jump to solution

 

Concepts

Whenever you wish to use BulkDataLoader to import records, you will need to use a TRIM SDK object called an Origin. This object encapsulates some defaults which can be useful in determining how new records are initialized and it also provides a way of identifying records in TRIM after an import run has completed. An Origin has an associated OriginHistory object which records details of each invocation of the Origin. In the TRIM user

interface, you can navigate to these runs by selecting the Origin (see

Tools Record Origins) and from there, navigate to all OriginHistory objects representing all the runs done for this Origin. This OriginHistory object is also used by the Event Processor to efficiently schedule content indexing, word indexing and retention trigger calculations.

How It Works

The BulkDataLoader provides a function called SubmitRecord(), which replaces the Record.Save() call you would normally use when creating new records. This function places the underlying TRIM database management layer into a "Bulk Loading" state, telling TRIM to write details of SQL INSERT transactions to temporary disk files, rather than having INSERT’s executed individually by the Database Engine.

A set, or batch, of records would be processed in this way, continually adding to the temporary files until all the records in the batch have been added. At this stage, the temporary disk files need to be passed up to the SQL engine for bulk loading. This is done by calling the ProcessAccumulatedData() function. The batch size can be any value (greater than zero), but the larger the batch size, the better the performance gain. For large import tasks in the millions of records, a batch size of 10,000 is reasonable. Any errors that occur during the loading of a single batch result in the whole batch being discarded, so this needs to be kept in mind if very large batch sizes are used.

View solution in original post

Trusted Contributor.. T0mTran Trusted Contributor..
Trusted Contributor..

Re: What are "Origins" and "TRIM Bulk Loader"?

Jump to solution

Thank a ton, Jason!

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.