Highlighted
Micro Focus Expert
Micro Focus Expert
454 views

<(RM) Support tip> Ryan Winston's optimized IDOL config template

I use the following as my template to optimize IDOL.  The comments state how to use certain settings.

With the IDOL Service Config, I update the timeouts, setting Timeout=600 and EngineTimeout=420.  I typically set the Threads=4 .  I add IndexThreadPriority=Lower to the Server section.

Content Service (comments provide details):

//---------------------------- Service Settings -------------------------------//
[Service]
ServicePort=9102
ServiceStatusClients=*.*.*.*
ServiceControlClients=*.*.*.*

//---------------------------- Server Settings --------------------------------//
[Server]
Port=9100
IndexPort=9101

QueryClients=*.*.*.*
AdminClients=*.*.*.*
IndexClients=*.*.*.*

Threads=4
// HP (RW) - Match threads to Total cores - 1 as a start. Threads all depends on what the server can handle. Total max threads needs to be divided across config files.
// e.g.; a server with 20 threads may be able to handle 30. So, that 30 needs to be divided by total number of IDOL functions running.  Default would have 10 a piece.
TotalResultsPrintDatabases=true
// HP (RW) - enables Document Content searching to work in Single Engine config
IndexThreadPriority=Lower
// HP (RW) - important when Index & Query are on the same machine.  Lower means Query gets priority over Indexing

TermSize=20  
// HP (RW) - TermSize cut to 20 to improve search efficiency. Is the default.
DefaultQueryOperator=AND
// HP (RW) - AND search more efficient than default OR;
AdvancedPlus=TRUE
// This option also enables terms with agent weights to be matched through phrase searches or proximity operators.
DeleteAfterAdd=TRUE
// deletes IDX files after they have been indexed into IDOL Server
SplitNumbers=FALSE

RepositoryStorage=TRUE
// stores data into a single nodetable
NodeTableCompression=lz4
// Allows you to compress the data in the nodetable directory, which reduces the IDOL Server disk footprint.
NodeTableSecureDelete=true
// In secure delete mode, content is permanently overwritten when the delete action is processed, rather than when a DRECOMPACT index action runs.
NodeTableStoreACLsInRefTable=true
// store ACL fields in the reference nodetable, as well as the nodetable. This option reduces the time it takes IDOL Server to look up ACLs, but slightly increases the storage space that IDOL Server requires.

MaxInputString=-1
MaxQueryTerms=-1
MaxFileUploadSize=-1
MaxResults=100000
QueryTimeoutInMilliseconds=600000
// HP (RW) - 10 mins; 5 mins when used alongside other Content Engines

// HP (RW) - Analyze terms: action=TermGetAll&MaxTerms=10000&termanalysis=true
// note: single instance words would not be searched for wildcards. If needed, set to 1
UnstemmedMinDocOccs=2
// The minimum number of documents that a term must appear in to be a spell check suggestion or to be matched by a wildcard term.
UnstemmedIndexNumbers=true
// Whether to add numeric terms to the unstemmed index
UnstemmedIndexNumbers0MaxLength=20
// The maximum length that unstemmed terms that do not contain any numeric values can contain.
UnstemmedIndexNumbers1MaxLength=9
// The maximum length that unstemmed purely numeric terms can contain.
UnstemmedIndexNumbers2MaxLength=9
// The maximum length that unstemmed alphanumeric terms can contain.

IndexNumbers1TruncateLength=6
// The maximum length that a purely numeric term can be. Purely numeric terms that are longer than this value are truncated.
IndexNumbers2TruncateLength=6
// The maximum length that an alphanumeric term can be. Alphanumeric terms that are longer than this value are truncated.
//IndexNumbersMaxValue=12
// The maximum value that a number can have and still be indexed into a database. Numbers exceeding this value are stored but not indexed.

IndexFieldCheckSum=TRUE

DelayedSync=true
// HP (RW) - for small customers who need near instant searching, DelayedSync can be set to false after the initial bulk reindex is complete (will sync after each dreadd)
MaxSyncDelay=28800
// HP (RW) - 8 hours for day-to-day
// MaxSyncDelay designates the maximum number of seconds that will elapse before IDOL must write to
// disk. For large indexing jobs, this should be set to something relatively large. IDOL will write
// to disk either when the time is reached or when the IndexCacheMaxSize is filled.

DateFormatCSVs=YYYY-MM-DD#SHH:MM:SS,YYYY-MM-DD,YYYYMMDDHHMMSS,MM/DD/YYYY,DD/MM/YYYY,MM/DD/YY,DD/MM/YY,AUTNDATE
KillDuplicates=*/DREREFERENCE
DocumentDelimiterCSVs=*/DOCUMENT
MinFreeSpaceMB=1024
CombineIgnoreMissingValue=true

[TermCache]
TermCacheMaxSize=0

[IndexCache]
// When performing large indexing jobs, the more processing that can be done in memory, the better
// the overall performance. IndexCacheMaxSize designates how much memory (in KB) is allocated to index time
// processing. When the memory buffer is filled, IDOL will write to disk.
// IndexCacheMaxSize=2097152 // 2 GB max

IndexCacheMaxSize=819200
// HP (RW) -  800 MB works as a happy medium for both day to day & bulk reindex

[SectionBreaking]
MinFieldLength=9000000
MaxSectionLength=900000000

//---------------------------- Path Settings ----------------------------------//
[Paths]
BitFieldPath=C:\HPRM\IDOL\TRIM Content Service 1\bitfield
DyntermPath=C:\HPRM\IDOL\TRIM Content Service 1\dynterm
IndexQueuePath=C:\HPRM\IDOL\TRIM Content Service 1\indexqueue
IndexTempPath=C:\HPRM\IDOL\TRIM Content Service 1\indextmp
MainPath=C:\HPRM\IDOL\TRIM Content Service 1\main
NodetablePath=C:\HPRM\IDOL\TRIM Content Service 1\nodetable
NumericPath=C:\HPRM\IDOL\TRIM Content Service 1\numeric
RefIndexPath=C:\HPRM\IDOL\TRIM Content Service 1\refindex
SecIndexPath=C:\HPRM\IDOL\TRIM Content Service 1\secindex
SortfieldPath=C:\HPRM\IDOL\TRIM Content Service 1\sortfield
StatusPath=C:\HPRM\IDOL\TRIM Content Service 1\status
StatePath=C:\HPRM\IDOL\TRIM Content Service 1\storedstate
TagPath=C:\HPRM\IDOL\TRIM Content Service 1\tagindex
LanguageDirectory=..\shared\langfiles

[Schedule]
Compact=true
CompactInterval=168:00
CompactTime=23:00
PreCompactionBackup=false
//(11 PM, 24 hour format, every 168 hours/once a week)
// HP (RW) - manual command should be run after a reindex if any search terms don't come back in testing;
// http://localhost:9001/DRECOMPACT?priority=100 or http://localhost:9101/DRECOMPACT?priority=100 if using Single Engine.

//---------------------------- Logging Settings -------------------------------//
[Logging]
LogDirectory=C:\HPRM\IDOL\TRIM Content Service 1\logs
LogTime=TRUE
LogOutputLogLevel=TRUE
LogLevel=Normal
LogExpireAction=Compress
LogOldAction=delete
LogMaxOldFiles=10
MaxLogSizeKbs=20480
0=ApplicationLogStream
1=QueryLogStream
2=IndexLogStream
3=ActionLogStream

[ApplicationLogStream]
LogFile=application.log
LogTypeCSVs=application

[QueryLogStream]
LogFile=query.log
LogTypeCSVs=query

[IndexLogStream]
LogFile=index.log
LogTypeCSVs=index

[ActionLogStream]
LogFile=action.log
LogTypeCSVs=action

//---------------------------- Field Processing -------------------------------//
[FieldProcessing]
0=SetIndexFields
1=SetDateFields
2=SetDatabaseFields
3=SetTitleFields
4=SetHighlightFields
5=SetReferenceFields
6=SetRelatedDocsReferenceFields
7=SetNumericDatefields
8=SetOtherReferenceFields

[SetIndexFields]
Property=IndexFields
PropertyFieldCSVs=*/DRECONTENT,*/DRETITLE,*/TS_TITLE,*/TS_OWNERLOCATION,*/TS_EXTENSION,*/TS_CONTACT,*/TS_NOTES

[SetDateFields]
Property=DateFields
PropertyFieldCSVs=*/DREDATE

[SetDatabaseFields]
Property=DatabaseFields
PropertyFieldCSVs=*/DREDBNAME

[SetTitleFields]
Property=TitleFields
PropertyFieldCSVs=*/DRETITLE

[SetHighlightFields]
Property=HighlightFields
PropertyFieldCSVs=*/DRETITLE,*/DRECONTENT

[SetReferenceFields]
Property=ReferenceFields
PropertyFieldCSVs=*/DREREFERENCE

[SetRelatedDocsReferenceFields]
Property=RelatedDocsReferenceFields
PropertyFieldCSVs=*/DREROOTFAMILYREFERENCE

[SetNumericDatefields]
Property=NumericDateFields
PropertyFieldCSVs=*/TD_DATECREATED,*/TD_DATEREGISTERED

[SetOtherReferenceFields]
Property=OtherReferenceFields
PropertyFieldCSVs=*/TN_URI,*/TS_NUMBER

//---------------------------Properties-------------------------------//
[IndexFields]
Index=TRUE

[DateFields]
DateType=TRUE

[DatabaseFields]
DatabaseType=TRUE

[TitleFields]
TitleType=TRUE

[ReferenceFields]
ReferenceType=TRUE
TrimSpaces=TRUE

[HighlightFields]
HighlightType=TRUE

[RelatedDocsReferenceFields]
ReferenceType=TRUE
TrimSpaces=TRUE

[NumericDateFields]
NumericDateType=TRUE

[OtherReferenceFields]
ReferenceType=TRUE
TrimSpaces=TRUE

//---------------------------- Language Types ---------------------------------//
[LanguageTypes]
LangDetectUTF8=TRUE
DefaultLanguageType=englishUTF8
DefaultEncoding=UTF8
LanguageDirectory=..\shared\langfiles
GenericStemming=true
GenericTransliteration=true
StopWordIndex=0
0=English

[English]
Encodings=UTF8:englishUTF8
Stoplist=international_Xlarge.dat
// HP (RW) - RW's massive 6000 term list to reduce overall terms
Stemming=TRUE
// HP (RW) - further stemming applied specifically to English
IndexNumbers=1
// HP (RW) - 0=Do not index numbers. 1=Index all numbers. 2=Index numbers only if they are part of a word (alpha-numeric)
HyphenChars=NONE
AugmentSeparators=-–—@“”‘’_.…
// word-separator characters

//---------------------------- Databases --------------------------------------//
[Databases]
NumDBs=1
0=HPRM_82

[HPRM_82]
DatabaseReadOnly=FALSE
Internal=FALSE
InvertedAgent=FALSE

2 Replies
Super Contributor.. Erik Wold Super Contributor..
Super Contributor..

Re: <(RM) Support tip> Ryan Winston's optimized IDOL config template

MaxFileUploadSize in the IDOL config allows for a "-1" setting which does not limit the size of files to index.

CM has a max setting of 2048MB in both Enterprise Studio and the client Reindex options, and there appears to be no way to allow a size greater.

How would you try to allow CM to index a file that sits in the realm of 2048 to unlimited?

Micro Focus Expert
Micro Focus Expert

Re: <(RM) Support tip> Ryan Winston's optimized IDOL config template

Erik, 

You could try modifying the TRIMConfig.tcfg manually to increase it past the UI's max. This can be done with other things, so let me know how it goes.

 

Thanks,
Ryan Winston

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.