Anonymous_User Absent Member.
Absent Member.
429 views

Best configuration for custom database collector


Hi,

I have created a custom oracle database collector.
This collector is used to connect to about 20 databases.
My question is what are the pros and cons for the different
configuration options.
To clarify, in my opinion there are 3 ways to configure the
components:
1) 1 collector -> 1 connector -> multiple event sources
2) 1 collector -> multiple connectors -> multiple event sources
3) multiple collectors -> multiple connectors -> multiple event
sources

If I understand correctly option 1 is not done because the connector
will hold the offset for the database.
But what about the other options, which one is preferred?

Thanks in advance


--
jcvader1
------------------------------------------------------------------------
jcvader1's Profile: https://forums.netiq.com/member.php?userid=502
View this thread: https://forums.netiq.com/showthread.php?t=42642

0 Likes
5 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Best configuration for custom database collector

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I believe with the Database connector the only supported option is #3
with a one-to-one-to-one relationship between collector, connector, and
event source. This is because the collector handles a lot of specifics
that normally do not need to be tied to the collector with other types
of event sources. The UI may let you do otherwise, but that's a bug in
the UI unless something has changed recently.

The downside to this is time taken to set up multiple collector and
connectors (you'd be setting up multiple event sources regardless). The
upside is that you are not limited by the amount of events that a single
collector can process, which is usually the bottleneck for any type of
collection. A collector runs its own thread, and that one thread can
utilize a whole processor, but if you have a big system with multiple
processors/cores/etc. then you're not going to fully utilize it with a
single collector as well as you could with n-1 collectors (where 'n' is
the number of processors) or higher.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQZLfqAAoJEF+XTK08PnB5gBcP/2Tfy5ACqA213+CqV5rJoJBe
aQvDbNcFOPPVPEWmU0P8DHm+natlx8BsjQO0IeXXihzFwuK76QJY9bbwTfPxzUTO
L8dvwYpYsT3kAsp8GBR1GStU4saAyjtFSt2ilICzStcVlcRnpRzy4b0w3rAgfc2U
gGQIeakgS5lhyhR6YVct5QZIHZ9g7IyuuMl3HgkgAo6ce0AkZXw0j5xR+LFesCxx
BOp2zD1GoYiH81Tz0oelQJmqYWHnqIBsxvxS6/q/ovkMgbmSLMOnHWrC4DHg/6Iq
DFWvuIkWTQSnVRPNDXfYoHtowxp4o4asrI0KOcs5Y+K4E0vZ+ILE/J3cTB8REjrA
+D6njqopYfqp/ooOYXbmjKyfmbHYRKIt5ndo7hhHagcJh1ZIJsjYBbyJ3Jh+N6MS
2sxGsLS86PjfHL2WNTRcIkWm0cTgaX3jFw2wNY6Vp0SudO2lq+Hrnxcvf5aUaC23
QAL70t7kgMePVhqtC4uF8x4soXNgXEp6RLAHpWaLIEDQcSSD1vgmB0dOjdOZj83s
rZbxGRPH2DiuaqeVoHQ2M6nZI5DLQBloCGUIG8+Iv7nQBduO0EhUa+4dw3SmsOx8
UPoC/ltxTvDGW00Yz1nJ0MmiQ0Q8kWc2wM5gi1znJQ9tMtIoi4/qaVcTNhEg89b+
dJ16+igUdxw1+kwl/qfF
=cVZe
-----END PGP SIGNATURE-----
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Best configuration for custom database collector


Thanks Aaron, David,

I am using the latest SDK, in fact I have 2 different queries in the
collector as well.

ab;201956 Wrote:
> The upside is that you are not limited by the amount of events that a
> single
> collector can process, which is usually the bottleneck for any type of
> collection.

Realizing the difficulty in answering the next question, I am going to
ask anyway.
Are there some astimates in how much events a collector can process.
I realize this has to do with the type of collector and probably the
type of processor, possibly memory etc.

The problem is I have only 1 server (4 cores) at the moment and I am
pushing sentinel/the server really to it's limits.
I noticed I loose events when pushing syslog to its limit (I received a
beta syslog collector that is doing better) especially when using the
Oracle RDD link also.
I'd like some guidlines to when you should use 2 collectors in stead of
1 or how you can investigate what is your bottleneck.
I hope I am not asking to much but I think it is important to know the
limitations of your configuration/system and I can't remember reading
anything in the documentation about a limit in eps for a collector.

Thanks in advance,
Anco


--
jcvader1
------------------------------------------------------------------------
jcvader1's Profile: https://forums.netiq.com/member.php?userid=502
View this thread: https://forums.netiq.com/showthread.php?t=42642

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Best configuration for custom database collector

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is probably a question better-asked in the Plug-In SDK forum which
maybe over on the Novell forums site. Anyway, a few data points that
may or may not be valid.

First, collector speed depends on processor primarily. The better the
processor, the faster the collector. If a collector isn't fighting with
other collectors' threads for a single processor/core then that'll help.
Having four processors my personal way of gauging things is one of
those processors manages things overall, and then three are left for
other threads like collectors. If you have more collectors than that
they'll be sharing the processors, which isn't a big deal when you have
dozens of events per second, or maybe even one hundred EPS per
collector. On the other hand, having your EPS up in the several-hundred
per collector means you're probably approaching some limits, but what
those limits are will depend entirely on your system. At one point in
time, long ago, there were understood limits of around 300 EPS per
collector, but I believe that was before the JavaScript language was
used (and JavaScript/ECMAScript is much faster). Also there are current
improvements in the base collector logic being put in place so newer
collectors should even be faster than their older ECMAScript
counterparts. Also factor in odd things like if you're using the
Generic Hostname Resolution collector, which you should NOT be using in
any circumstance (in my opinion) as it kills performance and, arguably,
provides little benefit in the process.

Anyway, it depends. If you are trying to get 10k EPS per collector, you
are above what is possible. If you're trying to do 1k EPS, it probably
depends on what the collector is doing, and every little bit of
performance you can get out by using efficient code will help a lot,
maybe even make something not possible become possible. If you're below
1k events coming in per second for a collector it may still not be
possible depending on the size of events or the realities of parsing
your events, but who knows. You could always create a collector with no
real parsing logic that either throws events away (to see how quickly
the rest of the system can crank through events without parsing) or you
could create a collector that simply takes the event, gives it a generic
and static name, throws the data into the 'Message' field, and then
moves on (minimal parsing) to see how fast that is. Those should give
you some high numbers from which you will decrease performance by adding
more functionality.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://www.enigmail.net/

iQIcBAEBAgAGBQJQbwUfAAoJEF+XTK08PnB5sfoP/0osbmH8jqhnYLdHf2c+ZuyQ
gkHhjV112MG/NK5e6qrcJALkcC67WlwdIfeczOoDUnXiF4rHr03RqkJOC0rWnrrK
TAdW1OfiXkUfuCO3VyZ/cviK+CH1dA6GEOzjiah8jJfODafCg+nBD6dRtg+ojBsZ
/OnTpLvmMuh0/z5DU5gCPk2C4GUfc9gkJXhHw0MHBfOfYY2qitwmJ/DbkiRkCYOZ
DSCFHpYp3nG7TP3MjQclKH3bMGlRrsK7fO04kU8ifzOYvbvUInUHkZ2UOFtEO3Xk
iiSdkOFobex3buTvlHvFZFzWK7OwB6ebRFkLzfRGNgpvRNUG8+WrdGo+5yGeapw6
OhdNsZE6vlInEJTM6j6FZX92UBxOcsmxrpxiJeHRxgMo3F8w3J15HmzJW+eNjucs
R1erXHi34B2p+y9chnHrlYQSrvx8P8lV5iKomFdAF4Zbhd3drGEJke66Iol8z0l9
PHmUhSFYidMhJPX+vBP5lu0rRKe03WL81CUB436wcr7h94uW4nzBg9bVt2xscQHM
UrXbIXxUtTzOGv92PFRq0SSglhzoDjo/QvBbZsSE44DJrKHJboIaRP9ifqf+Y0ku
KDjAJ5IL5Kv0JDqSVStKokN2DDo7vqXxv6fNkkprwYVfbHsGJXhS59otAfvS1Mpx
LCgG7l6dLwV2gITy71Cg
=TZL4
-----END PGP SIGNATURE-----
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Best configuration for custom database collector


That is information I can work with.
Very enlightening.
Thanks for your answer.


--
jcvader1
------------------------------------------------------------------------
jcvader1's Profile: https://forums.netiq.com/member.php?userid=502
View this thread: https://forums.netiq.com/showthread.php?t=42642

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Best configuration for custom database collector


In more recent versions of the SDK, the Collector stores the database
offset relative to the Event Source, not the Connector, so in theory you
can select option (1) for management simplicity.

That said, this feature has not been tested extensively and you may or
may not be using the latest latest SDK (I think this is post the
official 2011.1 release), so you should carefully test this option
before selecting it.

As Aaron pointed out, there are also other considerations such as
performance, etc.


--
DCorlette
------------------------------------------------------------------------
DCorlette's Profile: https://forums.netiq.com/member.php?userid=323
View this thread: https://forums.netiq.com/showthread.php?t=42642

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.