Anonymous_User Absent Member.
Absent Member.
341 views

Custom CollectorTime Stamps


I have another question which hopefully you guys can answer. So I have
some events coming into my custom collector which have the date of the
event embedded in the message from the event source and I want to set an
appropriate field on the Sentinel Event to the date stored in the
message. I have tried a couple of approaches to setting this and it
seems no matter which way I do this the values I set never seem to make
it through to the event and on some occasions the event never seems to
appear either in the active views or in the sentinel event searches on
the web ui, however really looking for a pointer on where I might be
going wrong and where you guys recommend embedding this information.
Might be worth mentioning that I have also tried trusting the event
source time in ESM.


The ways in which I have tried to set this time information have been
the following:


Code:
--------------------

var dateString = <date from message>;
e.ObserverEventTime = dateString;

--------------------


I also tried setting the ObserverEventTimeString as I had read
somewhere that this is processed and in turn sets the
ObserverEventTime.

After reading 'Handling Dates and Times in Collectors'
(http://www.novell.com/developer/collector_dates.html) I created a
javascript date object from the date string in the raw event and passed
this to setDeviceEventTime() but still no luck in setting anything in
the event.


Code:
--------------------

e.setDeviceEventTime(new date(dateString));

--------------------


I guess what I am looking to find out is where do you recommend setting
the event date in this sort of situation? Do you have any code examples
of this working as I am confused as to why the event sometimes doesn't
appear in Sentinel yet the collector doesn't send an error etc.

Look forward to your replies and advice,

Thanks 🙂


--
alanforrest
------------------------------------------------------------------------
alanforrest's Profile: http://forums.novell.com/member.php?userid=90508
View this thread: http://forums.novell.com/showthread.php?t=453930

0 Likes
6 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Custom CollectorTime Stamps

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

First, let me mention what I think is the most-common point to miss with
time issues:

If the event's time is not current, accurate, and precise, and I mean
very close to the present (off by seconds, maybe minutes, but not hours)
events that come into the system will not show up in an Active View.
Why? Because Active Views show current (based on time) events. If your
event source, or your collector, or system time on a CM, or whatever is
off by a bunch then the events as seen by Sentinel will not belong in a
view of what is active. You mentioned that "some occasions the event
never seems to appear either in the active views or in the sentinel
event seaarches on the web ui" which made me write what I did above.
Pretend in those cases the event WAS getting set per your settings.
When would they show up? Is the event string you are feeding to the
Date class being properly-parsed? If you pass in something terrible
like 03-26-12 does it know that's M-D-y and y-D-M for the format?
Worded another way, does it know it's March and not December? Does it
know '12' means '2012'? If it did parse them properly and they were
events with dates a couple of hours ago you'd never seen them in an
Active View and you would need to modify your Web UI search to include
more than an hour's time for searching.

Anyway, start there. Using the Collector debugger you can see how your
Date-typed variable is interpreting your input. Also be sure your
machines are all in time sync.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPcODqAAoJEF+XTK08PnB5JnQQAJa2ki06lWWwgpgvAgrMmNdY
+SjjR9NPLfgWAOSJyeSCtvq+l0gyPJA9SsT9o3Vhcm92BLBZ0JrIisk4ltb0YY2z
7lhsEj3/Qqs5VMPkHKvz45hS80iIchRspgFCXnpo5FnzjspJ83b6in18sipAXmOa
u/S1TV9L7qLWUxne/Z51ZVtNg7YaAy5j4YnFZ7Vgw/5cK7a0v9oW2+t5kWFVMQXC
kuHDqTvwVuRQacDE4dk4tin25fdX6ZKvCliXzQr7hhhoWKWhCOn4fyPafQ+t78S4
MGT0+pxgpZrYwRL4fY4tN0zmXNGSoyCBGAgLVNqZ08Udlyy5kw0xT4maX0dD+Ac9
kSMjBLaA8hu9+hYNgJxSdbKwPJGyN1RQLdY5CjvuhXlapJkA97ObQ8M04U5MVx8c
rpTEJISz8ghUabw3JAP9s8enW0cuemuDxHmCSKU+rG4YMMfY3Gafc3DVknCynVKw
+RmjILI7C3aVnGk8YbLwl+Jc2pkXH51CpaO0tXaqa3M+CkC69iG1GCWLXk5pj/XC
ml71HblUoRAo0DXtyYu+GnAjnPRecx7egsGddOqk6fD4R0YN2naruqmorsIw2DUZ
UhvoLs6bahafZRc2Jy3BvwxXwcvi12+2wxLgRxybmbKz5AasajOwGeDCL6Vcyuab
SPRnilN0C8AVeAXtMEqa
=TsUi
-----END PGP SIGNATURE-----
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Custom CollectorTime Stamps

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Forgot to include the second-most-common point. 'DeviceEventTime' is
what is really important when you use Trust Event Source Time. If you
do not use Trust Event Source Time then the EventTime field on the event
is set to the CM's time. If you do use Trust Event Source Time then
EventTime is set to DeviceEventTime. EventTime is what is used for
EVERYTHING from here on out... Correlation rules, Active Views,reports
most of the time, the Web UI, etc.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJPcOPlAAoJEF+XTK08PnB5qOQP/1EmwDxC0GnzuIxJSC5ZivA0
ZIO+QJ3n1sJkU8EGjNuhGcOLeJQg3uFqVh5XXDoGnQh5DtaQsapu9WUseDbgotBS
zcOm2Z1/zkALYj7jhV8qFcnsQRtlVYA4onIJrPT7vP1JzKCq6ywSvVVzxy7DKyDn
7qJLUiMxebWJDpqDAnp6fM4o/8+V4Zx7ZOD65LDMIUrxOSnntv7VO0jFHj+xmLGm
lkF1YFVVkzRRiXI6JIVsiHbiXcHFU/I9cEXvNrZFqtWDAPNg1e84yO+g8FSn8G3h
X+HAfUpaHwWmU4U7yk2bQCgJuYlg/hb/QeUJWWih7HBI4CWENcEy72jA3JoYhbzs
CQL1HvizE8G/xPrwb7rZWJ3yYJ+00JMNCd3AUvG6Fqyiev05hWGH8ZOA3hMsrBtU
y1UputoE7pQmycPcJeNDSAqDb4rv87MWll0GjWhMZCaFxYzY0FhuPR5qxVaT9Kj4
kWcxqY07n6JhmQwG4VZayN7Zn0Txel0l4oAoElULTL/62m6apSwznky1cugrC0At
/A1i56j39E55PrKn/JNFnFoH/x2HT/l7VNSYC9l6AeUJfVmEtWRd/VcVuPS7noYr
NkoUDTXApvz/Vq+lfNri9/FjHJWdYjJFE5vlltYxFJCPmI8zd4whba1NKTEw8TxC
h3lWYcIoKi7KA1TEB35H
=zVjR
-----END PGP SIGNATURE-----
0 Likes
Highlighted
Anonymous_User Absent Member.
Absent Member.

Re: Custom CollectorTime Stamps


Thanks for the reply. Yeah I kind of expected it not to show in the
active view, however my event search was set with the correct target
date range so
it should have shown in that. So my events were off by a few hours and
my search criteria was the last week or so ( its a dev server with
nothing
else being fed to it).

My date string appears to be fed properly, when I take it and output it
to a customerVar for example I can see a parsed version of the date
string
passed in being returned to this but when I pass this to the
setDeviceEventTime method the events fail to appear in the web ui.


Code:
--------------------

this.newDate = new Date.parse(dateString);
e.setDeviceEventTime(this.newDate);

--------------------


What should I be passing to this method, see above for what is being
passed at present. The new date is definitely being created as I can see
this in my
customerVar if I set it and comment out the setDeviceEventTime method
call. I have not had much luck with the debugger, I added some break
points in on my code
and clicked the run button and it seems to hang or freeze during
running not sure if that could be to do with the java version im running
on my desktop? sometimes
I can step over or into code further than others if i stop and re-run
the debugger, seems strange. Are there any docs for the debugger?


Any pointers appreciated 🙂

Thanks


--
alanforrest
------------------------------------------------------------------------
alanforrest's Profile: http://forums.novell.com/member.php?userid=90508
View this thread: http://forums.novell.com/showthread.php?t=453930

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Custom CollectorTime Stamps


HI Alan,

So let's see.....

Overall I think you're pretty close on the timestamp thing, in your
first post the problem might well have been that you wrote 'new date()'
instead of 'new Date()'. But basically you're just about there - as
you've figured out, you need to construct a JavaScript Date object from
the parsed input from the event, then you need to pass that Date object
to the setDeviceEventTime() method. There are a couple tips, however,
which I think are covered on that page you found, but which I'll clarify
here.

1) When you use the Date constructor to create your JS Date object,
you're passing a timestamp string. That string has to contain the year,
month, day, hours, minutes, and seconds. The Date constructor
understands a couple widely-used formats for strings, but this can
easily be affected by various problems like locale, weird abbreviations,
etc. There are basically two approaches to fixing this problem - the
first approach is to parse out the individual fields and reconstruct
them in a form that the normal Date constructor will understand, and the
second approach is to use the 'date.js' library's 'parseExact()' method,
and pass a format string (this method is easier, but somewhat slow).
Here's an example:

Date.parseExact("10/15/2004 12:02:23", "M/d/yyyy HH:mm:ss");

The format string here is actually the same as Java's SimpleDateFormat;
check out 'APIDocumentation - datejs - Date class and instance methods
provided by the Datejs library. - A JavaScript Date Library - Google
Project Hosting' (http://code.google.com/p/datejs/wiki/APIDocumentation)
and search for the parseExact method.

Alternatively, you can use a regex or something to parse your input
into individual fields and then reconstruct them into a more normalized
form:
rec.inputDate = "2012,02,23 12.34.12"; // weird input format
/(\d+),(\d+),(\d+) (\d+).(\d+).(\d+)/.test(rec.inputDate); // pass the
date through a regex
rec.normalizedTime = RegExp.$1 + "-" + RegExp.$2 + "-" + RegExp.$3 +
"T" + RegExp.$4 + ":" RegExp.$5 + ":" + RegExp.$6;
// Results in "2012-02-23T12:34:12" which is ISO8601 compliant (note
that I'm not 100% sure our version of JS supports ISO8601 yet, but you
can use RFC2822 in a pinch, and when we upgrade to the latest Rhino
ISO8601 will work
e.setDeviceEventTime(new Date(rec.normalizedTime));

2) The time you set here must NOT include any timezone information. The
problem here is that JS always uses local time, but that local time will
be according to the Collector Manager, which may not be the same as your
event source. Sentinel uses UTC. So the solution here is to pass
whatever local time format you get from your source to
setDeviceEventTime(), and then set the rec.s_TimeZone variable according
to what timezone the event source actually lives in. There's about a 80%
likelihood that the reason you can't see the events coming from the
Collector is because Sentinel thinks they happened 6 hours ago.

For testing, *always* do not trust the event source time. This means
that Sentinel will stamp the event with whatever the time was at the
time the Collector Manager processed the event, which might not be
correct but which means that your event will be sequenced fairly
recently so you can search for it. Then you can look at the
DeviceEventTime (or ObserverEventTime, in more recent schemas) and see
what time your Collector parsing thought the event occurred - if that
value doesn't match the actual event time, then you have a problem with
your parsing.
Do NOT set DeviceEventTimeString (ObserverEventTimeString), in general,
that will be automatically set by the template. It certainly is not
parsed by anything.

Now, how does rec.s_TimeZone get set? Several ways:
1) You can parse it out of your event data
2) If the event data doesn't include a timezone, you can right-click on
the Event Source node in ESM and set the TZ for that source there.
Whatever you set there will automatically show up as rec.s_TimeZone.
3) Note that if your event source reports UTC time, but actually lives
in a different timezone, then recent SDKs allow you to set the
instance.CONFIG.params.reportsUTC flag which tells the Collector NOT to
perform its automatic timezone shifting, but to report the local TZ. For
Sentinel 7 this allows the ObserverTZ* fields to be set correctly.

3) Debugging tips:
'Collector Troubleshooting and Debugging'
(http://www.novell.com/developer/collector_debug.html)
- OK, so here's what I always do:
a) Start the debugger. Hit "Play" once to load the code into the
debugger but do NOT start the Collector
b) Switch back to ESM and start the Event Source (otherwise you don't
get any data to parse)
c) Scroll all the way to the bottom of the code and find the line,
maybe 20 lines up, that says something about scriptInit(). Add a
breakpoint there.
d) Hit Play again to run the Collector.
- If after a few seconds I don't get a little yellow arrow on my
breakpoint, then I must have a syntax error in my code - no
Collector-specific code has yet been executed, but the various methods
have been initialized and syntax errors can cause the debugger to stop.
e) Hit the "Step Over" button several times until I'm in my main loop
and pointing at the line that says 'rec = conn.read()'
f) Hit "Step Over" again
- If the Collector doesn't come back immediately at this point, it
means I didn't get any data from the Connector - check that the Event
Source is started and has data to deliver.
g) Hit Step Over again until I'm pointing at the line that says
preParse(), then hit Step Into to drop into that method.
- If after I drop into the preParse() (or any 'rec' member method) I
see that CONNECTION_MODE is 'NODATA', this is the same problem as the
last step - I didn't get any data
h) At this point I can examine my input variables and step through the
code that I wrote.

HTH, let us know if you have any additional issues.


--
DCorlette
------------------------------------------------------------------------
DCorlette's Profile: http://forums.novell.com/member.php?userid=4437
View this thread: http://forums.novell.com/showthread.php?t=453930

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Custom CollectorTime Stamps


Hi David,

Thanks for the reply. I solved this by disabling the trust event time
source and it appears to be setting the observerEventTime which is great
🙂 So I now have both the recieved/event time and the time from the
event source in my event output which is exactly what I was after 😄 In
theory these times should be the same give or take a few secs but there
are a number of situations where I think there could be hours or days
apart so that is why I was keen to get both times in the event. As it
stands my test data for the event source is from November so it is
perfect for testing that part 😉

Also the tips for using the debugger are really handy, I think that's
where I have been going wrong as I assumed it would start the event
source when it was running I didn't realise I had to manually start
this.

Thanks Again Guys 😄


--
alanforrest
------------------------------------------------------------------------
alanforrest's Profile: http://forums.novell.com/member.php?userid=90508
View this thread: http://forums.novell.com/showthread.php?t=453930

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Custom CollectorTime Stamps


Hi Alan,

Let me explain one more thing then: the time parsed out of the event
source data is always placed into ObserverEventTime. The time that the
Collector Manager received and processed the event is always placed into
SentinelProcessTime. What the "Trust Event Source Time" option does is
allows you to decide whether you trust the event source time or whether
you want to use the time the event was processed as a proxy for the
event time. In the former case, ObserverEventTime is copied to
EventTime; in the latter, SentinelProcessTime is used.

This is important because the EventTime field is the field that
Sentinel uses to sequence the event relative to other events in the
stream. This includes things like what order it will show up in
searches, which partition it will be placed in, how the correlation
engine will sequence it, etc.

If it's not obvious, if you trust the source time and use old replay
data, then Sentinel will think it's an old stale event, put it in an old
partition, not send it to correlation (which is real-time only, with a
30s re-order buffer), etc.

But no matter how you set "trust...", you still always have both
timestamps in the event in ObserverEventTime and SentinelProcessTime.

HTH


--
DCorlette
------------------------------------------------------------------------
DCorlette's Profile: http://forums.novell.com/member.php?userid=4437
View this thread: http://forums.novell.com/showthread.php?t=453930

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.