Highlighted
Frequent Contributor.. ArcSight Newbie Frequent Contributor..
Frequent Contributor..
468 views

Flexconnector file text.qualifier is not correct as expected as it should be

Hi,

 

I got some incorrect parsing about a delimiter 

Let’s say, I have this type of log

1;2;3;4;5;6 and i use a delimiter ; to separate like this

----------------------
delimiter=;
text.qualifier="
trim.tokens=true
contains.empty.tokens=true
token.count=6

token[0].name=USR_KEY
token[0].type=String
token[1].name=ACC_KEY
token[1].type=String
token[2].name=USR_LAST_NAME
token[2].type=String
token[3].name=USR_FIRST_NAME
token[3].type=String
token[4].name=USR_MIDDLE_NAME
token[4].type=String
token[5].name=USR_DISPLAY_NAME
token[5].type=String

event.deviceCustomString1=USR_KEY
event.deviceCustomString2=ACC_KEY
event.deviceCustomString3=USR_LAST_NAME
event.deviceCustomString4=USR_FIRST_NAME
event.deviceCustomString5=USR_MIDDLE_NAME
event.deviceCustomString6=USR_DISPLAY_NAME
----------------------

I will get the correct parsing result as

event.deviceCustomString1=1
event.deviceCustomString2=2
event.deviceCustomString3=3
event.deviceCustomString4=4
event.deviceCustomString5=5
event.deviceCustomString6=6

If my log is 1;2;3;"4";5;6 i will get the incorrect result as

event.deviceCustomString1=1;2;3;”4”;5;6

I took a look into the agent.log and found the error -- Found less tokens than expected [1<6] for input ["1;2;3;""4"";5;6"]

I tried googling but it didn't seem to have any conclusion/solution to this even i tried multiple versions of connector, 7.6/7.9/7.10. i still got the same incorrect result.

Any ideas or workaround to solve this ?

 

Thank you,

Kumpon

Tags (1)
0 Likes
5 Replies
Honored Contributor.. dkuehner Honored Contributor..
Honored Contributor..

Re: Flexconnector file text.qualifier is not correct as expected as it should be

It´s weird that there are " around the whole message as well as double "" around the 4. No idea where those come from. The result it seems is, that the " surounding the whole message are used as text qualifiers and therefor everything is seen as one token. Could you try a different text qualifier please?

But I have several connectors working with the qualifier "... so it does work in general.

0 Likes
Frequent Contributor.. ArcSight Newbie Frequent Contributor..
Frequent Contributor..

Re: Flexconnector file text.qualifier is not correct as expected as it should be

Yes, it is weird. It takes the whole message as one token rather than a separated token.

 

I did try another delimiter |  and the result was still the same with one token holding all the message.

 

Right now, i don't want to waste time and start using Flexcon Regex to parse this but will be appreciated if anyone can find a solution for this.

 

P.S. I installed Flexcon File on Windows8.1 64-bit.

 

Kumpon

0 Likes
Frequent Contributor.. ArcSight Newbie Frequent Contributor..
Frequent Contributor..

Re: Flexconnector file text.qualifier is not correct as expected as it should be

I know why i had an error like this.

 

Basically it's about the log format. I will explain you here

 

1. Original log is in DSV file (Delimeter Separated Values) and the format of the log looks like this if openned with Microsoft Excel

1;2;3;"4";5;6

2. But the log format will change if openned with Notepad

""1";"2";"3";""4"";"5";"6""

 

As you can see the difference, the second log format has a double quote added for each one. Besides, the first double quote " infront of 1 and the last double after 6 " is like an opening and the clsoing of the whole message and that's the reason why i see the error log in agent.log as 

 

Found less tokens than expected [1<6] for input ["1;2;3;""4"";5;6"]

 

I am using Flexcon regex to solve this.

 

Thank you,

Kumpon

0 Likes
Honored Contributor.. dkuehner Honored Contributor..
Honored Contributor..

Re: Flexconnector file text.qualifier is not correct as expected as it should be

Hi,

maybe you should use another editor then?

0 Likes
Frequent Contributor.. ArcSight Newbie Frequent Contributor..
Frequent Contributor..

Re: Flexconnector file text.qualifier is not correct as expected as it should be

Yes,

 

I am working on Flexconnector Regex File to solve it and it is working well.

 

Thanks,

Kumpon

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.