CEF/ ESM Schema extention to support cloud solutions

[ysergeev]

CEF standard was developed ages ago. Nowadays, there are so many cloud-based log sources which require additional fields. Most of the time we use custom/flex strings, but the number of those are limited and it becomes a mess to maintain all of these across the board. It is time to consider to issue a new CEF version.

When we consume AWS, GCP, Azure logs we have such entities as:

- account id | tenant id (I mean AWS account id, Google Cloud org Id, or other unique identifier.)

- availability zone

- storage bucket (S3, storage, blob)

- container name/id

 - instance name (which may be not the same  as deviceHostName or destinationHostName)

 - instance id

- region

 - user type (root/ iam )

None of these have been properly represented in the schema since the cloud era began to spread. Therefore, random fields like fileId (access key ID) or filePath (full user path) are used to store some of the IDs or role names and so on.

[dalesio]

The ArcSight ecosystem is moving towards having AVRO as it base schema format.  AVRO has many features that neither ESM_Binary nor CEF currently support, such as schema flexibility and schema registry to ensure compatibility with different schema versions.

The July 2020 release will be the first release to embrace AVRO and future releases will continue to enhance this technology.

[Karl2]

Dear @dalesio , 

Is such movement going to enhance the current CEF and use AVRO as the schema format?

I do think that makes sense, but what about also providing the CEF Schema Registry that you can use in conjunction with any Kafka Cluster AND with the ArcSight Products?

https://community.microfocus.com/t5/ArcSight-Idea-Exchange/CEF-Schema-Registry/idi-p/

Regards,

Karl.