Future versions of ElasticSearch will require Java 11.

The current ES version works just fine with Java 8:

novell   22175  4.7 12.8 4543824 913768 ?      Sl   Oct05 2125:36 /opt/novell/sentinel/jdk/jre/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,JRE -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.io.tmpdir=/tmp/elasticsearch-4956161078137255192 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:ErrorFile=logs/hs_err_pid%p.log -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Xms512m -Xmx512m -XX:MaxDirectMemorySize=268435456 -Des.path.home=/opt/novell/sentinel/3rdparty/elasticsearch -Des.path.conf=/opt/novell/sentinel/3rdparty/elasticsearch/config -Des.distribution.flavor=default -Des.distribution.type=tar -Des.bundled_jdk=true -cp /opt/novell/sentinel/3rdparty/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -Epath.data=/var/opt/novell/sentinel/3rdparty/elasticsearch/data -Epath.logs=/var/opt/novell/sentinel/log
novell   22233  0.0  0.0  45972  1732 ?        Sl   Oct05   0:00  \_ /opt/novell/sentinel/3rdparty/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Hi Wencheng, in addition to what Norbert provided...


I thought you would raise this with technical support.


Anyway, first of all my apologies:
www.microfocus.com/.../s85system-requirements.html

Data indexing

    Elasticsearch 7.7.0

    Download URL: www.elastic.co/.../elasticsearch-7-7-0

...this is wrong. Once you're on Sentinel 8.5, the ES rpm is upgraded to 7.10.2. So there was no reason to downgrade that to 7.7.0.
We actually corrected this link, so not sure why it reverted back to 7.7.0. I'm checking it internally, and get it changed.

And this will be updated as well:
www.microfocus.com/.../t45h61q5z43n.html

This is correct:
Elasticsearch 7.10.2

Download URL: www.elastic.co/.../elasticsearch-7-10-2


Once you upgrade your Sentinel appliance to 8.5, it will, should use ES rpm -> "version":"7.10.2".

I executed this, on my Sentinel 8.5 install:
novell@xxx:/opt/novell/sentinel/bin> ./elasticsearchRestClient.sh x.x.x.x 9200 GET /_nodes/process
Status Code: 200
{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"elasticsearch","nodes":{"_9-1lhWzRPafyVUkGf6Xcg":{"name":"xxx","transport_address":"x.x.x.x:9300","host":"x.x.x.x","ip":"x.x.x.x","version":"7.10.2","build_flavor":"default","build_type":"tar","build_hash":"747e1cc71def077253878a59143c1f785afa92b9","roles":["data","data_cold","data_content","data_hot","data_warm","ingest","master","ml","remote_cluster_client","transform"],"attributes":{"ml.machine_memory":"21039345664","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"20"},"process":{"refresh_interval_in_millis":1000,"id":3554,"mlockall":false}}}}

...it seems ES is now -> "version":"7.10.2".

I only have the single machine in my ES cluster, so one sentinel core server.

x.x.x.x -> ip address sentinel server (run this on core server, and external node, but using the associated ip address of that machine)


Sentinel 8.4 is still using 7.7.0:
novell@xxx:/opt/novell/sentinel/bin> ./elasticsearchRestClient.sh 127.0.0.1 9200 GET /_nodes/process
Status Code: 200
{"_nodes":{"total":1,"successful":1,"failed":0},"cluster_name":"elasticsearch","nodes":{"0E8sZ3whSNO8CTDLxn6USA":{"name":"xxx","transport_address":"127.0.0.1:9300","host":"127.0.0.1","ip":"127.0.0.1","version":"7.7.0","build_flavor":"default","build_type":"tar","build_hash":"81a1e9eda8e6183f5237786246f6dced26a10eaf","roles":["master","ingest","transform","data","remote_cluster_client","ml"],"attributes":{"ml.machine_memory":"16810741760","xpack.installed":"true","transform.node":"true","ml.max_open_jobs":"20"},"process":{"refresh_interval_in_millis":1000,"id":3598,"mlockall":false}}}}


One of our customers was getting this kibana error, since upgrade:
{"type":"log","@timestamp":"2021-09-26T15:17:42Z","tags":["warning","plugins","licensing"],"pid":20914,"message":"License information could not be obtained from Elasticsearch due to [illegal_argument_exception] request [/_xpack] contains unrecognized parameter: [accept_enterprise] :: {\"path\":\"/_xpack?accept_enterprise=true\",\"statusCode\":400,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"request [/_xpack] contains unrecognized parameter: [accept_enterprise]\\\"}],\\\"type\\\":\\\"illegal_argument_exception\\\",\\\"reason\\\":\\\"request [/_xpack] contains unrecognized parameter: [accept_enterprise]\\\"},\\\"status\\\":400}\"} error"}

...and based on this url:
discuss.elastic.co/.../2

Their external es nodes were at SLES 12SP5 and elasticsearch at elasticsearch-7.7.0-1, so they didn't match. The core server was running 7.10.2. The ES version has to match on all nodes. OS can differ between nodes, however.
But make sure you check this as well, especially when adding external nodes to the ES cluster:
www.elastic.co/.../matrix

For example SLES 15 is not supported for ES 7.7, only from 7.9 onward.


We recommend strongly to have additional external nodes in the ES cluster, so it doesn't impact the Sentinel core server, check this:
www.microfocus.com/.../s85system-requirements.html

-> Elasticsearch Cluster Nodes section. It shows EPS level, and how many nodes you need.

And this:
www.microfocus.com/.../t45h61q5wyxv.html

www.microfocus.com/.../b1ndo3vk.html

www.microfocus.com/.../t462ks69vwjj.html

And this as well, when creating cluster:
www.microfocus.com/.../t4e5jeeqolch.html

-> Elasticsearch in Cluster Mode


Fix, when making sure all nodes were running same ES rpm, and OS was supported:
------------------------
Yes, that was the reason, kibana error, ES version mismatch ES nodes.

Just upgraded external ES nodes by:
- wget artifacts.elastic.co/.../elasticsearch-7.10.2-x86_64.rpm
- systemctl stop elasticsearch.service
- rpm -Uvh elasticsearch-7.10.2-x86_64.rpm
- systemctl daemon-reload
- systemctl start elasticsearch.service

...and kibana started right away.

The ES version is so critical.
------------------------


I hope this helps.


Thanks,


Henk

Hi  All

    Thanks all information...I will use ES7.10.x version 

but Henk...I am start learning the ES 

1. Could I use standalone sentinel 8.5 appliance to run ES function ? no other server .

2. About JDK11 , now ES require java11 and SLES12SP5 which is Appliance;s OS use JAVA1.8 and no java update on online update channel...Which correct procedure that I could upgrade or replace SLES embedded JAVA version?? Because base on Appliance update that we discuss ...

I am worried that if I manually download OpenJDK11 to upgrade the built-in JDK1.8... will it cause conflicts afterwards, execute patch/upgrade in the online channnel

Wencheng 

Ok Wencheng, yes use ES 7.10.x version instead.

Regarding ES:
1. Yes you can use standalone sentinel core 8.5 appliance server, to run ES function (single node cluster), but if that "production" server struggles performance wise, or runs out of disk space, etc, you've been warned.

2. Regarding JDK11, now ES requires java11. There is nothing you need to do in this respect (especially for appliance setup). I know it shows the error, but in all my experience working with ES, it has no impact on ES. And Norbert mentioned the same.

So please don't start experimenting with that, removing it, upgrading, etc. Just leave it as it is, and follow our ES docs online accordingly. If you have problems afterwards, raise it with technical support.

Thanks,


Henk

Hi Henk

    I upgrade elasticsearch to 7.10.2 as you provide...the openjdk message still occur , as you said.I ingore.

then conogiure limits.conf and jvm.options like document provide

Installing Elasticsearch - Sentinel Installation and Configuration Guide (microfocus.com)

Finally, I reboot sentinel server to check everything could be apply new setting.

But I check Threat Hunting dashboard...it still show Virtualization server is not ready yet.

Which step I still make incorrect or mistake-configure performance Tuning "Must be modify" base on Lab testing ??

Wencheng

Hi Wencheng,

As I said the best is to raise this with the support team, so they can troubleshoot it accordingly.
I said that a few times now.


I believe you're running Sentinel 8.5 appliance, so you might be hitting this:
www.microfocus.com/.../t4e7btwdtx15.html

...where keystore isn't created correctly:
<sentinel_installation_path>/opt/novell/sentinel/3rdparty/elasticsearch/config/elasticsearch.keystore.tmp

Sometimes a stop/start of sentinel service will give you clues as well (from command line). It should produce errors. Also we would require logs of this setup to verify the general health of the system, see what errors ES is producing, and kibana, etc, hence log it with the technical support team.

Also the team will cover this:
www.microfocus.com/.../t466v69o79ze.html

...but this as well:
www.microfocus.com/.../t45h61q5wyxv.html

www.microfocus.com/.../t4e5jeeqolch.html

...to make sure it's all configured correctly.


Thanks,


Henk