Having problems with your account or logging in?
A lot of changes are happening in the community right now. Some may affect you. READ MORE HERE
Highlighted
zinin Respected Contributor.
Respected Contributor.
1432 views

Event Broker 2.21 (Local Docker registry problem)

Hello, everyone!

We are trying to deploy our new EB installation with EB version 2.21.

We stopped on page 24 of EventBroker_DeploymentGuide_2.21.pdf, on deploying the product images:
1. We have the following configuration (+ External NFS Server):Screenshot_300.jpg

 

 

 

 

 

 

 

2. Images were succesfully loaded to the local Docker registry (all the previous steps were also succesfully completed):Screenshot_295.jpg

 

 

 

 

3. But, when we go to the ArcSight Installer web application and browse to the Deployment page we can see only Investigate there, which we didn't load:Screenshot_296.jpg

 

 

 

 

 

 

4. I rechecked all my commands with this guide - https://community.softwaregrp.com/t5/ArcSight-User-Discussions/ArcSight-Event-Broker-2-21-from-the-ground-up-build-guide-v1a/m-p/1662169#M44011, everything is OK too.

Obviously there are some problems with my local Docker registry, but I can't figure out what they are.

If someone had the same problem, please share your solution or help us to understand what was done wrong, and how to fix it.

Thanks in advance, best regards, Timur.

 

P.S.
5. I have the following warning "watch -n 15 kubectl get pods --all-namespaces" (ImagePullBackOff):Screenshot_299.jpg

 

 

 

 

 

 

 

 

 

6. Current "kubectl get nodes" result (all OK):Screenshot_297.jpg

 

 

 

 

7. Current "kubectl get nodes -L=kafka,zk" result (all OK):Screenshot_298.jpg

 

 

0 Likes
30 Replies
Marijo Mandic Acclaimed Contributor.
Acclaimed Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Hello,

1) Try to check checksum of "arcsight-eventbroker-2.21.9.tar" that you unpacked.
2) Try to redeploy core NAMESPACE and then reupload the EventBroker images of checksum is fine and redeployment is successful.
3) To redeploy core NAMESPACE:
a) SSH to Master1 node
b) cd /opt/arcsight/kubernetes/bin
c) ./kube-redeploy.sh
4) You could also try to check the logs (if there is something reasonable there):
a) get all the PODs:
kubectl get pods --all-namespaces -o wide
b) now for example the proxy POD that shows issue is named like "kube-registry-proxy-xyz":
kubectl logs kube-registry-proxy-xyz --namespace=core kube-registry-proxy
kubectl logs kube-registry-proxy-xyz --namespace=core kubernetes-vault-renew
kubectl logs kube-registry-proxy-xyz --namespace=core instal
5) Also yo ucould try to delete the POD to see if it recovers to running state (when deleted it will be recreated automatically again), the redeploy step will do this also, delete the POD:
kubectl delete pod kube-registry-proxy-xyz --namespace=core

Regards,

Marijo

0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Hello, Marijo

1) I checked the hash of the archive - everything is OK:

Screenshot_302.jpg

 

 

 

 

 

3) I ran the /opt/arcsight/kubernetes/bin/kube-redeploy.sh script:Screenshot_304.jpg

 

 

 

 

 


But it didn't help, the problem POD was still in "ImagePullBackOff" status.

5) I tried to delete the problem POD to see if it will be recreated automatically and recover to running state, and at first it went to "Termintating" state:

 Screenshot_307.jpg

 

 

 

 

 

 

 

  
But afterwards, it got back to "ImagePullBackOff" status again, but with a new name:

Screenshot_308.jpg

 

 

 

 

 

 

 

 4) I also cant check the logs, because of the following mistake:Screenshot_310.jpg

 

 

 

 

 

 

 

Any other suggestions/tips?

Thanks a lot again, appreciate your help

Regards,

Timur.

0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Now the problem POD is in "ErrImagePull" status. Any attempts to check the logs ends with the same mistake "Error from server (BadRequest): container "kube-registry-proxy" in pod "kube-registry-proxy-b5x8r" is waiting to start: trying and failing to pull image":

Screenshot_311.jpg

 

 

 

 

 

 

 

 

0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Oops - decided to check the node with the problem POD, and suddenly noticed that there is no "Mounted on: /opt" in "df -h" command output, obviously some mounting problems occured...I'll repair them and will get back to you with the results.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Event Broker 2.21 (Local Docker registry problem)

A similar issue occured with me, and i was lucky enough to find another one that had already debugged this issue.

For us, we use hardening on our linux machines, so "umask 027" was set.

If you installed the whole eventbroker utilizing umask 027, you will need to remove it and reinstall it with umask 022.

What causes the issue is that 027 set certain restrictions on who and what is allowed to mount certain parts, and we got issue deploying the actual images with the same issue.

There is also no screenshot of output from kubernetes events, could you also check with using, if i remember it correctly on the top of my head.

kubectl --namespace=NAMESPACe get events

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Thank you very much for that tip, Marius!
We currently have the following settings:

if [ $UID -gt 199 ] && [ "`/usr/bin/id -gn`" = "`/usr/bin/id -un`" ]; then
umask 002
else
umask 022
fi

I'll repair "Mounted on: /opt" on the problem node and will get back to you.

Thanks again, best regards, Timur!

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Event Broker 2.21 (Local Docker registry problem)

Do you set this on shell initialization in for example bashrc, or is this in /etc/profile?

Though if you run the umask command manually while logged in i guess that kinda confirms it :) It just seems quite similar.

There is also issues with kubernetes taking down pods all the time due to "insufficient resources", because even though almost everything is in /opt, it puts one specific folder in /var, and the disk space check there stopped us from loading images and installing.

We had to modify the path in the systemctl script, to both lower the size requirements, and the datadir location, which fixed it for us.

The above issue will be quite visible when listing the kubectl get events, either a -o wide just in case you have the wrong namespace, it should always be quite noisy, so if you get nothing in return then something is wrong.

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Well

1. I fixed the "Mounted on: /opt" mistake on the problem node and now all the PODs are OK and "Running":

Screenshot_317.jpg

 

 

 

 

 

 

 

 

2. I succesfully added the Event Broker offline images to the initial master node:Screenshot_318.jpg

 

 

 

 

But, there is still no Event Broker application in the ArcSight Installer Deployment page! There is still only Investigate option there, whose image I didn't load.

Will try to figure out something and will get back to you.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Event Broker 2.21 (Local Docker registry problem)

i would start with the kubectl events, it should be spamming out errors if it is something it has issues with accessing.

I remember i had some issues getting the correct events from kubectl, mostly related to namespacing. I got this from my notes i tok during the last troubleshooting session, and i will update the post when i find the more specific ones.

You might know all the ways to troubleshoot already though, but i always started with:

kubectl get events --all-namespaces

This is because certain events are split between namespaces, and this always pointed me in the right direction at least, issues related to images, mounts, systemctl override paramters that was needed etc etc.

I will add in the rest when i find them!

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Thanks again, Marius!

Ill start with analyzing all the events from:

kubectl get events --all-namespaces

I already looked at:

kubectl --namespace=core get events AND journalctl -f

But there are only WARNs about  '/etc/resolv.conf' (we have more than 3 domains) wich can't be the reason of my current problem:

Search Line limits were exceeded, some dns names have been omitted, the applied search line is:***

checkLimitsForResolvConf: Resolv.conf file '/etc/resolv.conf' contains search line consisting of more than 3 domains!

 

Looking forward to your next post, regards, Timur!

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: Event Broker 2.21 (Local Docker registry problem)

DNS Resolving is actually a fairly big part of it actually, all names needs to be resolvable in the same format they are configured in the software + reverse dns entries needs to be available for all IP addresses. If not, it can cause several issues.

If you only used core namespace i don't think you would see all the relevant events actually :) Might be possible to add a "-o wide" as well?

The specific DNS error you have is something i have on my working environment as well, so thats fine :)

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

DNS Resolving is surely a big part of it so we ensured host name resolution through DNS across all nodes (forward and reverse lookups).

 

The specific DNS error you have is something i have on my working environment as well, so thats fine :)

(:

I'll continue analyzing the logs today, hope that I'll be able to figure something out, will report later.

0 Likes
Marijo Mandic Acclaimed Contributor.
Acclaimed Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Hello,

1) If you still see only "Investigate" for Deployment in ArcSight Installer GUI (https://vip.domain.local:5443), check the images you have with this command (this lists all the images so you can see what is actually there):
docker images

2) With following command you can delete specific images (do not do it, but just as info):
docker rmi -f "IMAGE ID"

3) So for example EventBroker 2.21 with uploaded images for EventBroker so you can compare to yours:

EventBroker_2.21_fresh_install_without_Investigate_images.PNG

 

Regards,

Marijo

0 Likes
zinin Respected Contributor.
Respected Contributor.

Re: Event Broker 2.21 (Local Docker registry problem)

Hello and thank you Marijo for the tips.

This is mine docker images command output:

Screenshot_323.jpg

 

 

 

 

 

 

 

 

 

Looks OK according to your example. But unfortunately I still see only "Investigate" for Deployment in ArcSight Installer GUI. Thinking of rebooting the whole cluster, maybe it'll help.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.