ScorpionSting Absent Member.
Absent Member.
594 views

8.2: mongod continually crashes

I have an issue where mongod continually crashes, but logs are not overly helpful:

mongod.log


Sat Jan 12 09:58:47.865 [initandlisten] MongoDB starting : pid=72498 port=27017 dbpath=/var/opt/novell/sentinel/3rdparty/mongodb/data 64-bit host=sentinel
Sat Jan 12 09:58:47.865 [initandlisten] db version v2.4.9
Sat Jan 12 09:58:47.865 [initandlisten] git version: 52fe0d21959e32a5bdbecdc62057db386e4e029c
Sat Jan 12 09:58:47.865 [initandlisten] build info: Linux ip-10-2-29-40 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 BOOST_LIB_VERSION=1_49
Sat Jan 12 09:58:47.865 [initandlisten] allocator: tcmalloc
Sat Jan 12 09:58:47.865 [initandlisten] options: { auth: true, bind_ip: "127.0.0.1", dbpath: "/var/opt/novell/sentinel/3rdparty/mongodb/data", fork: true, journal: true, logappend: true, logpath: "/var/opt/novell/sentinel/log/mongod.log", oplogSize: 4645, port: 27017, replSet: "sidbReplSet" }
Sat Jan 12 09:58:47.865 [initandlisten]
Sat Jan 12 09:58:47.865 [initandlisten] ** WARNING: Readahead for /var/opt/novell/sentinel/3rdparty/mongodb/data is set to 512KB
Sat Jan 12 09:58:47.865 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
Sat Jan 12 09:58:47.865 [initandlisten] ** http://dochub.mongodb.org/core/readahead
Sat Jan 12 09:58:47.866 [initandlisten] journal dir=/var/opt/novell/sentinel/3rdparty/mongodb/data/journal
Sat Jan 12 09:58:47.866 [initandlisten] recover : no journal files present, no recovery needed
Sat Jan 12 09:58:47.925 [initandlisten] waiting for connections on port 27017
Sat Jan 12 09:58:47.931 [websvr] admin web console waiting for connections on port 28017
Sat Jan 12 09:58:47.947 [rsStart] replSet I am 127.0.0.1:27017
Sat Jan 12 09:58:47.955 [rsStart] replSet STARTUP2
Sat Jan 12 09:58:48.101 [initandlisten] connection accepted from 127.0.0.1:35568 #1 (1 connection now open)
Sat Jan 12 09:58:48.107 [conn1] command denied: { serverStatus: 1.0 }
Sat Jan 12 09:58:48.111 [conn1] end connection 127.0.0.1:35568 (0 connections now open)
Sat Jan 12 09:58:48.968 [rsSync] replSet SECONDARY
Sat Jan 12 09:58:48.968 [rsMgr] replSet info electSelf 0
Sat Jan 12 09:58:49.973 [rsMgr] replSet PRIMARY
Sat Jan 12 09:58:53.968 [initandlisten] connection accepted from 127.0.0.1:35582 #2 (1 connection now open)
Sat Jan 12 09:58:53.991 [conn2] authenticate db: alerts { authenticate: 1, user: "appuser", nonce: "d21d4571f166d876", key: "302f18c36aaa6b078390876e7f7d6fc5" }
Sat Jan 12 09:58:53.995 [conn2] authenticate db: alerts { authenticate: 1, user: "appuser", nonce: "5dbe488aa5a08384", key: "4ae2bf5fd2f0d76930b9c56eb3a47033" }
Sat Jan 12 09:58:54.014 [initandlisten] connection accepted from 127.0.0.1:35584 #3 (2 connections now open)
Sat Jan 12 09:58:54.049 [conn3] authenticate db: admin { authenticate: 1, user: "dbauser", nonce: "5897f5d916b93816", key: "02d58fd2f68cc2d92839040e52e5fff4" }
Sat Jan 12 09:58:54.096 [conn3] authenticate db: admin { authenticate: 1, user: "dbauser", nonce: "3d969d2f63ddc730", key: "3ac3ad43848f8ef89c301be5daefbbad" }
Sat Jan 12 09:58:54.168 [conn3] end connection 127.0.0.1:35584 (1 connection now open)
Sat Jan 12 09:58:54.268 [conn2] end connection 127.0.0.1:35582 (0 connections now open)
Sat Jan 12 09:59:02.560 [initandlisten] connection accepted from 127.0.0.1:35594 #4 (1 connection now open)
Sat Jan 12 09:59:02.566 [conn4] authenticate db: admin { authenticate: 1, user: "dbauser", nonce: "afd8f425df0142d", key: "98a620d238d7b55947d18f80346fabab" }
Sat Jan 12 09:59:02.568 [conn4] end connection 127.0.0.1:35594 (0 connections now open)
Sat Jan 12 09:59:02.568 [initandlisten] connection accepted from 127.0.0.1:35596 #5 (1 connection now open)
Sat Jan 12 09:59:02.569 [conn5] authenticate db: admin { authenticate: 1, user: "dbauser", nonce: "dabc34f27544005c", key: "413d4661f93b3952798e93ea63bcd374" }
Sat Jan 12 09:59:02.569 [initandlisten] connection accepted from 127.0.0.1:35598 #6 (2 connections now open)
Sat Jan 12 09:59:02.570 [conn6] authenticate db: admin { authenticate: 1, user: "dbauser", nonce: "c1a339448dbbb12a", key: "fb7f81d812e0dea31ccc21f726807cb2" }
Sat Jan 12 09:59:44.792 [conn6] end connection 127.0.0.1:35598 (1 connection now open)
Sat Jan 12 09:59:44.793 [conn5] end connection 127.0.0.1:35596 (1 connection now open)
Sat Jan 12 09:59:46.136 [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
Sat Jan 12 09:59:46.137 [signalProcessingThread] now exiting
Sat Jan 12 09:59:46.137 dbexit:
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: going to close listening sockets...
Sat Jan 12 09:59:46.137 [signalProcessingThread] closing listening socket: 13
Sat Jan 12 09:59:46.137 [signalProcessingThread] closing listening socket: 14
Sat Jan 12 09:59:46.137 [signalProcessingThread] closing listening socket: 15
Sat Jan 12 09:59:46.137 [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: going to flush diaglog...
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: going to close sockets...
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: waiting for fs preallocator...
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: lock for final commit...
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: final commit...
Sat Jan 12 09:59:46.137 [signalProcessingThread] shutdown: closing all files...
Sat Jan 12 09:59:46.137 [signalProcessingThread] closeAllFiles() finished
Sat Jan 12 09:59:46.137 [signalProcessingThread] journalCleanup...
Sat Jan 12 09:59:46.137 [signalProcessingThread] removeJournalFiles
Sat Jan 12 09:59:46.145 [signalProcessingThread] shutdown: removing fs lock...
Sat Jan 12 09:59:46.152 dbexit: really exiting now


mongoconnector.log


Sat 12 Jan 2019 09:58:48 AM AEDT Security Intelligence database is running.
2019-01-12 09:59:02,449 - INFO - Beginning Mongo Connector
2019-01-12 09:59:02,569 - INFO - OplogThread: Initializing oplog thread
2019-01-12 09:59:02,571 - INFO - MongoConnector: Starting connection thread MongoClient('127.0.0.1', 27017)
2019-01-12 09:59:02,575 - INFO - OplogThread: dumping collection alerts.alerts
2019-01-12 09:59:02,593 - INFO - Starting new HTTP connection (1): 127.0.0.1
2019-01-12 09:59:12,604 - WARNING - POST http://127.0.0.1:9200/_bulk [status:N/A request:10.011s]
Traceback (most recent call last):
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/connection/http_urllib3.py", line 46, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=headers, **kw)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/urllib3-1.8.3-py2.6.egg/urllib3/connectionpool.py", line 496, in urlopen
body=body, headers=headers)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/urllib3-1.8.3-py2.6.egg/urllib3/connectionpool.py", line 327, in _make_request
self, url, "Read timed out. (read timeout=%s)" % read_timeout)
ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=10)
2019-01-12 09:59:12,605 - INFO - > {"index": {"_type": "string", "_id": "0325D836-B6F5-1036-BE36-000C294C00E8", "_index": "alerts.alerts"}}


Makes a few json attempts which result in:


2019-01-12 09:59:12,606 - WARNING - Connection <Urllib3HttpConnection: http://127.0.0.1:9200> has failed for 1 times in a row, putting on 60 second timeout.
2019-01-12 09:59:12,607 - INFO - Resurrecting connection <Urllib3HttpConnection: http://127.0.0.1:9200> (force=True).
2019-01-12 09:59:12,607 - INFO - Starting new HTTP connection (2): 127.0.0.1
2019-01-12 09:59:22,616 - WARNING - POST http://127.0.0.1:9200/_bulk [status:N/A request:10.009s]
Traceback (most recent call last):
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/connection/http_urllib3.py", line 46, in perform_request
response = self.pool.urlopen(method, url, body, retries=False, headers=headers, **kw)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/urllib3-1.8.3-py2.6.egg/urllib3/connectionpool.py", line 496, in urlopen
body=body, headers=headers)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/urllib3-1.8.3-py2.6.egg/urllib3/connectionpool.py", line 327, in _make_request
self, url, "Read timed out. (read timeout=%s)" % read_timeout)
ReadTimeoutError: HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=10)
2019-01-12 09:59:22,616 - INFO - > {"index": {"_type": "string", "_id": "0325D836-B6F5-1036-BE36-000C294C00E8", "_index": "alerts.alerts"}}


Until it finally does a:


2019-01-12 09:59:42,638 - WARNING - Connection <Urllib3HttpConnection: http://127.0.0.1:9200> has failed for 4 times in a row, putting on 480 second timeout.
Traceback (most recent call last):
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/oplog_manager.py", line 522, in do_dump
upsert_all(dm)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/oplog_manager.py", line 506, in upsert_all
dm.bulk_upsert(docs_to_dump())
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/doc_managers/__init__.py", line 33, in wrapped
reraise(new_type, exc_value, exc_tb)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/doc_managers/__init__.py", line 27, in wrapped
return f(*args, **kwargs)
File "/opt/novell/sentinel/3rdparty/mongoconnector/doc_managers/elastic_doc_manager.py", line 131, in bulk_upsert
for ok, resp in responses:
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/helpers/__init__.py", line 104, in streaming_bulk
resp = client.bulk(bulk_actions, **kwargs)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/client/utils.py", line 68, in _wrapped
return func(*args, params=params, **kwargs)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/client/__init__.py", line 646, in bulk
params=params, body=self._bulk_body(body))
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/transport.py", line 276, in perform_request
status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/elasticsearch-1.1.1-py2.6.egg/elasticsearch/connection/http_urllib3.py", line 51, in perform_request
raise ConnectionError('N/A', str(e), e)
ConnectionFailed: ConnectionError(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=10)) caused by: ReadTimeoutError(HTTPConnectionPool(host='127.0.0.1', port=9200): Read timed out. (read timeout=10))
2019-01-12 09:59:42,665 - ERROR - OplogThread: Failed during dump collection cannot recover! Collection(Database(MongoClient('127.0.0.1', 27017), u'local'), u'oplog.rs')
2019-01-12 09:59:42,884 - ERROR - MongoConnector: OplogThread <OplogThread(Thread-2, started 140660917139200)> unexpectedly stopped! Shutting down
2019-01-12 09:59:42,884 - INFO - MongoConnector: Stopping all OplogThreads
Sat 12 Jan 2019 09:59:45 AM AEDT
Sat 12 Jan 2019 09:59:46 AM AEDT Security Intelligence database is running at PID 72498.
Sat 12 Jan 2019 09:59:46 AM AEDT Stopping Security Intelligence database...
Sat 12 Jan 2019 09:59:47 AM AEDT Stopped the Security Intelligence database.
2019-01-12 09:59:47,334 - INFO - Beginning Mongo Connector
Exception in thread Thread-1:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/mongo_connector-1.2.1_-py2.6.egg/mongo_connector/connector.py", line 274, in run
main_conn = MongoClient(self.address, tz_aware=True)
File "/opt/novell/sentinel/3rdparty/mongoconnector/site-packages/pymongo-2.7.1-py2.6-linux-x86_64.egg/pymongo/mongo_client.py", line 369, in __init__
raise ConnectionFailure(str(e))
ConnectionFailure: [Errno 111] Connection refused


What I do notice is that the script /opt/novell/sentinel/3rdparty/mongoconnector/startmc_1.sh calls for a "KEY_FILE" of /var/opt/novell/sentinel/data/tmp/${KEY_FILE} which doesn't exist. I can run the $("/opt/novell/sentinel/bin/sidb_reconfig.sh" -g | tail -n 1 | xargs echo -n) as novell and correctly get the jcf result.

Should this file be created dynamically? I've tried watching the directory during a rc startSIdb but the file doesn't seem to appear...

With mongo dead, elasticsearch also fails as 9200 isn't open.

Visit my Website for links to Cool Solution articles.
0 Likes
1 Reply
AutomaticReply Absent Member.
Absent Member.

Re: 8.2: mongod continually crashes

ScorpionSting,

It appears that in the past few days you have not received a response to your
posting. That concerns us, and has triggered this automated reply.

These forums are peer-to-peer, best effort, volunteer run and that if your issue
is urgent or not getting a response, you might try one of the following options:

- Visit https://www.microfocus.com/support-and-services and search the knowledgebase and/or check
all the other self support options and support programs available.
- Open a service request: https://www.microfocus.com/support
- You could also try posting your message again. Make sure it is posted in the
correct newsgroup. (http://forums.microfocus.com)
- You might consider hiring a local partner to assist you.
https://www.partnernetprogram.com/partnerfinder/find.html

Be sure to read the forum FAQ about what to expect in the way of responses:
http://forums.microfocus.com/faq.php

Sometimes this automatic posting will alert someone that can respond.

If this is a reply to a duplicate posting or otherwise posted in error, please
ignore and accept our apologies and rest assured we will issue a stern reprimand
to our posting bot.

Good luck!

Your Micro Focus Forums Team
http://forums.microfocus.com



0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.