In UCMDB, the regular Universal Discovery jobs are scheduled to run, such as once a day or once a week. During the discovery interval, UCMDB cannot detect what has been changed in remote nodes. For the traditional IT infrastructure, this kind of discovery is good enough to reflect the topology of the data center.
In recent years, the Cloud is becoming quite popular among the IT world. Amazon Web Services, Azure, OpenStack, VMware vCloud, Cloud Foundry, and Docker are rapidly adopted by many companies. One of the common characteristics of these technologies is that changes happen easily and frequently. However, Universal Discovery scheduled jobs cannot accurately handle such changes in real time. Also, the regular discovery jobs scan all data, which is time-consuming. Therefore, Event Based Discovery was introduced in UCMDB. This discovery is based on the event sent out from Cloud providers in near real time. Also, event based discovery only scans the changed data, which saves much more time.
In CMS 2019.08, we are introducing another event based discovery: AWS Event Based Discovery that uses the Amazon SDK for Java to access AWS Simple Queue Service (SQS) service to detect the changes of AWS resources. This discovery brings great performance, stability, controllability, and data quality. If you want to have a better understanding of this discovery, keep reading...
Changes in the SQS queue can be retrieved and sent to the UCMDB server in minutes. The AWS Event Based Discovery job only discovers the changed resources instead of going through all resources, which significantly reduces the data volume. Furthermore, this job can retrieve the resource changes concurrently, resulting in higher throughput. You can set the AWS Event Based Discovery job parameter values to the recommended ones for the best performance. For details, refer to Tips in the AWS Event Based Discovery Job section.
Data Quality (Data Consistency)
Currently, AWS can only publish resource change notifications to standard SQS queues instead of FIFO queues, and therefore the change notifications fetched from SQS queues might be out of order. To avoid overwriting newer resource data with obsolete data, we implemented the following functionality:
- The retrieved AWS resource change notifications are stored in the Data Flow Probe database, and then Probe can sort and process them in order.
- The last capture time of each AWS resource is also maintained in the Data Flow Probe database, and hence Probe can only accept newer resource data.
This functionality also works between AWS via AWS Config Discovery and AWS Event Based Discovery. Consequently, we keep data consistent and the data quality is great!
Discovery & CMDB