Highlighted
Valued Contributor.. Valued Contributor..
Valued Contributor..
441 views

How to deal with chunks in executeTopologyQueryByName SOAP calls?

Dear Experts,

 

I am trying to give out one of the UCMDB TQLs for another team to call via Webservices API for their automated data quality improvement purposes, as their tooling is one of our data sources. Long story short, I am facing "chunking" limitation. Using the method executeTopologyQueryByName and the TQL is simple Node-->IP with some conditions. For just a couple of CIs it works as perfect, but for all the CIs, which is around 30k objects the SOAP answer is chunked and I cannot figure out how to see all the chunks.

 

The default setting ucmdb.wsapi.query.chunk.size is 500, I did not dare to make it 30k yet.

 

Any hints how to do this?

 

Regards,

Alex.

0 Likes
4 Replies
Highlighted
Regular Contributor.. Regular Contributor..
Regular Contributor..

Hi Alex,

when one deals with large amount of data, they indeed come in chunks. In anycase, in the SOAP response, there should be a ChunkInfo information, which will tell how much chunks you'd get.

You can search for ChunkInfo and pullTopologyMapChunks in the UCMDB Developer Reference Guide for precise information.

Now, I'll paste below a code extract I wrote a log time ago, using a client code I generated using Axis2 wsdltoJava. The code may not fit 100% to what you wrote, but at least, it should help you to understand how to get chunk info, and how to retrieve data by chunk.

 

---

 public IDs getCIsByType(String ciType) {
  
  IDs result = new IDs();
  ArrayList<ID> resultList = new ArrayList<ID>();
  
  GetCIsByType req = new GetCIsByType();
  req.setCmdbContext(getContext());
  
  req.setType(ciType);
  
  CustomProperties cProps = new CustomProperties();
  
  PredefinedProperties pProps = new PredefinedProperties();

  SimplePredefinedPropertyCollection sppColl = new SimplePredefinedPropertyCollection();
  
  ArrayList<SimplePredefinedProperty> sppList = new ArrayList<SimplePredefinedProperty>();
  
  SimplePredefinedProperty spp = new SimplePredefinedProperty();
  spp.setName(Name.CONCRETE);
  
  sppList.add(spp);
  
  sppColl.setSimplePredefinedPropertyList(sppList);
  
  pProps.setSimplePredefinedProperties(sppColl);
  
  cProps.setPredefinedProperties(pProps);
  
  PropertiesList pList = new PropertiesList();
  
  
  cProps.setPropertiesList(pList);
  
  req.setProperties(cProps);
  
  try {
   
   GetCIsByTypeResponse resp = getStub().getCIsByType(req);
   ChunkInfo chunkInfo = resp.getChunkInfo();
   
   // if all data are in 1 chuck, the number of chunks is 0
   if (chunkInfo.getNumberOfChunks() == 0) {
    // all data are contained in one chunk
    System.out.println("Found " + resp.getCIs().getCIList().size() + " CIs:");
    for (CI ci: resp.getCIs().getCIList()) {
     this.displayCI(ci);
     resultList.add(ci.getID());
    }
    
   }
   
   // if there are more then 1 chunks, chunk number starts at 1
   for (int i = 1; i <= chunkInfo.getNumberOfChunks(); i++ ) {
    ChunkRequest chunkReq = new ChunkRequest();
    chunkReq.setChunkInfo(chunkInfo);
    chunkReq.setChunkNumber(i);

    PullTopologyMapChunks req2 = new PullTopologyMapChunks();
    req2.setCmdbContext(getContext());
    req2.setChunkRequest(chunkReq);

    PullTopologyMapChunksResponse resp2 = getStub().pullTopologyMapChunks(req2) ;

    for(int m=0 ; m < resp2.getTopologyMap().getCINodes().getCINodeList().size(); m++) {

     List<CINode> ciNodeList = resp2.getTopologyMap().getCINodes().getCINodeList();
     for (CINode ciNodes : ciNodeList) {
      for (CI ci: ciNodes.getCIs().getCIList()) {
       this.displayCI(ci);
       resultList.add(ci.getID());
      }
     }
    }
    
   }
   
   //release chunks to free memory on server side. This is to be done after we processed all chunks of data
   ReleaseChunks req3 = new ReleaseChunks();
   req3.setChunksKey(chunkInfo.getChunksKey());
   req3.setCmdbContext(getContext());
   
   try {
    getStub().releaseChunks(req3);
   } catch (RemoteException e) {
    //handle exception
   } catch (UcmdbFault e) {
    //handle exception
   }

   
   result.setIDList(resultList);
   return(result);
   
  } catch (RemoteException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  } catch (UcmdbFault e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }
  
  return(null);
  
 } // end getCIsByType()

---

 

Hope this helps

 

Best regards,

 

Pierre

 

0 Likes
Highlighted
Valued Contributor.. Valued Contributor..
Valued Contributor..

Great reply, Pierre! That is actually very helpful! 🙂

 

P.S. Is your last name by chance Driutti?

 

Kudos,

Alex.

0 Likes
Highlighted
Valued Contributor.. Valued Contributor..
Valued Contributor..

Pierre, maybe you'd have a clue how to do these three cals (executeTopologyQueryByName and then pullTopologyMapChunks) via the soap envelope? I am using SOAP UI tool for this purpose and actually seeing rather a strange response in chunkInfo and cannot use it.

 

This is my initiall call:

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:quer="http://schemas.hp.com/ucmdb/1/params/query" xmlns:typ="http://schemas.hp.com/ucmdb/1/types" xmlns:prop="http://schemas.hp.com/ucmdb/1/types/props">
<soap:Header/>
<soap:Body>
<quer:executeTopologyQueryByName>
<quer:cmdbContext>
<typ:callerApplication>SOAP UI</typ:callerApplication>
</quer:cmdbContext>
<quer:queryName>Missing TADDM Servers based on TAD4D</quer:queryName>
</quer:executeTopologyQueryByName>
</soap:Body>
</soap:Envelope>

 

This is the response:

<soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope">
<soapenv:Header/>
<soapenv:Body>
<tns:executeTopologyQueryByNameResponse xmlns:ns0="http://schemas.hp.com/ucmdb/1/types" xmlns:ns1="http://schemas.hp.com/ucmdb/ui/1/types" xmlns:ns2="http://schemas.hp.com/ucmdb/1/types/impact" xmlns:ns3="http://schemas.hp.com/ucmdb/1/types/query" xmlns:ns4="http://schemas.hp.com/ucmdb/1/types/props" xmlns:ns5="http://schemas.hp.com/ucmdb/1/types/history" xmlns:ns6="http://schemas.hp.com/ucmdb/1/types/classmodel" xmlns:ns7="http://schemas.hp.com/ucmdb/discovery/1/types" xmlns:ns8="http://schemas.hp.com/ucmdb/1/types/update" xmlns:tns="http://schemas.hp.com/ucmdb/1/params/query">
<tns:topologyMap>
<ns0:CINodes/>
<ns0:relationNodes/>
</tns:topologyMap>
<tns:chunkInfo>
<ns0:numberOfChunks>99</ns0:numberOfChunks>
<ns0:chunksKey>
<ns0:key1>Missing TADDM Servers based on TAD4D</ns0:key1>
<ns0:key2>Missing TADDM Servers based on TAD4D6cddb969c5d29fe865084363cc1721a2</ns0:key2>
</ns0:chunksKey>
</tns:chunkInfo>
</tns:executeTopologyQueryByNameResponse>
</soapenv:Body>
</soapenv:Envelope>

Then I try to use chunkInfo:

<soap:Envelope xmlns:soap="http://www.w3.org/2003/05/soap-envelope" xmlns:quer="http://schemas.hp.com/ucmdb/1/params/query" xmlns:typ="http://schemas.hp.com/ucmdb/1/types" xmlns:prop="http://schemas.hp.com/ucmdb/1/types/props">
<soap:Header/>
<soap:Body>
<quer:pullTopologyMapChunks>
<quer:cmdbContext>
<typ:callerApplication>SOAP UI</typ:callerApplication>
</quer:cmdbContext>
<typ:ChunkRequest>
<typ:chunkNumber>1</typ:chunkNumber>
<typ:chunkInfo>
<typ:numberOfChunks>99</typ:numberOfChunks>
<typ:chunksKey>
<typ:key1>Missing TADDM Servers based on TAD4D</typ:key1>
<typ:key2>Missing TADDM Servers based on TAD4D6cddb969c5d29fe865084363cc1721a2</typ:key2>
</typ:chunksKey>
</typ:chunkInfo>
</typ:ChunkRequest>
</quer:pullTopologyMapChunks>
</soap:Body>
</soap:Envelope>

 

and end up with exception:

<soapenv:Envelope xmlns:soapenv="http://www.w3.org/2003/05/soap-envelope">
<soapenv:Header/>
<soapenv:Body>
<soapenv:Fault>
<soapenv:Code>
<soapenv:Value>soapenv:Sender</soapenv:Value>
</soapenv:Code>
<soapenv:Reason>
<soapenv:Text xml:lang="en-US">operation pullTopologyMapChunks error: error was occured when trying to invoke operation [TqlQueryGetResultMapChunk] cause:appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [200] The TQL query is missing]
appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [200] The TQL query is missing]
CMDB Operation Internal Error: class appilog.framework.shared.manage.impl.MamResponseException : appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [200] The TQL query is missing]
CMDB Operation Internal Error: class com.mercury.topaz.cmdb.shared.tql.exception.TqlException : Could not find the pattern in result cache[]. Probably, time period for keeping chunk (configured intql.resultutils.chunk.keepingperiod.seconds) has been passed : operation Tql Query: Get Result Chunk From Repository. Pattern [], Chunk [0] : class com.mercury.topaz.cmdb.shared.base.CmdbException :
CMDB Internal Error: Error while handling request: {request: ID='bd538a1cc5de7e83826a27769c618232' Message='General CMDB request' Operation='com.mercury.topaz.cmdb.server.tql.operation.query.impl.TqlQueryGetChunkResultFromCache@54ff73ad' Customer ID='1'
Context='CMDB Context: Customer id = '1' , User ID = '1' , Caller Application = 'SOAP UI,loggedInUser:{SOAP_API_USER}''} : operation Tql Query: Get Result Map Chunk. Pattern [Missing TADDM Servers based on TAD4D]. ChunkRequest [Current:0. Last:99. Pattern:] : class com.mercury.topaz.cmdb.shared.base.CmdbException :
CMDB Internal Error: Error while handling request: {request: ID='26d089e2bd03ede8c2956b67585de1c0' Message='General CMDB request' Operation='com.mercury.topaz.cmdb.shared.tql.operation.query.impl.TqlQueryGetResultMapChunk@592de890' Customer ID='1'
Context='CMDB Context: Customer id = '1' , User ID = '1' , Caller Application = 'SOAP UI,loggedInUser:{SOAP_API_USER}''}; nested exception is:
java.lang.Exception:</soapenv:Text>
</soapenv:Reason>
<soapenv:Detail>
<tns:ucmdbFault xmlns:ns0="http://schemas.hp.com/ucmdb/1/types" xmlns:ns1="http://schemas.hp.com/ucmdb/ui/1/types" xmlns:ns2="http://schemas.hp.com/ucmdb/1/types/impact" xmlns:ns3="http://schemas.hp.com/ucmdb/1/types/query" xmlns:ns4="http://schemas.hp.com/ucmdb/1/types/props" xmlns:ns5="http://schemas.hp.com/ucmdb/1/types/history" xmlns:ns6="http://schemas.hp.com/ucmdb/1/types/classmodel" xmlns:ns7="http://schemas.hp.com/ucmdb/discovery/1/types" xmlns:ns8="http://schemas.hp.com/ucmdb/1/types/update" xmlns:tns="http://schemas.hp.com/ucmdb/1/params/ucmdb">
<tns:msg>error was occured when trying to invoke operation [TqlQueryGetResultMapChunk] cause:appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [200] The TQL query is missing]
appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [200] The TQL query is missing]
CMDB Operation Internal Error: class appilog.framework.shared.manage.impl.MamResponseException : appilog.framework.shared.manage.impl.MamResponseException: [ErrorCode [200] The TQL query is missing]
CMDB Operation Internal Error: class com.mercury.topaz.cmdb.shared.tql.exception.TqlException : Could not find the pattern in result cache[]. Probably, time period for keeping chunk (configured intql.resultutils.chunk.keepingperiod.seconds) has been passed : operation Tql Query: Get Result Chunk From Repository. Pattern [], Chunk [0] : class com.mercury.topaz.cmdb.shared.base.CmdbException :
CMDB Internal Error: Error while handling request: {request: ID='bd538a1cc5de7e83826a27769c618232' Message='General CMDB request' Operation='com.mercury.topaz.cmdb.server.tql.operation.query.impl.TqlQueryGetChunkResultFromCache@54ff73ad' Customer ID='1'
Context='CMDB Context: Customer id = '1' , User ID = '1' , Caller Application = 'SOAP UI,loggedInUser:{SOAP_API_USER}''} : operation Tql Query: Get Result Map Chunk. Pattern [Missing TADDM Servers based on TAD4D]. ChunkRequest [Current:0. Last:99. Pattern:] : class com.mercury.topaz.cmdb.shared.base.CmdbException :
CMDB Internal Error: Error while handling request: {request: ID='26d089e2bd03ede8c2956b67585de1c0' Message='General CMDB request' Operation='com.mercury.topaz.cmdb.shared.tql.operation.query.impl.TqlQueryGetResultMapChunk@592de890' Customer ID='1'
Context='CMDB Context: Customer id = '1' , User ID = '1' , Caller Application = 'SOAP UI,loggedInUser:{SOAP_API_USER}''}; nested exception is:
java.lang.Exception:</tns:msg>
<tns:stacktrace>stack trace is N/A</tns:stacktrace>
</tns:ucmdbFault>
</soapenv:Detail>
</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>

 

How to work this around?

 

Regards,

Alex.

 

 

0 Likes
Highlighted
Regular Contributor.. Regular Contributor..
Regular Contributor..

Hi again Alex,

 

It looks you need to put the ChunkRequest in the  PullTopologyMapChunks.

See below the piece of code I've just tested:

---

 public void executeTopologyQueryByName(String queryName) {
  
  ExecuteTopologyQueryByName request = new ExecuteTopologyQueryByName();
  
  request.setCmdbContext(getContext());
  
  request.setQueryName(queryName);
  
  request.setQueryTypedProperties(null);
  
   try {
    ExecuteTopologyQueryByNameResponse response = getStub().executeTopologyQueryByName(request);
   
     ChunkInfo chunkInfo = response.getChunkInfo();
    
     logger.info("Nb of chunks: " + chunkInfo.getNumberOfChunks());
   
     // if all data are in 1 chuck, the number of chunks is 0
    
     if (chunkInfo.getNumberOfChunks() == 0) {

      this.processTopologyMap(response.getTopologyMap());

     } else {

      // if there are more then 1 chunks, chunk number starts at 1
     
      for (int i = 1; i <= chunkInfo.getNumberOfChunks(); i++ ) {
      
       ChunkRequest chunkReq = new ChunkRequest();
      
       chunkReq.setChunkInfo(chunkInfo);
      
       chunkReq.setChunkNumber(i);

       PullTopologyMapChunks req2 = new PullTopologyMapChunks();
      
       req2.setCmdbContext(getContext());
      
       req2.setChunkRequest(chunkReq);

       PullTopologyMapChunksResponse resp2 = getStub().pullTopologyMapChunks(req2) ;
      
       logger.info("Processing Chunk [" + i + "]");

       this.processTopologyMap(resp2.getTopologyMap());
      


      }

      //release chunks to free memory on server side. This is to be done after we processed all chunks of data
      ReleaseChunks req3 = new ReleaseChunks();
      req3.setChunksKey(chunkInfo.getChunksKey());
      req3.setCmdbContext(getContext());

      try {
       getStub().releaseChunks(req3);
      } catch (RemoteException e) {
       //handle exception
      } catch (UcmdbFault e) {
       //handle exception
      }


     }
         } catch (RemoteException e) {
             logger.error(ExceptionUtils.getExceptionStackTraceAsString(e));
         } catch (UcmdbFault e) {
             logger.error(ExceptionUtils.getExceptionStackTraceAsString(e));
         }
  
  
  
 }

 
 private void processTopologyMap(TopologyMap tMap) {
  
  for (CINode ciNode : tMap.getCINodes().getCINodes()) {
   logger.info("Got [" + ciNode.getCIs().getCIs().size() + "] CIs for node [" + ciNode.getLabel() + "]");  
  }
  
  for (RelationNode relNode : tMap.getRelationNodes().getRelationNodes()) {
   logger.info("Got [" + relNode.getRelations().getRelations().size() + "] relations for link [" + relNode.getLabel() + "]");
  }
  
 }

---

 

It produced the following output:

 

---

2016-01-29 14:57:08,281 - INFO [UcmdbWebService.executeTopologyQueryByName:552] - Nb of chunks: 16

2016-01-29 14:57:08,285 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,321 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [1]

2016-01-29 14:57:08,322 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,327 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,347 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [2]

2016-01-29 14:57:08,348 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,356 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,379 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [3]

2016-01-29 14:57:08,379 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,386 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,399 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [4]

2016-01-29 14:57:08,400 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,405 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,419 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [5]

2016-01-29 14:57:08,420 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,423 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,440 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [6]

2016-01-29 14:57:08,441 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,444 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,456 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [7]

2016-01-29 14:57:08,456 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,459 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,470 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [8]

2016-01-29 14:57:08,471 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,475 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,486 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [9]

2016-01-29 14:57:08,487 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,490 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,499 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [10]

2016-01-29 14:57:08,500 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,503 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,513 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [11]

2016-01-29 14:57:08,513 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,516 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,529 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [12]

2016-01-29 14:57:08,530 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,533 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,545 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [13]

2016-01-29 14:57:08,545 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,548 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,559 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [14]

2016-01-29 14:57:08,559 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,563 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,577 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [15]

2016-01-29 14:57:08,577 - INFO [UcmdbWebService.processTopologyMap:617] - Got [500] CIs for node [ConfigurationItem]

2016-01-29 14:57:08,580 - INFO [AuthChallengeProcessor.selectAuthScheme:101] - basic authentication scheme selected

2016-01-29 14:57:08,588 - INFO [UcmdbWebService.executeTopologyQueryByName:580] - Processing Chunk [16]

2016-01-29 14:57:08,588 - INFO [UcmdbWebService.processTopologyMap:617] - Got [23] CIs for node [ConfigurationItem]

---

So, it seemed to work fine.

Hope this helps

 

best regards,

Pierre

 

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.