This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Round Robin load balancing in Lua for IDOL Connector actions

Has any body implemented some kind of Round Robin load balancing with Lua Script?. I need to process a huge volume of URL's with Webconnectors. URL's are stored on a CSV file and I'm processing this, first with a FileSystem connector/CFS and there with Lua Script I send ACI commands to the WebConnector. I would like to have a set of WebConnectors available and implement in the CFS with Lua a kind of Round Robin balancing to send the Action requests to the set of WebConnectors. Any help/reference on this would be apreciated.

Regards, Javier

  • Verified Answer

    The WebConnector fetchAction=synchronize supports &identifiers=<AUTN_IDENTIFIER> and also &config=<BASE64_TASK_CFG>.  I'd probably go with the &config where each action represents a config to process the each URL.  Set depth=0.  You can use SitemapFile or URL.  With SitemapFile you could group together sets of URLs.

    You didn't post the exact format of the URL CSV.  But a simple script in the scripting language of your choice could parse it and submit a=fetch.

    A DAH could sit in front of a cluster of connectors to distribute action=fetch&fetchaction=synchronize.

    -----

    What I describe above isn't exactly an answer to your question, but feels like a solution for your use case.   Feel free to review the above and reply with questions / concerns with this suggestion.

  • Thank you for your prompt response. I just want to confirm the following:

    Does DAH is able to process FETCH actions and send them to IDOL Connectors?.

    Regards, Javier

  • Jeffrey, thank you for your suggestion about DAH in front of a cluster of connectors. It works perfect as a load balancer. That is what we needed, we just implemented and is running now in production.

    Regards, Javier.