This discussion has been locked.
You can no longer post new replies to this discussion. If you have a question you can start a new discussion

Does the 12.7 version of the Web Connector include built-in Lua functions?

After doing some research on the latest versions, I noticed that the 12.7 version of the Instagram connector, Facebook connector, and others all have documentation for built-in Lua functions. These are found under Administration Guide / Connector Lua Functions / Built-In Lua Functions and include functions such as "get_param" or "set_datastore_param". These same functions are not listed anywhere under the Web Connector documentation. Are they not supported?

Parents
  • Hi ,

    The particular lua functions you mention, and related sections of the documentation, are specific to a certain set of connectors. The Facebook and Instagram Connectors, along with a few other social media-related ones, use APIs to obtain data that's structured as a feed (formatted either as JSON or XML), and these functions help in parsing that feed, to determine what constitutes a document, how to determine which links need following, etc.

    Other connectors, such as the Web Connector, obtain data in different formats, such as files on a filesystem, individual web pages, or results from a database query. These don't necessarily require the same sort of fine-tuning for connector actions to run and for records to be passed through the rest of the ingest process. Where there are possibilities to alter this information, such as identifying patterns in a web page to treat as links, this may show up in configuration parameters such as LinkAttributes .

    Most connectors, including the Web Connector, will have a set of common built-in lua functions that you'll find in the Reference, under Lua Functions and Methods . These are most commonly run as IngestActions to do things like adjust document fields before sending them on for ingestion. The "Manipulate Documents" section of the administration guide will usually provide some details under Run Lua Scripts .

  • Hello Geoff,

    Thank you for the quick reply. I bring this up as there used to be a generic social media connector before it was split up into many different connectors. This form of social media connector supported these build-in lua functions I mentioned, but it is outdated. If there was to be an upgrade to a newer form of connector that use APIs to obtain data that's structured as a feed, what connector would be best? It seems that the web connector functions differently and could not be used as a replacement.

  • Right, the Web Connector is more designed to deal with fully-rendered web pages, as a browser would. There isn't an out-of-the-box replacement for the old Social Media Feed Connector, so this may require more of a custom solution. (Though if the APIs in question belong to a commonly used product/site, it wouldn't hurt to put in an enhancement request. New connectors are added all the time.)

Reply
  • Right, the Web Connector is more designed to deal with fully-rendered web pages, as a browser would. There isn't an out-of-the-box replacement for the old Social Media Feed Connector, so this may require more of a custom solution. (Though if the APIs in question belong to a commonly used product/site, it wouldn't hurt to put in an enhancement request. New connectors are added all the time.)

Children
  • One final question for you. Is it possible to use these Social Media Connectors for sources other than the ones they were made for? For instance, using the Facebook connector and adding in a custom FetchTask to connect to a datastore that isn't related to Facebook.