kuronen Super Contributor.
Super Contributor.
206 views

SAP XML and future data

I've got a custom made SAP XML driver which has been polished to work quite well but one thing is giving me headache and all ideas are welcome.

I have one E1PLOGI XML element per SAP personality and under that I have a certain infotype which represents a work contract. There are numerous other infotypes representing different data and all that associates these infotypes to the contract infotypes are dates.

I accept different kinds of infotype XML elements for parsing if the element in question falls within the contract dates with following rules:

- if contract is in the past we get the latest entry that intersects the contract period

- if contract is in the present time we get the entry that is valid now

- if contract is in the future we get the first entry that intersects the contract period

 

However we have a problem if we have both present and future data included as we only have the present data in the contract object. In order to update to the future data we'd need to run the XML through the driver in that future date so that it would pick up the data from XML file but that will not happen unless someone in HR changes something.

So I am thinking that maybe I will just mark the contract future date in one attribute and make job to take care of that. Or I could somehow save all the future data in some multivalue attribute(s). How would you do this?

Running all data regularly is very intensive and would not like to take that option.

Labels (1)
0 Likes
11 Replies
Knowledge Partner
Knowledge Partner

Re: SAP XML and future data

I do not think many people already use XML IDocs yet, can you tell us a bit about how you've set them up and what they look like in your case?

______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
kuronen Super Contributor.
Super Contributor.

Re: SAP XML and future data

Here is a simplified sample where you can see infotypes 0001 (which is interpreted as "contract") and 0002 within on person element E1PLOGI. Each infotype and their subelements have multiple versions timestamped with ENDDA and BEGDA elements.

There are dozens of infotypes depending on what they are used for, all residing under the E1PLOGI element. Infotype elements dot not include any association to the other infotype elements other than dates.

   <IDOC BEGIN="1">
    <E1PLOGI SEGMENT="1">
      <OBJID>123456</OBJID>
      <E1PITYP SEGMENT="1">
        <OBJID>123456</OBJID>
        <INFTY>0000</INFTY>
        <BEGDA>18000101</BEGDA>
        <ENDDA>99991231</ENDDA>
        <E1P0000 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0000</INFTY>
          <ENDDA>20120902</ENDDA>
          <BEGDA>20110101</BEGDA>
          <SEQNR>000</SEQNR>
        </E1P0000>
        <E1P0000 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0000</INFTY>
          <ENDDA>20121231</ENDDA>
          <BEGDA>20120903</BEGDA>
          <UNAME>123BELL</UNAME>
        </E1P0000>
        <E1P0000 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0000</INFTY>
          <ENDDA>20130430</ENDDA>
          <BEGDA>20130101</BEGDA>
          <UNAME>123BELL</UNAME>
        </E1P0000>
      <E1PITYP SEGMENT="1">
        <OBJID>123456</OBJID>
        <INFTY>0001</INFTY>
        <BEGDA>18000101</BEGDA>
        <ENDDA>99991231</ENDDA>
        <E1P0001 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0001</INFTY>
          <ENDDA>20120902</ENDDA>
          <BEGDA>20110101</BEGDA>
          <BUKRS>14567</BUKRS>
          <WERKS>14567B8</WERKS>
          <PERSG>55</PERSG>
          <PERSK>22</PERSK>
          <VDSK1>14567B8</VDSK1>
          <BTRTL>14567B9</BTRTL>
          <KOSTL>14567B960E</KOSTL>
          <STELL>00000000</STELL>
          <SNAME>DOE JOHN</SNAME>
          <ENAME>Doe John</ENAME>
          <OTYPE>S</OTYPE>
          <SBMOD>14567B8</SBMOD>
          <KOKRS>14567</KOKRS>
          <Z1P0001_IAM1 SEGMENT="1">
            <ZPSC_X1>123</ZPSC_X1>
            <ZPSC_X2>D</ZPSC_X2>
            <ZPSC_X3>434343</ZPSC_X3>
            <ZPSC_X4>113</ZPSC_X4>
            <POS_DESC_ID>45345</POS_DESC_ID>
            <Z1P0001_IAM2 SEGMENT="1">
              <ORG>MYORG</ORG>
              <SHORT>123456</SHORT>
              <LANGU>E</LANGU>
              <STEXT>Senior Planning Officer</STEXT>
            </Z1P0001_IAM2>
            <Z1P0001_IAM2 SEGMENT="1">
              <ORG>MYORG</ORG>
              <SHORT>123456</SHORT>
              <LANGU>U</LANGU>
              <STEXT>Xyz</STEXT>
            </Z1P0001_IAM2>
          </Z1P0001_IAM1>
          <Z1P0001_IAM4 SEGMENT="1">
            <BEGDA>20110101</BEGDA>
            <ENDDA>20120902</ENDDA>
            <PERSONID_EXT>75474747</PERSONID_EXT>
          </Z1P0001_IAM4>
        </E1P0001>
        <E1P0001 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0001</INFTY>
          <ENDDA>20160902</ENDDA>
          <BEGDA>20120903</BEGDA>
          <BUKRS>14567</BUKRS>
          <WERKS>14567B8</WERKS>
          <PERSG>55</PERSG>
          <PERSK>22</PERSK>
          <VDSK1>14567B8</VDSK1>
          <BTRTL>14567B9</BTRTL>
          <KOSTL>14567B960E</KOSTL>
          <STELL>00000000</STELL>
          <SNAME>DOE JOHN</SNAME>
          <ENAME>Doe John</ENAME>
          <OTYPE>S</OTYPE>
          <SBMOD>14567B8</SBMOD>
          <KOKRS>14567</KOKRS>
          <Z1P0001_IAM1 SEGMENT="1">
            <ZPSC_X1>123</ZPSC_X1>
            <ZPSC_X2>D</ZPSC_X2>
            <ZPSC_X3>434343</ZPSC_X3>
            <ZPSC_X4>113</ZPSC_X4>
            <POS_DESC_ID>45345</POS_DESC_ID>
            <Z1P0001_IAM2 SEGMENT="1">
              <ORG>MYORG</ORG>
              <SHORT>123456</SHORT>
              <LANGU>E</LANGU>
              <STEXT>Senior Planning Officer</STEXT>
            </Z1P0001_IAM2>
            <Z1P0001_IAM2 SEGMENT="1">
              <ORG>MYORG</ORG>
              <SHORT>123456</SHORT>
              <LANGU>U</LANGU>
              <STEXT>Xyz</STEXT>
            </Z1P0001_IAM2>
          </Z1P0001_IAM1>
          <Z1P0001_IAM4 SEGMENT="1">
            <BEGDA>20120903</BEGDA>
            <ENDDA>20160902</ENDDA>
            <PERSONID_EXT>75474747</PERSONID_EXT>
          </Z1P0001_IAM4>
        </E1P0001>
        <E1P0001 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0001</INFTY>
          <ENDDA>20200101</ENDDA>
          <BEGDA>20160903</BEGDA>
          <BUKRS>14567</BUKRS>
          <WERKS>14567B8</WERKS>
          <PERSG>55</PERSG>
          <PERSK>22</PERSK>
          <VDSK1>14567B8</VDSK1>
          <BTRTL>14567B9</BTRTL>
          <KOSTL>14567B960E</KOSTL>
          <STELL>00000000</STELL>
          <SNAME>DOE JOHN</SNAME>
          <ENAME>Doe John</ENAME>
          <OTYPE>S</OTYPE>
          <SBMOD>14567B8</SBMOD>
          <KOKRS>14567</KOKRS>
          <Z1P0001_IAM1 SEGMENT="1">
            <ZPSC_X1>123</ZPSC_X1>
            <ZPSC_X2>D</ZPSC_X2>
            <ZPSC_X3>434343</ZPSC_X3>
            <ZPSC_X4>113</ZPSC_X4>
            <POS_DESC_ID>45345</POS_DESC_ID>
            <Z1P0001_IAM2 SEGMENT="1">
              <ORG>MYORG</ORG>
              <SHORT>123456</SHORT>
              <LANGU>E</LANGU>
              <STEXT>Senior Planning Officer</STEXT>
            </Z1P0001_IAM2>
            <Z1P0001_IAM2 SEGMENT="1">
              <ORG>MYORG</ORG>
              <SHORT>123456</SHORT>
              <LANGU>U</LANGU>
              <STEXT>Xyz</STEXT>
            </Z1P0001_IAM2>
          </Z1P0001_IAM1>
          <Z1P0001_IAM4 SEGMENT="1">
            <BEGDA>20160903</BEGDA>
            <ENDDA>20200101</ENDDA>
            <PERSONID_EXT>75474747</PERSONID_EXT>
          </Z1P0001_IAM4>
        </E1P0001>
      </E1PITYP>
      <E1PITYP SEGMENT="1">
        <OBJID>123456</OBJID>
        <INFTY>0002</INFTY>
        <BEGDA>18000101</BEGDA>
        <ENDDA>99991231</ENDDA>
        <E1P0002 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0002</INFTY>
          <ENDDA>20130903</ENDDA>
          <BEGDA>20150902</BEGDA>
	  <XYZ>123</XYZ>
	</EIP0002>
        <E1P0002 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0002</INFTY>
          <ENDDA>20150903</ENDDA>
          <BEGDA>20190902</BEGDA>
	  <XYZ>123</XYZ>
	</EIP0002>
        <E1P0002 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0002</INFTY>
          <ENDDA>20190903</ENDDA>
          <BEGDA>20200902</BEGDA>
	  <XYZ>123</XYZ>
	</EIP0002>
        <E1P0002 SEGMENT="1">
          <PERNR>123456</PERNR>
          <INFTY>0002</INFTY>
          <ENDDA>20200903</ENDDA>
          <BEGDA>20220902</BEGDA>
	  <XYZ>123</XYZ>
	</EIP0002>
       </EIPITYP>
      </E1PLOGI>
     </IDOC>

So far the best option seems to be just to keep track of future changes with a multivalue attribute and have a trigger job poll the attribute resulting in XML file search + update. Querying an XML file with IDM policies is not very comfortable but I've partially done it already.

0 Likes
Knowledge Partner
Knowledge Partner

Re: SAP XML and future data

So SAP basically converted IDocs to XML and kept the same structure and contents. Instead of position-delimited fields that vary from line to line it's now proper XML, which is nice and makes it easier to read and work with, especially in an XML-centric context like IDM.

You still run into the fundamental "issue" (or "feature") of SAP vs. IDM:

a) SAP stores historical, current and future data, while IDM only holds current data by default

b) SAP events happen when data (current, historic or future) is edited, not necessarily when current data must be changed in IDM.

To handle this properly, I do the following:

1) have the SAP guys always send a full data record incl. all historic, current and future data (not a std config in HCM, but can be done with reasonable effort).

No idea if XML "IDocs" already come that way, can you tell me how you set up the export in SAP in detail, so I can try it out in my test system?

2) Store all data including their validity (BEGDA-ENDDA) on the target object in IDM. We use a multivalue case-sensitive attribute like this:

 

auxSAPDataTimeline: start=20190901#end=20201231#attr=costCenter#value=12345678

 

3) Run a service driver that maintains a list of timestamps when data changes will occur on the target object, e.g:

 

auxSAPDataChangeDate: 20190901
auxSAPDataChangeDate: 20201231

 

4) add a trigger job to the same loopback driver, which will start an ldapsearch for all target objects with auxSAPDataChangeDate<=$today$ and update data attributes (costCenter in this case) from auxDataTimeline where necessary. Run that job every morning (or every hour with a reasonable search result limit so you do not run out of memory when they do mass updates in SAP on Jan. 1st...) and on Sept. 1st costCenter will be set to 12345678 auto-magically.

That setup works quite well for us since years.

 

______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
kuronen Super Contributor.
Super Contributor.

Re: SAP XML and future data

Different from my idea you would query the new data from the timeline attribute as I would query the source XML file.

Yours is kind of better to implement without xpath ecmascript and file operations. On the other hand you will have a multitude of values in the auxSAPDataTimeline attribute and you need to populate the timeline attribute which may not be straightforward? How do you handle schema/filter changes?

A very interesting solution to the problem and will give it more thought, thanks.

Regular full dumps would solve the problem but the amount of data is quite overwhelming and I had to do a lot of optimizing to make it handle it. I would not prefer to do full syncs without a dedicated server.

I am not in control of the SAP side as it comes from a different organization so I cannot help you with SAP side of things. I know they've done quite a bit of work to implement the export since they serve multiple organizations with similar APIs.

0 Likes
Knowledge Partner
Knowledge Partner

Re: SAP XML and future data


@kuronen wrote:

Different from my idea you would query the new data from the timeline attribute as I would query the source XML file.

Basically, the SAP driver maintains the timeline attribute, and the loopback driver maintains the "real" attributes.

Storing the raw SAP data and synching from there in a second step sounds like good idea as well. I'd probably not use the XML files, though, but dump their contents into a database and hook a jdbc driver into that. Which should be quite easy to map since the XML data basically comes from a SQL database (each infotype is stored in it's own table in SAP). On the other hand, that would kind of render the XML files useless, they could as well just give you read access to the DB directly... 🙂


Yours is kind of better to implement without xpath ecmascript and file operations. On the other hand you will have a multitude of values in the auxSAPDataTimeline attribute and you need to populate the timeline attribute which may not be straightforward?


Since we store the IDV side (edir namespace and syntax, only what passes the filter) and not the raw SAP data, it's not too much to handle. We also optimize timelines i.e. if we get the same value from SAP for 2019010-20191231 and 20200101-20201231, we merge them to a single timeline item. This happens a lot in SAP because infotypes (which are basically collections of fields) are copied as a whole with a new validity period as soon as a single field changes. So if an infotype has 50 fields and one changes, 49 fields can be optimized that way. Each time, something is changed in the infotype, that is.

It's also nice to be able to see at each object directly, how it has changed and will change over time.


How do you handle schema/filter changes?

Like in every other driver:  change filter/mapping and resync objects


Regular full dumps would solve the problem but the amount of data is quite overwhelming and I had to do a lot of optimizing to make it handle it. I would not prefer to do full syncs without a dedicated server.


You do not need full dumps of all records, but each record has to have the full dataset for that particular object. You only need records that changed, but for those you need all fields, not just the ones that changed.

I wrote this because classic IDocs usually send only changed infotype records and leave it to the receiving (usually another SAP) system to merge those into the existing data. That merge mechanism can be different for each infotype and the receiving system  needs to know *how* to merge the data properly. In IDM we basically no not know all the requirements, constraints and details of each infotype (which can be configured differently at each customer to make things even more interesting...), so working based on complete records makes life a lot easier and more reliable.


I am not in control of the SAP side as it comes from a different organization so I cannot help you with SAP side of things. I know they've done quite a bit of work to implement the export since they serve multiple organizations with similar APIs.


Do you know if that export is complete custom developed of did they build/customize based on a standard report or program?

 

______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
kuronen Super Contributor.
Super Contributor.

Re: SAP XML and future data

Do you know if that export is complete custom developed of did they build/customize based on a standard report or program?

I do not know that. From my point of view it seems they did spend months implementing it so lots of effort went into it. It may have something to do with SAP IDM but I did not ask them much about it. It includes all the data of a person regardless of the amount of data changed. If surname changes all data with history is sent for that person.

For timestamping I am parsing the XML with XSLT and I am earmarking every attribute with additional metadata with XML attributes something like this:

<add-attr attr-name="zyx" begda="20180101" endda="20190101" class="C"...>

 

Which makes it easy to parse, sort etc. Wouldn't it be nice to be able to store such metadata for attribute? That would solve the timeperiod problem but understandably it would be very challenging to implement and retain performance of eDirectory.

0 Likes
Knowledge Partner
Knowledge Partner

Re: SAP XML and future data

For timestamping I am parsing the XML with XSLT and I am earmarking every attribute with additional metadata with XML attributes something like this:

 

<add-attr attr-name="zyx" begda="20180101" endda="20190101" class="C"...>

 

 

Which makes it easy to parse, sort etc. Wouldn't it be nice to be able to store such metadata for attribute? That would solve the timeperiod problem but understandably it would be very challenging to implement and retain performance of eDirectory


Until IDM implements timelining (maybe with the upcoming new DB backend...?) it's just token do-reformat-op-attr away from the timeline format we store in IDV as mentioned earlier:

 

<do-for-each>
	<arg-node-set>
		<token-text xml:space="preserve">costCenter</token-text>
		<token-text xml:space="preserve">departmentNumber</token-text>
		<token-text xml:space="preserve">workforceID</token-text>
	</arg-node-set>
	<arg-actions>
		<do-reformat-op-attr name="$current-node$">
			<arg-value type="string">
				<token-text xml:space="preserve">start=</token-text>
				<token-xpath expression="$current-node/ancestor::*[@attr-name]/@begda"/>
				<token-text xml:space="preserve">#end=</token-text>
				<token-xpath expression="$current-node/ancestor::*[@attr-name]/@endda"/>
				<token-text xml:space="preserve">#attr=</token-text>
				<token-xpath expression="$current-node/ancestor::*/@attr-name"/>
				<token-text xml:space="preserve">#value=$current-node$</token-text>
			</arg-value>
		</do-reformat-op-attr>
		<do-rename-op-attr dest-name="auxSAPDataTimeline" src-name="$current-node$"/>
	</arg-actions>
</do-for-each>

 

You'll need a bit more code to merge the resulting add-attr/modify-attr/attr nodes into one and clean up the XSLT of course.

______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
Knowledge Partner
Knowledge Partner

Re: SAP XML and future data

Having worked with iDOCS postitionally delimited, I have to say, that XML version is a sight for sore eyes. So much easier than old school idocs.

Still have to think about your specific question though.

The HR driver does it two different ways.  As the shim processes the iDOC it carves out into a new file, the .futr file, all future events with a date.  It polls the file system for files and checks all futr files if they are due yet (I Think by file name, not contents).  Then it replays the future event on the future date.

The other way in the Consulting Edition drivers, it stores the entire state history in a long object and then I forget how it decides when to play forward an event.

0 Likes
kuronen Super Contributor.
Super Contributor.

Re: SAP XML and future data

I could also spit out a bunch of contracts, each for every future changes but knowing the inconsistency of the dates it would probably result in problematic decisions and way too many objects when periods of different ITs would not be identical. That would kind of be the same functionality as the factory driver I guess, done with directory objects instead of .futr files.

0 Likes
Knowledge Partner
Knowledge Partner

Re: SAP XML and future data

Not sure what you mean by "inconsistency" of the timelines. People change addresses at different times than job contracts or their surname. That's why different infotypes have different timelines. "Inconsistent" in my understanding would be e.g. an overlap or a gap between timelines of fields that are supposed to have exactly one value at all times like surname or gender.

______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
kuronen Super Contributor.
Super Contributor.

Re: SAP XML and future data

Yes it feels like inconsistent when time periods of each infotype can and will be anything imaginable. Some infotype time periods spanning over multiple infotype 1 contracts while a single contract can contain zero or multiple infotype periods.

It is not inconsistent by definition. That just comes from our history where the contract infotype was supposed to be the mother of all infotypes where others follow it's timeperiods.

I am going to at least timestamp the contract objects with the future times and possibly make a hybrid of current solution with future values added to a new attribute.

Thanks for your help. Appreciate it greatly.

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.