rcano
Visitor.
2129 views

Delimited Text Driver - ImgSource Extension not working

Hello,

I'm trying to upload *.jpg image files to eDirectory with the InputFIle ImageSource Extension for the delimited text driver, but I can't get it working.

When the .jpg image is processed,the driver generates an empty *.tmp file and nothing happens after that. This is what I can see in the trace file:

[12/29/17 17:44:33.592]:DrvPhotoUpload PT:DrvPhotoUpload: Asking the defined InputSource for files...
[12/29/17 17:44:33.592]:DrvPhotoUpload PT:DrvPhotoUpload: found 1 files.
[12/29/17 17:44:33.593]:DrvPhotoUpload PT:DrvPhotoUpload: creating input files.
[12/29/17 17:44:33.593]:DrvPhotoUpload PT:DrvPhotoUpload: inputFileName=/var/idm/photos/downloaded/1514565873593-1.tmp
[12/29/17 17:44:33.594]:DrvPhotoUpload PT:DrvPhotoUpload: returning consolidated input file.
[12/29/17 17:44:33.594]:DrvPhotoUpload PT:DrvPhotoUpload: Renamed file path:/var/idm/photos/downloaded/1514565873593-1.tmp.bak
[12/29/17 17:44:33.594]:DrvPhotoUpload PT:DrvPhotoUpload: File/var/idm/photos/downloaded/1514565873593-1.tmp.bakcould not be deleted.
[12/29/17 17:44:33.595]:DrvPhotoUpload PT:Receiving DOM document from application.
[12/29/17 17:44:33.595]:DrvPhotoUpload PT:
<nds dtdversion="2.0">
<source>
<product build="20170601_1032" instance="DrvPhotoUpload" version="4.0.2.0A">Identity Manager Driver for Delimited Text</product>
</source>
<input>
<status level="success" type="heartbeat"/>
</input>
</nds>


This is my driver configuration parameters:

<?xml version="1.0" encoding="UTF-8"?><driver-config name="Delimited Text">
<driver-options>
<configuration-values>
<definitions>
<gcv-ref driver-param-name="field-delimiter" name="driver.field.delimiter"/>
<!-- <definition display-name="xlfid(NOVLDTXTBASE.initial.settings.param.field-delimiter)Field Delimiter:" name="field-delimiter" type="string">
<value>,</value>
</definition> -->
<gcv-ref driver-param-name="field-names" name="driver.field-names"/>
<!-- <definition display-name="xlfid(NOVLDTXTBASE.initial.settings.param.field-names)Field Names (Field1, Field2, Field3...):" name="field-names" type="string">
<value>WorkforceID,LastName,FirstName,Title,Email,WorkPhone,Fax,WirelessPhone,Description</value>
</definition> -->
<definition display-name="Object Class Name:" name="object-class-name" type="string">
<value>User</value>
</definition>
<definition display-name="Allow Driver to Consume Its Own Output?" name="allow-loopback" type="string">
<value>no</value>
</definition>
<gcv-ref driver-param-name="drivertraceData" name="driver.traceData"/>
</definitions>
</configuration-values>
</driver-options>
<subscriber-options>
<configuration-values>
<definitions>
<definition display-name="Output File Path:" name="output-dir" type="string">
<?generatePrompt?>
<value xml:space="preserve"/>
</definition>
<definition display-name="Output File Extension:" name="output-ext" type="string">
<value>.csv</value>
</definition>
<definition display-name="Add Header to output file:" name="addOutputHeader" type="enum">
<description>Select yes to include the xml declaration in output xml file.</description>
<enum-choice display-name="Yes">yes</enum-choice>
<enum-choice display-name="No">no</enum-choice>
<value>no</value>
</definition>
<definition display-name="Destination File Character Encoding (leave blank for default):" name="output-char-encoding" type="string">
<value/>
</definition>
<definition display-name="Maximum Number of Transactions per Output File:" name="transactions-per-file" type="string">
<value>200</value>
</definition>
<definition display-name="Maximum Time in Seconds before Flushing All Transactions:" name="file-time-out" type="string">
<value>30</value>
</definition>
<definition display-name="Allow Duplicate Records:" name="allowDuplicates" type="enum">
<description>Select yes to enable the duplicate records.</description>
<enum-choice display-name="Yes">yes</enum-choice>
<enum-choice display-name="No">no</enum-choice>
<value>no</value>
</definition>
<definition display-name="Time of Day (Local Time) to Flush All Transactions:" name="time-of-day" type="string">
<value/>
</definition>
</definitions>
</configuration-values>
</subscriber-options>
<publisher-options>
<configuration-values>
<definitions>
<definition display-name="InputSource Class" name="input-source" type="string">
<value>com.novell.nds.dirxml.driver.delimitedtext.imagefile.ImageFileInputSource</value>
</definition>
<definition display-name="InputSource init string" name="input-source-params" type="string">
<value>srcdir=/var/idm/photos/downloaded;consolidate=false;debug=true</value>
</definition>
<definition display-name="Field Delimiter" name="field-delimiter" type="string">
<value>#</value>
</definition>
<definition display-name="Field Names" name="field-names" type="string">
<value>idx,name,prefix,suffix,size,modified,pic64</value>
</definition>
<definition display-name="Input File Path:" name="input-dir" type="string">
<?generatePrompt?>
<value xml:space="preserve">/var/idm/photos/downloaded</value>
</definition>
<definition display-name="Input File Extension:" name="input-ext" type="string">
<value>.csv</value>
</definition>
<definition display-name="Source File Character Encoding (leave blank for default):" name="input-char-encoding" type="string">
<value/>
</definition>
<definition display-name="Rename File Extension:" name="input-rename-ext" type="string">
<value>.bak</value>
</definition>
<definition display-name="Polling Rate (in seconds):" name="input-poll-rate" type="string">
<value>60</value>
</definition>
<definition display-name="Publisher heartbeat interval" name="pub-heartbeat-interval" type="integer">
<description>Configures the driver shim to send a periodic status message on the Publisher channel when there has been no Publisher traffic for the given number of minutes.</description>
<value>1</value>
</definition>
</definitions>
</configuration-values>
</publisher-options>
</driver-config>


Can anybody tell me what I'm doing wrong?

Thanks in advance,
Rodrigo Cano
Labels (1)
0 Likes
32 Replies
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

<definition display-name="InputSource init string"
name="input-source-params" type="string">
<value>srcdir=/var/idm/photos/downloaded;consolidate=false;debug=true</value>

</definition>


This looks wrong. Take another look at the docs. I don’t believe you are
using the right params.
Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

Alex McHugh wrote:

In a working image import driver here I use the following:

<definition display-name="Input Source Extension Class:" name="input-source"
type="string">
<value
xml:space="preserve">com.novell.nds.dirxml.driver.delimitedtext.imagefile.ImageF
ileInputSource</value>
</definition>
<definition display-name="Input Source Extension Parameters:"
name="input-source-params" type="string">
<value
xml:space="preserve">srcdir=/var/idm/in/images;renameto=done;consolidate=false;d
ebug=true</value>
</definition>

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

Lothar Haeger <lothar.haeger@is4it.de> wrote:
> Alex McHugh wrote:
>
> In a working image import driver here I use the following:
>
> <definition display-name="Input Source Extension Class:" name="input-source"
> type="string">
> <value
> xml:space="preserve">com.novell.nds.dirxml.driver.delimitedtext.imagefile.ImageF
> ileInputSource</value>
> </definition>
> <definition display-name="Input Source Extension Parameters:"
> name="input-source-params" type="string">
> <value
> xml:space="preserve">srcdir=/var/idm/in/images;renameto=done;consolidate=false;d
> ebug=true</value>
> </definition>
>


Yeah. The original poster didn’t have the rename option there (instead
relied on the regular DLT import rename function). Not sure if that is such
a great idea.

The original trace posted had the line:


File/var/idm/photos/downloaded/1514565873593-1.tmp.bakcould not be deleted.

Which led me to think there is an issue with rename.

Aside from that: post the driver filter. I suspect something is missing
there.


Alex McHugh - Knowledge Partner - Stavanger, Norway
Who are the Knowledge Partners
If you appreciate my comments, please click the Like button.
If I have resolved your issue, please click the Accept as Solution button.
0 Likes
rcano
Visitor.

Re: Delimited Text Driver - ImgSource Extension not working

Hello, sorry for the delay.

I modified the publisher parameters to include the renameto option.

Now, the image files are processed and renamed correctly to *.done as I specified in the configuration, however, the error when the driver tries to delete the "tmp" file is still there:

[01/02/18 11:45:16.402]:DrvPhotoUpload PT:DrvPhotoUpload: Asking the defined InputSource for files...
[01/02/18 11:45:16.403]:DrvPhotoUpload PT:DrvPhotoUpload: found 5 files.
[01/02/18 11:45:16.403]:DrvPhotoUpload PT:DrvPhotoUpload: creating input files.
[01/02/18 11:45:16.403]:DrvPhotoUpload PT:DrvPhotoUpload: inputFileName=/var/idm/photos/downloaded/1514889916403-5.tmp
[01/02/18 11:45:16.407]:DrvPhotoUpload PT:DrvPhotoUpload: returning consolidated input file.
[01/02/18 11:45:16.408]:DrvPhotoUpload PT:DrvPhotoUpload: Renamed file path:/var/idm/photos/downloaded/1514889916403-5.tmp.bak
[01/02/18 11:45:16.408]:DrvPhotoUpload PT:DrvPhotoUpload: File/var/idm/photos/downloaded/1514889916403-5.tmp.bakcould not be deleted.


This is my driver configuration now:
<?xml version="1.0" encoding="UTF-8"?><driver-config name="Delimited Text">
<driver-options>
<configuration-values>
<definitions>
<gcv-ref driver-param-name="field-delimiter" name="driver.field.delimiter"/>
<!-- <definition display-name="xlfid(NOVLDTXTBASE.initial.settings.param.field-delimiter)Field Delimiter:" name="field-delimiter" type="string">
<value>,</value>
</definition> -->
<gcv-ref driver-param-name="field-names" name="driver.field-names"/>
<!-- <definition display-name="xlfid(NOVLDTXTBASE.initial.settings.param.field-names)Field Names (Field1, Field2, Field3...):" name="field-names" type="string">
<value>WorkforceID,LastName,FirstName,Title,Email,WorkPhone,Fax,WirelessPhone,Description</value>
</definition> -->
<definition display-name="Object Class Name:" name="object-class-name" type="string">
<value>User</value>
</definition>
<definition display-name="Allow Driver to Consume Its Own Output?" name="allow-loopback" type="string">
<value>no</value>
</definition>
<gcv-ref driver-param-name="drivertraceData" name="driver.traceData"/>
</definitions>
</configuration-values>
</driver-options>
<subscriber-options>
<configuration-values>
<definitions>
<definition display-name="Output File Path:" name="output-dir" type="string">
<?generatePrompt?>
<value xml:space="preserve"/>
</definition>
<definition display-name="Output File Extension:" name="output-ext" type="string">
<value>.csv</value>
</definition>
<definition display-name="Add Header to output file:" name="addOutputHeader" type="enum">
<description>Select yes to include the xml declaration in output xml file.</description>
<enum-choice display-name="Yes">yes</enum-choice>
<enum-choice display-name="No">no</enum-choice>
<value>no</value>
</definition>
<definition display-name="Destination File Character Encoding (leave blank for default):" name="output-char-encoding" type="string">
<value/>
</definition>
<definition display-name="Maximum Number of Transactions per Output File:" name="transactions-per-file" type="string">
<value>200</value>
</definition>
<definition display-name="Maximum Time in Seconds before Flushing All Transactions:" name="file-time-out" type="string">
<value>30</value>
</definition>
<definition display-name="Allow Duplicate Records:" name="allowDuplicates" type="enum">
<description>Select yes to enable the duplicate records.</description>
<enum-choice display-name="Yes">yes</enum-choice>
<enum-choice display-name="No">no</enum-choice>
<value>no</value>
</definition>
<definition display-name="Time of Day (Local Time) to Flush All Transactions:" name="time-of-day" type="string">
<value/>
</definition>
</definitions>
</configuration-values>
</subscriber-options>
<publisher-options>
<configuration-values>
<definitions>
<definition display-name="InputSource Class" name="input-source" type="string">
<value>com.novell.nds.dirxml.driver.delimitedtext.imagefile.ImageFileInputSource</value>
</definition>
<definition display-name="InputSource init string" name="input-source-params" type="string">
<value>srcdir=/var/idm/photos/downloaded;consolidate=false;renameto=done;debug=true</value>
</definition>
<definition display-name="Field Delimiter" name="field-delimiter" type="string">
<value>#</value>
</definition>
<definition display-name="Field Names" name="field-names" type="string">
<value>idx,name,prefix,suffix,size,modified,pic64</value>
</definition>
<definition display-name="Input File Path:" name="input-dir" type="string">
<?generatePrompt?>
<value xml:space="preserve">/var/idm/photos/downloaded</value>
</definition>
<definition display-name="Input File Extension:" name="input-ext" type="string">
<value>.jpg</value>
</definition>
<definition display-name="Source File Character Encoding (leave blank for default):" name="input-char-encoding" type="string">
<value/>
</definition>
<definition display-name="Rename File Extension:" name="input-rename-ext" type="string">
<value>.bak</value>
</definition>
<definition display-name="Polling Rate (in seconds):" name="input-poll-rate" type="string">
<value>60</value>
</definition>
<definition display-name="Publisher heartbeat interval" name="pub-heartbeat-interval" type="integer">
<description>Configures the driver shim to send a periodic status message on the Publisher channel when there has been no Publisher traffic for the given number of minutes.</description>
<value>1</value>
</definition>
</definitions>
</configuration-values>
</publisher-options>
</driver-config>


And this is my filter:
<?xml version="1.0" encoding="UTF-8"?><!DOCTYPE filter PUBLIC "dirxmlfilter" "C:\NetIQ\idm\apps\Designer\plugins\com.novell.idm.filter_4.0.0.201509291552\DTD\dirxmlfilter.dtd"><filter>
<filter-class class-name="User" publisher="sync" subscriber="ignore">
<filter-attr attr-name="HipraFoto" merge-authority="default" publisher="notify" publisher-optimize-modify="true" subscriber="ignore"/>
</filter-class>
</filter>


The HipraFoto attribute is mapped in the schema to pic64, the attribute specified in the attr list of the imageFIle extension in the publisher configuration. Probably I'm not understanding well how this works and I need to change something here.

Thank you,
Rodrigo Cano
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

If you don't have anything against the generic file driver you can follow my coolsolution on how to import photos for users:
https://www.netiq.com/communities/cool-solutions/using-generic-file-driver-importing-photos/

0 Likes
Highlighted
rcano
Visitor.

Re: Delimited Text Driver - ImgSource Extension not working

joakim_ganse;2472584 wrote:
If you don't have anything against the generic file driver you can follow my coolsolution on how to import photos for users:
https://www.netiq.com/communities/cool-solutions/using-generic-file-driver-importing-photos/



Hi Joakim, thanks for your interest.

I saw your coolsolution the same day I tried the Delimiter Text driver option. In fact, I got it working with the Generic File driver after following your guidelines in the coolsolution article, so my problem is solved right now.

Anyway, I want to make it work with the Delimiter driver too, its a personal challenge :D.
0 Likes
rcano
Visitor.

Re: Delimited Text Driver - ImgSource Extension not working

Another thing that I noticed is the empty size of the *.tmp file. Seems the driver is not writing any content into the file for some reason, probably I'm missing something in the publisher configuration.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

rcano wrote:

> Another thing that I noticed is the empty size of the *.tmp file. Seems
> the driver is not writing any content into the file for some reason,
> probably I'm missing something in the publisher configuration.


May sound too obvious, but did you check file system access rights?

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
rcano
Visitor.

Re: Delimited Text Driver - ImgSource Extension not working

lhaeger;2472592 wrote:
rcano wrote:

> Another thing that I noticed is the empty size of the *.tmp file. Seems
> the driver is not writing any content into the file for some reason,
> probably I'm missing something in the publisher configuration.


May sound too obvious, but did you check file system access rights?

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)


The driver shim is executing as root so it has all the necessary rights. In fact, the *.tmp file is being created by the driver itself.
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

I cannot think of any reason, short of something like AppArmor or SELinux,
that would prevent deleting a file when running as 'root'; even if the
file is in use the filesystem should allow deleting. Maybe if this is a
non-standard filesystem, like an SMB or NFS mountpoint or something, other
rules could apply, but otherwise this makes little sense. Nothing blocks
'root' unless it is the kernel itself, which is why you can block it with
AppArmor or SELinux, or you can specifically squash its abilities across
NFS with root-squashing (not technically limiting root normally as never
giving it any possibilities for that filesystem in the kernel).

The file mentioned in the trace does exist, right? Exactly as specified
in the trace? You mentioned empty (zero-byte) files, but that does not
mean the path necessarily matches correctly.

Since the Generic File Driver option works I'd give up on this one; the
other shim is superior anyway, so fixing old broken stuff does not make
much sense to me. I tried it once and it was less-than fun.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

ab wrote:

> Since the Generic File Driver option works I'd give up on this one; the
> other shim is superior anyway


Generally true, but Joakims approach does not allow to outsource the image
reading/writing to a RL. May not be a limitation in this case but can be a
problem on busy engine servers where you do not want to e.g. bulk load
thousands to GB-sized images in the same memory space as all other drivers.

Stefaan mentioned in the comments to Joakim's CS article that he was looking
into creating a specific file reader for this purpose to overcome that
limitaion, though.

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
Knowledge Partner Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

On 01/02/2018 07:16 AM, Lothar Haeger wrote:
> ab wrote:
>
>> Since the Generic File Driver option works I'd give up on this one; the
>> other shim is superior anyway

>
> Generally true, but Joakims approach does not allow to outsource the image
> reading/writing to a RL. May not be a limitation in this case but can be a
> problem on busy engine servers where you do not want to e.g. bulk load
> thousands to GB-sized images in the same memory space as all other drivers.


I am not sure I understand the concern. Whether a Remote Loader (RL) is
used or not, eventually the data will be sent to the engine for processing
.. The actual file I/O is not going to prevent that from eventually
happening by being pushed to the RL, right? I feel like I am missing
something.

There are definitely other features that may be nice to test with the
shim, like image sizes and such, though pre-processing with something else
could also be easily setup; yes, nice to have in the shim. If this is
what you mean, then I agree.

I also think this 'renameto' option is poorly-named; appendsuffix would
make a lot more sense.

--
Good luck.

If you find this post helpful and are logged into the web interface,
show your appreciation and click on the star below.

If you want to send me a private message, please let me know in the
forum as I do not use the web interface often.
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

ab wrote:

> I am not sure I understand the concern.


Whatever can go wrong in the shim e.g. with corrupt images will not affect the
engine if it runs in a RL.

> There are definitely other features that may be nice to test with the
> shim, like image sizes and such, though pre-processing with something else
> could also be easily setup; yes, nice to have in the shim. If this is
> what you mean, then I agree.


If I was writing an InputFile extension for images to the Generic File driver,
I'd definitely include min/max/resize/crop/pad/convert options in the shim,
yes. Exposing EXIF data as well as file and image meta data as generic
attributes would make a lot of sense, too.

--
http://www.is4it.de/en/solution/identity-access-management/

(If you find this post helpful, please click on the star below.)
______________________________________________
https://www.is4it.de/identity-access-management
0 Likes
Knowledge Partner
Knowledge Partner

Re: Delimited Text Driver - ImgSource Extension not working

I use it with a RL. That is no problem.
Stefaan was writing about including the ecmascript functionality so the b64 transformation would be unnecessary.
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.