The Generic File Driver is similar to the Text Driver shipped with IDM, but has more options, and has the capability to read virtually any file type. Out of the box, the following file types are supported in both read and write: XML, CSV and XLS (the latter using POI from Apache), and in read-only (since version 1.1): images(png, jpeg, bmp,...) and raw files.
Main differentiators to NetIQ's driver:
Produces and consumes regular NDS documents (no need for XSLT conversion).
Accepts add, modify or instance documents on the subscriber channel.
Auto-query back for missing attributes on modify commands.
GCV (Ecmascript) for generating associations and source DN's.
Publisher channel processes files in a streaming mode. A maximum of 20 records are prefetched/buffered in memory (whenever support by the file type: CSV and XML).
Publisher channel supports interruption (driver stop command) during file processing.
Publisher channel can add metadata about the file or record processed (eg: record number, filename, last record indicator, etc).
The original version I wrote in 2007 is located at http://www.novell.com/coolsolutions/tools/18671.html. This update comes with an IDM 4 package making the installation simpler and has some new features and fixes (see below).
See the referenced version for a list of the initial features.
Installation
The attached zip file (GenFileDriver_1_2.zip) contains the installation documentation and all required runtime files.
Configuration
All configuration options should have a good description. You can also find them back online.
New features and fixes
version 1.2, 11-DEC-2018 (package 0.9, sample config package 0.1)
Shim bug: Shim locked when an empty CSV file was published, and the driver was configured to use the schema from the CSV file (the shim waited for the schema forever).
Shim feature: Converted status to heartbeat events (to have the driverstorage updated).
Shim feature: Moving files now uses java 7 native move file method (and falls back to java 6 code when this fails). Note: shim is still compiled for older java 6 jre, but with java 7 source compatibility.
version 1.1, 17-JAN-2018 (package 0.9, sample config package 0.1)
Shim feature: Added option to define sensitive attributes
Shim feature: Added ImageFileReader and RawFileReader
Shim feature: CSVFileWriter: added option to always surround fields with quotes.
version 1.0, 8-JAN-2017 (base package 0.8, Sample configuration 0.1)
Shim feature: provided an uncompressed jar, so that the shim can be discovered by the Windows remote loader. See forum post. Thanks AB!
Release bug: Version 0.9 release seemed to have failed to include some features/bug fixes. All 0.9 fixes and features should now actually be included.
version 0.9, 7-DEC-2015 (base package 0.8, Sample configuration 0.1)
Shim bug: in certain conditions, doing a retry in the publisher channel on the first record of a file, caused the file not to be reprocessed when restarting the driver.
Shim bug: forum issue with time limited subscriber files.
Package feature: split up of package into base and sample configuration.
Package feature: added Managed System Information Package as an optional package.
Shim feature – publisher – new option to generate a log file for the publisher channel: every status message received as a result of a record, is written to a log file. Base package includes new configuration parameters.
Shim feature – subscriber – added option to keep a local copy of the generated files.
Shim feature – subscriber – changed file closing when maxRecordsPerFile is defined. Previously, when maxRecordsPerFile was set to n, the file was closed just before writing the (n 1)th file. New implementation closes the file after writing the nth record.
version 0.8A, 3-MARCH-2015 (package 0.7)
Shim feature – general – reverted back to java 1.6 compatible ‘move file’ operation.
version 0.8, 14-SEPT-2014 (package 0.7)
Package bug: fixed typo in example xslt template for XML publisher files. See forum thread.
Technical – general refactoring of the shim.
Shim feature - publisher - added option to remove old publisher files. Disabled by default for backward compatibility.
Shim feature - publisher - added option to disable the publisher channel.
Shim feature - publisher - added additional options in the command to generate ("Dynamic Input based" and "Delete")
Shim feature - subscriber - added option to disable the "max number of records in generated files" by setting it to 0 (zero).
Shim feature - subscriber - when a modify command is 'empty' (=all fields defined in the schema are missing), but the content indicates that the output file should be closed (eg: the file close field is set to true), no query back is performed in order to add a record to the output file. It is assumed that the only purpose of this modify command is to close the output file.
Shim feature - general - added option to replace illegal ECMA script characters in field names (eg for calculating the association). Disabled by default for backward compatibility.
Shim feature - general - added option to include the driver’s ECMA script libraries into the shim (for complex ECMA script operation). Disabled by default for backward compatibility.
version 0.7b, 14-MAY-2014 (package 0.6)
Shim bug: case issue with the new configuration parameter to configure the command generated on the publisher channel
version 0.7, 07-MAY-2014 (package 0.6)
Shim bug: specifying no encoding on the XML reader caused an Exception (see also this cool solution post)
Shim feature: added option to configure the command generated on the publisher channel (add, modify or dynamic).
Package bug: xml and xls file readers had invalid class names (see the same cool solutions post)
Package feature: changed xls entry field to multi-line
version 0.6, 28-FEB-2014 (package 0.5)
Shim/package feature: Added configuration parameter to use Quartz scheduler to close subscriber generated files using a cron-like expression string.
Shim feature: Added slf4j binding (separate jar) for Idm tracing, allowing Quartz logs to be shown in idm trace files.
Shim bug: possible NPE when the publisher folder becomes unreachable (eg due to network outage).
Shim feature: Added option for every strategy to receive driver shutdown notification by implementing IShutdown.
Shim bug/feature: Return an empty instance document when we receive an associated query on the subscriber channel. Reason: support of the merge attribute filter rules in case of a 'sync' event.
version 0.5, 21-JAN-2013 (package 0.4)
Shim bug: Fixed an issue wrt the driver not adding the metadata on the publisher channel.
version 0.4, 24-APR-2012 (package 0.4)
Shim bug: Fixed an issue with the driver schema (appeared to be hard coded).
Shim feature: Added beta support for query back. Query back only supported on publisher channel (when reading in a file). Feedback welcome.
Package bug: Fixed an issue that caused the driver configuration not to be imported when the package was imported.
Package bug: Changed creation policy from validating the Given Name to validating the Surname.
Package feature: Added package 'Delimited Text Driver Password Synchronization Package' as an optional package for this driver.
Package feature: Added remote loader prompts to the package.
version 0.3, 12-OCT-2011
Shim feature: automatic association and src-dn calculation (using ecmascript in the driver configuration). No longer need for style sheets for just creating association and src-dn values. The driver shim now does this for you.
Shim bug: fixed missing remove-all-values on the publisher documents.
Shim feature: added the ability (csv output files) to flush the file buffer after every record written (useful for when you are developing/testing your driver).
Shim feature: when receiving a modify event on the subscriber channel, the shim will query back for all missing attribute values in order to be able to write a complete record (no need for a stylesheet to query for missing attributes and keeping this stylesheet in sync with the schema).
Shim bug/feature: the subscriber channel now fully support add, modify and instance events. Instance and add events are assumed to contain all data and do not perform a query back. Only the modify event performs a query back
Download older version
The latest version can be downloaded on the top of this page. If you are looking for an older version, the following versions are still available for download here:
Is it possible that the shim does not have enough rights on the file? From the trace, it seems unable to move the file using java native methods and thus tries a copy/delete sequence. On Linux, failure to move the file is either because of:
* missing file rights
* source and target are on different mount points.
In both cases, the shim falls back to a copy/delete sequence. If it fails to delete the original file after the copy, then you can see this scenario.
I 'm using v1.1.201801170014 build 2018-01-17 00:14:00.
Sometimes the driver begins looping like this, do you know if this is fixed in a later version?
[08/25/20 13:43:32.131]:driverName\PT: Failed to move or copy the file to the destiantion. [08/25/20 13:43:32.132]:driverName\PT: poll - Move failed! [08/25/20 13:43:32.132]:driverName\RegExpFileLocator: getFileList start (/var/idmlogs/t/stuff/import) [08/25/20 13:43:32.132]:driverName\RegExpFileLocator: File matches regexp:export.xml [08/25/20 13:43:32.132]:driverName\RegExpFileLocator: getFileList done [08/25/20 13:43:32.132]:driverName\FilePropertySorter: getNextFile: found file(s) [08/25/20 13:43:32.133]:driverName\PT: File to process:export.xml [08/25/20 13:43:32.133]:driverName\PT: Moving file from /var/idmlogs/t/stuff/import/export.xml to /var/idmlogs/t/stuff/tmp/2020.08.25_11.36.34/export.xml [08/25/20 13:43:32.133]:driverName\PT: Trying to move via copy-delete sequence. [08/25/20 13:43:32.133]:driverName\PT: Exception while copying the file: [08/25/20 13:43:32.133]:driverName\PT: java.io.FileNotFoundException: /var/idmlogs/t/stuff/tmp/2020.08.25_11.36.34/export.xml (No such file or directory) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:162) at info.vancauwenberge.filedriver.util.Util.copyFile(Util.java:211) at info.vancauwenberge.filedriver.util.Util.moveFile(Util.java:274) at info.vancauwenberge.filedriver.filepublisher.FileDriverPublicationShimImpl.poll(FileDriverPublicationShimImpl.java:1021) at info.vancauwenberge.filedriver.filepublisher.FileDriverPublicationShimImpl.start(FileDriverPublicationShimImpl.java:879) at info.vancauwenberge.filedriver.shim.FileDriverPublicationShim.start(FileDriverPublicationShim.java:143) at com.novell.nds.dirxml.engine.Publisher.run(Publisher.java:607) at java.lang.Thread.run(Thread.java:748)
I am wondering if it would be possible to distinguish between input fields (names and order) and output fields. This would give a chance to write files differently than reading. This was one of the features of the MF Delimited Text Driver I liked even if they used XSLT for that. I have this request very often.
Indeed. This was initially coded in 2006 and never upgraded. From the feedback I got till now, this feature (XLS read/write) is not used frequently. For this reason, I included the following comments in the documentation "Note: this was not (re)tested recently. The original development and POI jar was way back around 2006. If you have any experience with the XLSFileWriter, please let me know."
Hello Stefaan! We've started using this for image imports and it works great! We next tried to use this with XLS files to make it easier for admin staff to deal with the files but we've run up against what appears to be either a bug or a version conflict with POI. When we drop the file we get: Message: Code(-9010) An exception occurred: java.lang.NoSuchMethodError: org.apache.poi.hssf.usermodel.HSSFRow.getCell(S)Lorg/apache/poi/hssf/usermodel/HSSFCell; at info.vancauwenberge.filedriver.filereader.xls.XlsFileReader.readRecord(XlsFileReader.java:201) at info.vancauwenberge.filedriver.filepublisher.FileDriverPublicationShimImpl.processFile(FileDriverPublicationShimImpl.java:1092) at info.vancauwenberge.filedriver.filepublisher.FileDriverPublicationShimImpl.poll(FileDriverPublicationShimImpl.java:1022) at info.vancauwenberge.filedriver.filepublisher.FileDriverPublicationShimImpl.start(FileDriverPublicationShimImpl.java:882) at info.vancauwenberge.filedriver.shim.FileDriverPublicationShim.start(FileDriverPublicationShim.java:143) at com.novell.nds.dirxml.engine.Publisher.run(Publisher.java:607) at java.lang.Thread.run(Thread.java:748)
Looking at the source, it looks like the call to getCell is on the HSSFCell object and that doesn't have the method. We were trying this with the 4.0.1 POI on IDM 4.7.1. Is there a particular version of the POI this was built against? Perhaps this is a version conflict with another jar file? Thank you in advance for an amazing piece of code! Will
It is. This is what happens after a long day and then trying something new. I had imported the project from someone else's export and the package was missing from my driver set. After thinking about it for a minute I realized what the issue was and after adding the missing package to my driver set was able to successfully create the driver using the instructions provided. Thanks for the reply as it would have been helpful had I not realized my own mistake.
I know I must be missing something simple but I am unable to create this driver in Designer. I downloaded the 1.2 zip file from here, extracted the contents, and imported the base package jar files into my Desinger project. However, when I attempt to create a new driver and select the Generic File Base (version 1.0.0.20181004231854) I cannot get past the optional features selection in my Designer 4.6. I have tried with nothing selected, everything selected, and pretty much every combination of selections available from the 3 optional feature packages available. They all result in an error stating "The requestion operation for the package 'Generic File Base' cannot be completed due to the following package dependencies" and it is shows a package named "Common Settings". Upon closing that prompt I get a followup message "The dependencies of the following base package could not be resolved: - Generic File Base 1.0.0.20181004231854". Is this package not compatible with Designer 4.6? Am I missing something Java related on my machine? What am I missing? This driver sounds perfect for my needs but I can't seem to make it work and would appreciate any feedback on where I went wrong.
Support
Documentation
Learning Services
Partner Programs