Knowledge Partner
Knowledge Partner
171 views

Order of processing files in Delim Text driver

I am reasonably sure the Delim text driver uses oldest file first, but
what happens if you dump 10,000 files into the directory (one event
each) within a one second interval (Usual file system datestamp
granularity, I think?).

How does it order them for processing then?

Is any of that above statement correct?
Labels (1)
0 Likes
6 Replies
Anonymous_User Absent Member.
Absent Member.

Re: Order of processing files in Delim Text driver

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Goes by system default, by default. 🙂

https://www.netiq.com/documentation/idm402drivers/delimited/data/bf16brl.html

Now, what does "system default" mean? I'd guess that actually means the
order that the filesystem returns things by default. If you really,
really care, define an InputSorter as mentioned above.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQKV/wAAoJEF+XTK08PnB57scQAMTThBKgC6giByRWqYYN0Lz3
ezt6kxQ/h1F+2iDNZwsQ0iwlxOEA6VdingDZlQIRM2X6WKHV8NK6YCPP0kVbkiDI
wK8O5Oyo/25Jhed49BUWhbIxiGwdu2YFSypJntxbNRx3uItmFXdTfBO1hvUcdUhk
Tdb48lNkTYRuVcFn7YbuDQO0iERJxJCdFxCJbwNMEHWTavcY7RC8MGD8O39sPAOt
5bGj9HI6xIrAZALY8SVVnIWF08kztTpEEiA3NdGp8glsQ0wWCDQkSYueK91FKxw9
H/jo7UZJfiGbFMrRUWVY6rd9PkKqACSy82B7H6n8+j7lZuQgETpU4EeZ/mpkFcag
cGZX5wnesnpdgugvGOH22VtV4Y+KSAzmAYP5OVQ42HLR8EuXmZ4KaHc1xjXDBtqN
MCFKvIAwbWZm/J+Bt6YOO2vZRFWNJHgE9jqdRLjW594Zls+forJzui0BxSc0TBDP
GXN3JRwcaSt8XULVRsFI8Gk7M/lgIkYFD2tr5MY2cKf9psIqf4Dm7m5i9+//ouDO
4EyxgBTw8zb5i9YINPNnPx7LfprOlY/gUGdjnvMqDTKqE5mKqFj2KI4ka61VFlGr
X9lhrwwznefnUH2tIudeXz/g6iQX41RvelEGVBsx4GNyifOgDIIFzKoasLO6M6q9
ddJTFEze6VKHjvCTJTrb
=RoZp
-----END PGP SIGNATURE-----
0 Likes
Knowledge Partner
Knowledge Partner

Re: Order of processing files in Delim Text driver

On 8/13/2012 4:13 PM, ab wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Goes by system default, by default. 🙂
>
> https://www.netiq.com/documentation/idm402drivers/delimited/data/bf16brl.html
>
> Now, what does "system default" mean? I'd guess that actually means the
> order that the filesystem returns things by default. If you really,
> really care, define an InputSorter as mentioned above.


What do you think the default for Linux is? Think it differs by file
system?


0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Order of processing files in Delim Text driver

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

My guess is either timestamp (my timestamps go to nanoseconds on btrfs,
fwiw), or maybe inode (usually order of creation but not necessarily).
File name may come into play too, but I really don't know how to tell
without doing a bunch of research.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQKXq2AAoJEF+XTK08PnB5O1YQAMzhcwjRiqPkDbmkyW1syX94
EJMN+WGdltZj1nNUsDltJdslN/lgXajOxHlLoGm8neECKlALFrHfPXBlH5fk1FIe
4JTGhhDt53K3rUXCyG+l34q51R/SvA+Aci3/UBfpTD0kKa2MdUrrnnH9D+DYXcUZ
5b6XGhGfaNInqfB+7vpuVvAkHpCBagt9VYKtxbo4vgyMAk/oF4c2tCfkmxbcWpmr
wAL5bdAQEEcKUMkxdrcda4v8z4TfOqwDhRhytNdnhF1iUJ5dQQDGTfeR+hMtty6n
gz9JCo7+NuSrTy9yjSdOL1mUiqOl89cJREK4UOUKtFSder0MoikDRAZXb71/44nT
XhVYchXeY3T98vdtHEdJjtDxwijJF1zX/00mGEfaCUZmK/R6dHOZ6QOrhuMTEUb8
hS26Eep6uK3YnGRi9383nPQcFXOK3UY/B4DwGKfYOUiGvcE/uAbUAl54ZLfxZZE/
k9rvYh/TCuUXMyvP2LTO0UdyxF3xyZebhksTMYZUFqNqgadEYyP0vov0yc/C6SWk
OgN3acxSlGrknwWDR/clR2TPVLWAkooXjBrntW3UhQfWU2iLvpgcFVn8596cPll2
4Gxo3tx36Ju+oLd7AgwWb/KB8Xr5hERVdJsntF7BmmGPBgtxLBmUTCwt32axpXoj
9emtkj7rUKx492QIZ7pR
=IaOn
-----END PGP SIGNATURE-----
0 Likes
Knowledge Partner
Knowledge Partner

Re: Order of processing files in Delim Text driver

> My guess is either timestamp (my timestamps go to nanoseconds on btrfs,
> fwiw), or maybe inode (usually order of creation but not necessarily).
> File name may come into play too, but I really don't know how to tell
> without doing a bunch of research.


That sounds like an excuse for some interesting research then!

I checked ext3fs file systems all around. (Such boring folk these
server guys.)

0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Order of processing files in Delim Text driver

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Well let me add more. Checking ext4 also has nanoseconds, xfs has
nanoseconds, and I cannot find reiserfs but who cares. ext2 has second
granularity but if you're still using that for anything important then,
well, you're insane.

Good luck.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.18 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBAgAGBQJQKbUZAAoJEF+XTK08PnB5l00QAL7WTdmLENygI9TmRLnTFO6e
8KIEGRNeBVao2vmTjX0qt+mxEIOCMHepV8NyRbKVG4kRukAaysj2OJmMDKIXhxlR
B/vii0u8Gc4eOL+tduX0Sghy+50H/9hYVxG4ZZsnm0U2w6Yx11A0OWTvuh8gxTrw
3gqpjfdoDEJLHyiEd5kzYFJStRXt2ydQFxfsQSy+by5Al5dVjFQzsZ2/R1BmUmlc
Xj50JautVdRTXu8XrlC+kOwKzkY3nbbTV2g3wGUc5TerxBcMYVgjRtIBXBZJp/Yu
2gztrdAO+bapiy2jMVXOXW9yH3VkQ6abKPP4nbhS3zTRbF0J3oitbw9Qir6MoLQ8
38p0MagHZa/Qh6Sd3pLdcjKkT8yhdu8dTpKBsnHuQSJdbZrMkeqrZAEcyph28Jph
k4UWaXgyXXo2VC50KXkkqVfXWqPKgjQOgXGpBnu6HqDu+fegbHLdgqQ/+bPBGUXZ
puv0XBI8qtRKln0pQV19EUiUJdOXix2XN4H18wsAW+TX4FeGKHLgas9IX4P6DxJq
W9hY76+HsuKZDBaC7JFrP45enHcgXZ8+SKMGXmE8lDRMoSDTs1InWg3B8JMju0b8
FzmKXRYoJzdB2OhCNYGh6RLhvV9htt5yUdZi0YqcNqlpd6iWBkkF6cQWfIev4UDF
7cYqHKRQlx58YTNLbXlG
=USXl
-----END PGP SIGNATURE-----
0 Likes
Anonymous_User Absent Member.
Absent Member.

Re: Order of processing files in Delim Text driver

On 8/13/12 2:13 PM, ab wrote:
>
> Now, what does "system default" mean? I'd guess that actually means the
> order that the filesystem returns things by default. If you really,
> really care, define an InputSorter as mentioned above.
>


Most modern high performance file systems use some sort of sorted tree
structure inside of their directories, meaning that files are typically
returned in lexical order.

Most everything else just keeps a list, reusing empty slots, so you get
files back in the order they were created until you start deleting
files, and then all bets are off.

--
Shon
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.