GroupWise 7 and OES2/SLES10 File Systems

0 Likes
over 13 years ago

Problem



During BrainShare 2008, the most common question that the GroupWise team and the GroupWise SysOps were asked was, "What file system should I use for Linux?" This article explains the choices and why I reach the conclusion that I do ...

Solution



There are 4 (or 5) choices of file system for GroupWise 7 on SLES10 or OES2:

- XFS
- EXT3 (and EXT3 plus H-Tree)
- ReiserFS
- NSS

Let's look at these in turn and then explain their advantages and disadvantages.

XFS

XFS is extremely fast, but it uses some very aggressive caching to achieve the throughput. It's questionable in its management. For example, try formatting a USB stick with XFS, then copy a large file to the USB stick. When the copy has finished, umount the drive. When you next plug the USB stick in, is the file readable? (I have failed to have the copy complete in 10 test attempts.) This makes it totally unsuitable for a cluster, and probably too vulnerable for a standalone GroupWise 7 Server. Discounted.

EXT3

EXT3 is slow without the H-Tree and so is discounted.

With H-Tree, EXT3 becomes a very strong performer. However, there is a price to pay for the increased performanace. GroupWise uses telldir(), seekdir(), and readdir(), in the calling of files, all of which return a cookie that is nominally the position in the file. It's unfortunately based an assumption that was true when the interface was designed but is not currently true. The assumption is that directories will always be laid out linearly, so a file offset will be enough to identify the directory entry. Unfortunately, this offset is a signed int, where only positive values are valid. This translates to a cookie size of 31 bits in which to uniquely identify the position.

The problem arises with ext3 hashed directories, because they use a 64-bit hash that consists of a 32-bit major and a 32-bit minor hash. Ext3 returns the major hash, which is additionally truncated by one bit. This not only eliminates the collision handling that the kernel handles internally, but also creates even more collisions for telldir(). Discounted.

So, now we are down to a 2 horse race, ReiserFS Vs NSS.

NSS and ReiserFS

In the original OES, based on SLES9, the performance of NSS was severely lacking in comparison to ReiserFS, having only some 80% or less of the performance of Reiser. With the new SLES10-based OES2, that degradation in performance of NSS compared to ReiserFS appears to have been reduced to a more acceptable 5% or so – so long as the volume is created with Salvage disabled. For a long time, ReiserFS has had a reputation of being quick but fragile, and when the file system tree has to be rebuilt, you are more likely to get a few sticks and a pile of leaves than a whole tree. NSS rebuilds are remarkably complete (as one would expect from a file-server file system); therefore, NSS provides an excellent alternative to ReiserFS.

Looking at telldir() and ReiserFS, ReiserFS doesn't display the same problem because it uses a much smaller hash space. It has a 32-bit total, where bits 7-30 describe the hash, and bits 0-6 describe the generation number that handles collisions. Because the last bit is unused, ReiserFS doesn't run into problems with telldir(). The trade-off is that ReiserFS supports a small hash space with a maximum of 127 collisions, so it's much more prone to spuriously returning -EBUSY when the maximum number of collisions has been reached. I expect that XFS could have the same problem as EXT3 plus H-Tree, since it uses 64-bit offsets that end up getting truncated.

In conclusion there are really just 2 choices. The most performing (stable) system is ReiserFS, which should be used when every cycle is critical - but beware of overloading the system. The best compromise between speed and resilience is NSS with Salvage disabled.

Based on the above, I would recommend that the default file system to be used for a GroupWise system running on OES2 is NSS.

I hope this helps some folk with the sleepless nights.

Labels:

How To-Best Practice
Comment List
Anonymous
  • The biggest downfall I see with Novell people is an assumption that Novell software is automatically superior to anything else.

    LVM is at least as powerful as NSS pools, you can extend volumes to free space on existing disk, or add new disk and extend the volume group to include the new disk. You can also do volume level snapshots.

    Reiser and XFS partitions can be extended on the fly to grow to the size of the bigger volume. Ext3 will require unmounting, extending and remounting.

    EVMS is even more powerful, but I'm not as familiar with that as I am with LVM.
  • I was one of the many who asked you this very question at BrainShare -

    I am setting up my OES2 servers now and was curious if you have any additional recommendations along with no salvage. Should I stick with the Unix namespace? Disable oplocks?

    Are there any issues with running the agents as a non-root user on NSS?
  • Rest easy - you have to activate it (on SLES9 it wasn't even supported)

    To enable H-Tree look up dir_index

    T
  • So that begs the question. How do I check to see if HTree for my Ext3 partitions is enabled or not?

    when created an EXT3 filesystem in SLES10 is htree automatically enabled?

  • Believe it or not, this is a very interesting topic in the hallways of the Workgroup business unit these days. Both Alex Evans (GroupWise PM) and I have fielded dozens of questions and comments about the choice of file systems when running GroupWise on Open Enterprise Server 2. Clearly, NSS is a great file system, and its performance on OES2 is far superior to OES1, but this is not a one-size-fits-all issue.

    We know that our customers would like Novell's guidance on this. We are starting to articulate use cases that would provide a reference for customers of both GroupWise and OES2 on Linux when deciding which file system to use: NSS, ext3 or Reiser. We think that with the upcoming release of OES2 SP1 and Bonsai, it's a good time to get this down on paper (or up on a wiki) and out to our customers.

    Please keep your eyes open for that in the near future.

    Regards, Jason Williams & Alex Evans
  • I'm a Volvo man my self :-)

    I believe that they will be forthcoming from Novell later this year - I was pushing hard at BrainShare.

    T
  • On OES1 there was only one choice - Reiser.

    The whole point of the article is that the release of SLES10/OES2 has changed the dynamics :-)

    NSS when OES was first released was like a 2 legged dog walking backwards - terrible (shudder) but the changes since then have revolutionised the performance.

    Unless you are about to redeploy on OES2 - stay where you are.

    T
  • I'm not the man to go into file systems - but if you look at all of the systems in use today, the only one that really requires to be defragmented on a regular basis in NTFS - it really is a poor piece of design in my opinion.

    A Windows server is the slowest of a NetWare, Linux, Windows trio with the fastest now being the Linux box.

    If you look at SAMBA a Linux server can handle more clients than a windows server!

    I don't think this answers your question, but anecdotal evidence would imply that it is slower...

    HTH
  • Ext3 is slow with greater than 512 files in a folder.

    Ext3 plus H-Tree is fast - but can return the wrong file (not corruption) which is bad for an email system.

    I like Ext3 and use it as my preferred file system, but in this case because of the way that the GW flaim database works, the performance drops off drastically once the 512 files limit is reached.

    I believe that Novell will be doing some benchmark tests on the different file systems soon - I was pushing like heck for them at BrainShare :-)

    T
  • I would have preferred to use Ext3 my self - but without H-Tree when each folder gets to more than 512 files the performance drops off drastically - ask Steven Tweedy (He lives here in Edinburgh)

    As H-Tree and telldir() are broken (in their interaction) we can't speed it up. I am looking forward to Ext4 but then again, we have been doing that for the past decade :-)

    As the saying goes YMMV

    Have fun

    T
Related Discussions
Recommended