Highlighted
idgandrewg Absent Member.
Absent Member.
3260 views

Very very slow file access iSCSI NSS on SLES11/XEN/OES11

Hi,

Like many Novell customers while carrying out a hardware refresh we are moving off traditional Netware 6.5 to OES11 and at the same time virtualising our environment.

We have new Dell Poweredge 620 serves attached by 10gig iSCSI to Equalogic SAN

Installed SLES will all patches and updates and XEN and then created OES11 SP2 virtual machines, these connect to NSS volume by iSCSI

Migrated files from traditional netware server to new hardware and stated testing and ran into very very slow files access times

A 3.5mb pdf file takes close to 10 minutes to open from local PC with Novell Client installed, same if no client and open via cifs. Opening same file off traditional NW6.5 server takes 3-4 seconds.

We have had a case open with Novell for almost 2 months but they are unable to resolve.

To test other options we installed VMWare ESXi on the internal usb flash drive and booted off that, created same OES11 VM and connected to NSS on SAN and same pdf open in seconds.

The current stack of SLES11/XEN/OES11 is not able to be put into production

Any ideas where the bottleneck might be? We think is in XEN.

Thanks
Labels (1)
Tags (3)
0 Likes
6 Replies
gleach1 Absent Member.
Absent Member.

Re: Very very slow file access iSCSI NSS on SLES11/XEN/OES11

last time I used XEN I had all sorts of issue with cpu usage and memory on my VM's - in the end we changed to vmware

i'd check the cpu and memory usage on your guests and make sure you aren't running into any issues

are you able to test mapping the LUN from the SAN as a virtual disk rather than directly using iSCSI?

novell were never to resolve my issues with XEN performance either hence why we moved to vmware and gave up on it

0 Likes
idgandrewg Absent Member.
Absent Member.

Re: Very very slow file access iSCSI NSS on SLES11/XEN/OES11

Memory and CPU are OK
ESXi as host with OES via iSCSI "works"
Booting without the XEN kernel and dd copying files is fast, same file copy when booted into XEN slow
Looks like we might have to go VMWare.
0 Likes
idgandrewg Absent Member.
Absent Member.

Re: Very very slow file access iSCSI NSS on SLES11/XEN/OES11

It looks like the offload settings are the problem on the host two interfaces p3p1 and p3p2

using ethtool -K I turned off rx tx sg tso gso gro lro rxvlan txvlan rxhash on p3p1 and p3p2 - FAST file access - very fast.

To test I turned back on rx tx sg tso gso gro lro rxvlan txvlan rxhash on p3p1 and p3p2 and files access SLOOOOOOOOOWWWWWWW
Turn off again and fast file access - able to repeat slow or fast file access by tuning p3p1 and p3p2 on or off with ethtool -K

I am told Interfaces p3p1 and p3p2 are enslaved interfaces in the bond configuration of br2. ?????

Waiting for support to tell me what the implications are of this finding and best way to fix
0 Likes
Knowledge Partner
Knowledge Partner

Re: Very very slow file access iSCSI NSS on SLES11/XEN/OES11

idgandrewg;2249362 wrote:
Waiting for support to tell me what the implications are of this finding and best way to fix


Hi,

As also mentioned in the SUSE forums, there is the option of using the Equallogic Hit Kit. One of the tools, next to the great autoconfigure options it has, is the eqltune tool.

Some of the stuff that I've found important:
-gro is known read performance killer. Switch if off on the iSCSI interfaces.
- if possible (meaning you have decent hardware), disable flowcontrol as this generally offers stability but at the cost of performance. If your hardware is decent, this form of traffic control should not be needed.
-To have multipath work correctly over iSCSI and starting SLES 11 SP1. Make sure kernel routing and arp handling are set correctly (not directly relevant if you only have 1 10 GB link):
net.ipv4.conf.[iSCSI interfaceX name].arp_ignore = 1
net.ipv4.conf.[iSCSI interfaceX name].arp_announce = 2
net.ipv4.conf.[iSCSI interfaceX name].rp_filter = 2

Test if traffic is actively routed over both iSCSI interfaces:
ping -I [iSCSI interfaceX name] [iSCSI Group IP EQL]

-Make sure network buffers etc are adequately set as recommended by Dell (set in /etc/sysctl.conf):
#NetEyes Increase network buffer sizes for iSCSI
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 8192 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.core.wmem_default = 262144
net.core.rmem_default = 262144

-Settings for the /etc/iscsi/iscsid.conf I'm using:
node.startup = automatic # <--- review and set according to environment
node.session.timeo.replacement_timeout = 60
node.conn[0].timeo.login_timeout = 15
node.conn[0].timeo.logout_timeout = 15
node.conn[0].timeo.noop_out_interval = 5
node.conn[0].timeo.noop_out_timeout = 5
node.session.err_timeo.abort_timeout = 15
node.session.err_timeo.lu_reset_timeout = 20 #Default is 30
node.session.err_timeo.tgt_reset_timeout=20 #Default is 30
node.session.initial_login_retry_max = 12 # Default is 4
node.session.cmds_max = 1024 #< --- Default is 128
node.session.queue_depth = 128 #< --- Default is 32
node.session.iscsi.InitialR2T = No
node.session.iscsi.ImmediateData = Yes
node.session.iscsi.FirstBurstLength = 262144
node.session.iscsi.MaxBurstLength = 16776192
node.conn[0].iscsi.MaxRecvDataSegmentLength = 131072 #A lower value improves latency at the cost of higher IO throughput
discovery.sendtargets.iscsi.MaxRecvDataSegmentLength = 32768
node.session.iscsi.FastAbort = No # < --- default is “Yes”


-Have Jumbo frames configured on the iSCSI interfaces & iSCSI switch.

If you are using multipathd instead of the dm-switch provided with the Equallogic Hit kit, make sure the /etc/multipath.conf holds the optimal settings for the Equallogic arrays.


Ever since Xen with SLES 11 SP1 we have been seeing strong performing virtual servers. We still use 1GB connections (two 1GB connections for each server, serving upto 180~190Mb/s).

There could be a difference with the 10GB setup, where multipath is not really needed or used (depending on the scale of your setup). One important thing is that the iSCSI switches are doing their thing correctly. But seeing you've already found better results tuning network parameters on the Xen host, seems to indicate that's ok.

Cheers,
Willem
0 Likes
idgandrewg Absent Member.
Absent Member.

Re: Very very slow file access iSCSI NSS on SLES11/XEN/OES11

Thanks very much Willem - will work on your recommendations
0 Likes
idgandrewg Absent Member.
Absent Member.

Re: Very very slow file access iSCSI NSS on SLES11/XEN/OES11

On both hosts edited the ifcfg file for the two interfaces that we had identified as causing the slow down - p3p1 and p3p2 and set ethtool_options.

This means the setting are now presistent after a reboot

barney # cd /etc/sysconfig/network/

barney:/etc/sysconfig/network # vi ifcfg-p3p1
BOOTPROTO='none'
BROADCAST=''
ETHTOOL_OPTIONS='-K iface rx off tx off sg off tso off gso off gro off lro off rxvlan off txvlan off rxhash off'
IPADDR=''
MTU=''
NAME='Intel Ethernet controller'
NETMASK=''
NETWORK=''
REMOTE_IPADDR=''
STARTMODE='auto'
USERCONTROL='no'
0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.