Application Delivery Management
Application Modernization & Connectivity
CyberRes
IT Operations Management
The purpose of this lab guide is to show, on a small scale, how a Teaming Clustered system is put together. The Lab will use a XEN Host, 2 Teaming (Tomcat) Servers, 2 Index Servers (Lucene), 1 MySQL Server. This lab assumes you know your way around SLES and XEN but provides significant guidance with OCFS2 and Linux Virtual Services.
See the Adv Installation section of the Teaming Installation Guide for more information
Install and Configure the Dom0: SLES11SP1 with XEN
The first step is to install the XEN host. A minimum of 8G of memory is recommended.
Install and configure XEN
Next Install OCFS2
Dom0:~# rpm -qa|grep ocfs2
ocfs2-kmp-default-1.4_2.6.32.12_0.6-4.10.14
ocfs2-tools-o2cb-1.4.3-0.11.19
ocfs2console-1.4.3-0.11.19
ocfs2-tools-1.4.3-0.11.19
With SLES11 and beyond; OCFS2 comes from the HA extension.
Create a partition – Do not format, or Mount
Dom0:~#chkconfig o2cb on
Dom0:~#chkconfig ocfs2 on
Now go ahead and format it using ocfs2:
Dom0:~# mkfs.ocfs2 -L xen_cluster /dev/sda4
mkfs.ocfs2 1.4.2
Cluster stack: classic o2cb
Filesystem label=xen_cluster
Block size=4096 (bits=12)
Cluster size=4096 (bits=12)
Volume size=22274056192 (5438002 clusters) (5438002 blocks)
169 cluster groups (tail covers 18994 clusters, rest cover 32256 clusters)
Journal size=139210752
Initial number of node slots: 8
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Writing backup superblock: 3 block(s)
Formatting Journals: done
Formatting slot map: done
Writing lost found: done
Formatting quota files: done
mkfs.ocfs2 successful
Run ocfs2console tool to add Dom0 node entry to cluster configuration (/etc/ocfs2/cluster.conf)
Cluster / Configure Nodes
This will create /etc/ocfs2/cluster.conf
We'll change it later when we add the DomUs.
Dom0:~# less /etc/ocfs2/cluster.conf
node:
ip_port = 7777
ip_address = xxx.xxx.137.23
number = 0
name = Dom0
cluster = ocfs2
cluster:
node_count = 1
name = ocfs2
Next run o2cb configure
Dom0:~ # /etc/init.d/o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
<ENTER> without typing an answer will keep that current value. Ctrl-C
will abort.
Load O2CB driver on boot (y/n) [y]:
Cluster stack backing O2CB [o2cb]:
Cluster to start on boot (Enter "none" to clear) [ocfs2]:
Specify heartbeat dead threshold (>=7) [31]:
Specify network idle timeout in ms (>=5000) [30000]:
Specify network keepalive delay in ms (>=1000) [2000]:
Specify network reconnect delay in ms (>=2000) [2000]:
Writing O2CB configuration: OK
Cluster ocfs2 already online
After running /etc/init.d/o2cb configure, here's the OCFS2 installed modules you should see:
Dom0:~#lsmod
ocfs2_dlmfs 20408 1
ocfs2_stack_o2cb 6792 0
ocfs2_dlm 170824 2 ocfs2_dlmfs,ocfs2_stack_o2cb
ocfs2_nodemanager 213800 11 ocfs2_dlmfs,ocfs2_stack_o2cb,ocfs2_dlm
ocfs2_stackglue 16312 1 ocfs2_stack_o2cb
configfs 41496 2 ocfs2_nodemanager
If the RPMs install fine but you do not see the modules,
verify that you have the correct versions.
further troubleshooting:
http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.2/01-cluster_start_stop.pdf
Watch dmesg for indication of problems... below the load was successful
Dom0~#dmesg
OCFS2 Node Manager 1.5.0
[ 21.829433] OCFS2 DLM 1.5.0
[ 21.855134] ocfs2: Registered cluster interface o2cb
[ 21.875734] OCFS2 DLMFS 1.5.0
[ 21.876109] OCFS2 User DLM kernel interface loaded
[ 23.561971] BIOS EDD facility v0.16 2004-Jun-25, 1 devices found
[ 25.226601] alloc irq_desc for 448 on node 0
[ 25.226604] alloc kstat_irqs on node 0
[ 25.231692] alloc irq_desc for 449 on node 0
[ 25.231692] alloc kstat_irqs on node 0
[ 25.231708] suspend: event channel 41
[ 399.404878] ocfs2_dlm: Nodes in domain ("866418CE7D6C44938A340EAD9CADFC32"): 0
To summarize: We've created an unformatted partition. Using that partition we've created a cluster of 1 ( the Dom0). When we build out the DomUs they will be added to the cluster. We'll revisit the ocfs2 config at that time.
Install 5 VMs with SLES11SP1 - carve up the memory appropriately i.e. Dom0 8G / 1G / 1G
The 2 servers Teaming 1 and Teaming 2 need access to the same disk. In the real world this might be a SAN to store Terabytes of content. Teaming allows for more than 2 Tomcat servers but for the simplicity of this lab, 2 will prove the concept.
Once the Base OS is installed on Teaming 1 (VM) and Teaming 2 (VM), go back to the Dom0 (Host) and add in the shared disk.
Shutdown the VMs (Teaming 1 and Teaming 2)
On the Host Dom0, edit the /etc/xen/vm/DomU file to add 'phy:/dev/sda4,hdd,w!'
disk=[ 'file:/images/DomU1/disk0,xvda,w', 'phy:/dev/sda4,hdd,w!' ]
Note the "!" this is very important. Do not omit this!
Force the changes to be implemented. Starting and stopping the VMs is not enough
Dom0~#cd /etc/xen/vm
Dom0~#xm delete Teaming1
Dom0~#xm create ./Teaming1
Repeat for Teaming2
Note this procedure does NOT delete the image. It just deletes (and recreates the XEN DB config) the meta data that is associated to it.
Check it from the VM:
Teaming1:~ # hwinfo –disk
05: None 00.0: 10600 Disk
[Created at block.243]
UDI: /org/freedesktop/Hal/devices/computer_storage_unknown_disk
Unique ID: b7l.Fxp0d3BezAE
Parent ID: cBJm.hKT3jlotBo5
SysFS ID: /class/block/xvda
SysFS BusID: vbd-51712
SysFS Device Link: /devices/xen/vbd-51712
Hardware Class: disk
Model: "Disk"
Driver: "vbd"
Driver Modules: "xenblk"
Device File: /dev/xvda
Device Number: block 202:0-202:15
Geometry (Logical): CHS 1566/255/63
Size: 25165824 sectors a 512 bytes
Config Status: cfg=no, avail=yes, need=no, active=unknown
Attached to: #2 (Storage controller)
06: None 00.0: 10600 Disk
[Created at block.243]
UDI: /org/freedesktop/Hal/devices/computer_storage_unknown_disk_0
Unique ID: aQVk.Fxp0d3BezAE
Parent ID: caE1.hKT3jlotBo5
SysFS ID: /class/block/hdd
SysFS BusID: vbd-5696
SysFS Device Link: /devices/xen/vbd-5696
Hardware Class: disk
Model: "Disk"
Driver: "vbd"
Driver Modules: "xenblk"
Device File: /dev/hdd
Device Files: /dev/hdd, /dev/disk/by-uuid/3807d378-466f-439b-b40d-a0ee1e4a1b04, /dev/disk/by-label/xen_cluster
Device Number: block 22:64-22:127
Geometry (Logical): CHS 2708/255/63
Size: 43504020 sectors a 512 bytes
Config Status: cfg=new, avail=yes, need=no, active=unknown
Attached to: #3 (Storage controller)
On the Teaming1 and Teaming2 Servers:
Modify the /etc/hosts and include the VM name as well as the host name for the other teaming DomU servers. Even though the /etc/ocfs2/cluster.conf includes an ip address the names that are listed for each ocfs2 node need to resolve to the appropriate ip address.
Install the ocfs2 modules:
Follow the prompt to reboot the Teaming1 and 2 VMs
Teaming1:`#mkdir /etc/ocfs2/
create a new cluster.conf
node:
ip_port = 7777
ip_address = xxx.xxx.137.58
number = 1
name = Teaming1
cluster = ocfs2
node:
ip_port = 7777
ip_address = xxx.xxx.137.59
number = 2
name = teaming2
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
copy the cluster.conf to the other DomU (Teaming2)
on both Teaming Servers:
Teaming1:~#chkconfig o2cb on
Teaming1:~#chkconfig ocfs2 on
Teaming1:~#/etc/init.d/o2cb configure
Add shared disk to fstab
Teaming1:#less /etc/fstab
/dev/xvda1 swap swap defaults 0 0
/dev/xvda2 / ext3 acl,user_xattr 1 1
proc /proc proc defaults 0 0
sysfs /sys sysfs noauto 0 0
debugfs /sys/kernel/debug debugfs noauto 0 0
devpts /dev/pts devpts mode=0620,gid=5 0 0
/dev/hdd /ocfs2 ocfs2 _netdev 0 0
Teaming1:~#mkdir /ocfs2
Teaming1:~#mount -t ocfs2 /dev/hdd /ocfs2
Teaming1:~#cd /ocfs2
Teaming1:~#touch success
change to Teaming2
Teaming2:~#mkdir /ocfs2
Teaming2:~#mount -t ocfs2 /dev/hdd /ocfs2
Teaming2:~#cd /ocfs2
Teaming2:~#ls -l
total 4
drwxr-xr-x 3 root root 3896 2010-05-19 14:54 ./
drwxr-xr-x 23 root root 4096 2010-05-19 14:34 ../
drwxr-xr-x 2 root root 3896 2010-05-18 16:31 lost found/
-rw-r--r-- 1 root root 0 2010-05-19 14:54 success
That completes the OCFS2 configuration portion.
Use the Teaming Installation Guide to build out the teaming systems. Look for Multi-Server Configurations and Clustering in the installation guide.
Be sure to use the newly created shared disk in the "Data location" rather than the defaults on the teaming1 and teaming 2 installs.
Also for the simplicity ( of this lab) , use port 80 for the tomcat service (run as root).
Note this is a significant security hole. Do not run Tomcat as "root" on your production system.
Select a Virtual IP ADDR or DNS Host for the clustered Teaming/Tomcat servers. To the Browser the clustered system needs to appear as one server.
Test the Teaming1 and 2 servers by accessing them directly to make sure they are functional before proceeding
See http://www.ultramonkey.org/papers/lvs_tutorial/html for help with LVS.
The Linux Virtual Server has three different ways of forwarding packets: Network Address Translation (NAT), IP-IP encapsulation or tunnelling and Direct Routing.
For this guide, I chose:
Direct Routing: Packets from end users are forwarded directly to the real server. The IP packet is not modified, so the real servers must be configured to accept traffic for the virtual server's IP address. This can be done using a dummy interface, or packet filtering to redirect traffic addressed to the virtual server's IP address to a local port. The real server may send replies directly back to the end user. That is if a host based layer 4 switch is used, it may not be in the return path.
First thing we need to accomplish is to add a second IP address to be used as the one stop shop address for all of the Teaming Tomcat servers. This is the virtual address that was referenced above. While it's relatively easy to add a secondary address to none XEN host, it's a bit tricky with a XEN bridge.
Here's the XEN host:
Work around :On the Non XEN host (with a secondary addr) take a look at /etc/sysconfig/network/ifcfg-eth0
copy over the 3 lines that it added to the end of XEN host's /etc/sysconfig/network/ifcfg-br0.
IPADDR_L4_addr='131.156.137.33'
NETMASK_L4_addr='255.255.252.0'
LABEL_L4_addr='L4_addr'
Dom0:~ #rcnetwork restart
Shutting down network interfaces:
br0 done
eth0 device: Intel Corporation 82566DM Gigabit Network Connection (rev 02)
No configuration found for eth0
Nevertheless the interface will be shut down. done
vif13.0
No configuration found for vif13.0
Nevertheless the interface will be shut down. done
vif17.0
No configuration found for vif17.0
Nevertheless the interface will be shut down. done
Shutting down service network . . . . . . . . . done
Setting up network interfaces:
eth0 device: Intel Corporation 82566DM Gigabit Network Connection (rev 02)
No configuration found for eth0 unused
vif13.0
No configuration found for vif13.0 unused
vif17.0
No configuration found for vif17.0 unused
br0
br0 Ports: [eth0]
br0 forwarddelay (see man ifcfg-bridge) not ready. Proceeding in background.
br0 IP address: xxx.xxx.xxx.xxx/22
br0:L4_addr IP address: xxx.xxx.xxx.xxx/22
Dom0:/etc/xen/vm # ip addr
br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 00:1c:c0:ed:43:09 brd ff:ff:ff:ff:ff:ff
inet xxx.xxx.xxx.xxx/22 brd xxx.xxx.xxx.xxx scope global br0
inet xxx.xxx.xxx.xxx/22 brd xxx.xxx.xxx.xxx scope global secondary br0:L4 _addr
inet6 fe80::21c:c0ff:feed:4309/64 scope link
valid_lft forever preferred_lft forever
Use yast2 software management to add ipvsadm
Dom0:~#ipvsadm
FATAL: Module ip_vs not found.
Can't initialize ipvs: Protocol not available
Are you sure that IP Virtual Server is built in the kernel or as module?
Doh ...missing one... using yast install: cluster-network-kmp-default
reboot the Dom0
Dom0:~#ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
Config the Virtual L4 switch (LVS)
Dom0:~#ipvsadm -A -t 131.156.137.33:80 -p -s rr #listen on 80, make the connection sticky, Scheduler:use round robin load balancing
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 131.156.137.33:http rr persistent 360
Config the DomU servers that will back end the LVS L4 switch:
ipvsadm -a -t 131.156.137.30:80 -r 131.156.137.31:80 -g
-a --add-server Add a real server to a virtual service. -t, --tcp-service service-address Use TCP service. -g, --gatewaying Use gatewaying (direct routing). -r, --real-server server-address
ipvsadm -a -t 131.156.137.30:80 -r 131.156.137.32:80 -g
Dom0~#ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 131.156.137.30:http rr persistent 360
-> 131.156.137.31:http Route 1 0 0
-> 131.156.137.32:http Route 1 0 0
This config will be gone with a reboot. Therefore it should be copied to a startup script in /etc/init.d/
create configuration: ipvsadm -S > /etc/ipvsadm.save
restore config in init script: ipvsadm -R < /etc/ipvsadm.save
On the Host Dom0: Enable IP forwarding. This can be done by adding the following
net.ipv4.ip_forward = 1
to /etc/sysctl.conf and then running "sysctl -p"
Now on the Teaming/Teaming (DomU) servers
Bring up 131.156.137.30 on the loopback interface. This is best done as part of the networking configuration of your system. But it can also be done manually. On Linux this can be done using the following command.
ifconfig lo:0 131.156.137.30 netmask 255.255.255.255 # note the ff.ff.ff.ff mask this is not a mistake
DomU1:~ # ifconfig
eth0 Link encap:Ethernet HWaddr 00:16:3E:49:3E:79
inet addr:xxx.xxx.xxx.xxx Bcast:xxx.xxx.xxx.255 Mask:255.255.252.0
inet6 addr: fe80::216:3eff:fe69:3e79/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:135213 errors:0 dropped:0 overruns:0 frame:0
TX packets:10121 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:19157728 (18.2 Mb) TX bytes:926086 (904.3 Kb)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:2 errors:0 dropped:0 overruns:0 frame:0
TX packets:2 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:100 (100.0 b) TX bytes:100 (100.0 b)
lo:0 Link encap:Local Loopback
inet addr:131.156.137.30 Mask:255.255.255.255
UP LOOPBACK RUNNING MTU:16436 Metric:1
repeat on the other teaming server.
Start up the servers:
Lucene Node 1
Lucene Node 2
MySQL
Teaming 1
Teaming 2
Access the virtual address that you defined above with the browser
Success if you are able to connect and login.
You can monitor the load balancer connection:
Dom0:~# ipvsadm
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 131.156.137.30:http rr persistent 360
-> 131.156.137.31:http Route 1 0 1
-> 131.156.137.31:http Route 1 0 0