High Availability Storage with iSCSI Target on Debian 8

Software

  • Linux-HA – Linux clustering software.
  • DRBD – Distributed Replicated Block Device. Allows you to RAID1 partitions over IP.
  • iscsitarget – Linux implementation of an iSCSI target.

 

Configuration

This guide is based on the following:

  • Two nodes (Debian 8.5 AMD64)
  • Each node has 2x NICs (1x on network and 1x for DRBD data).
  • Nodes:
    • san01 (“node1”) / 192.168.0.242 / eth0
      • DRBD sync network: node1-drbd / 10.50.40.21 / eth1
    • san02 (“node2”) / 192.168.0.243 / eth0
      • DRBD sync network: node2-drbd / 10.50.40.22 / eth1
  • Cluster IP address: 192.168.0.245

Note: Unless explicitly stated (i.e. commands prefixed with [node1] or [node2]), commands and configurations should be completed on both nodes.

LVM

We install lvm2 and create a VG.

apt-get install -y lvm2
pvcreate  /dev/sdb

Physical volume “/dev/sdb” successfully created

vgcreate drbddev01 /dev/sdb

Volume group “drbddev01” successfully created

Create DRBD meta data Logical Volume on Volume Group drbddev01:

lvcreate -L1G -ndrbd-metadata drbddev01

Logical volume “drbd-metadata” created

Create DRBD config Logical Volume on Volume Group drbddev01:

lvcreate -L1G -niscsi-config drbddev01

Logical volume “iscsi-config” created

Create a Logical Volume to become a test LUN later on:

lvcreate -l100%FREE -nlun.test drbddev01

Logical volume “lun.test” created

Hosts File

Edit /etc/hosts (removing the loopback entry for the host):

192.168.0.242   san01.lplinux.com.ar    san01
192.168.0.243   san02.lplinux.com.ar    san02
10.50.40.21     iscsinode01.lplinux.com.ar      iscsinode01
10.50.40.22     iscsinode02.lplinux.com.ar      iscsinode02

DRBD & Heartbeat

Install packages drbd8-utils and heartbeat.

apt-get install drbd8-utils heartbeat

Change permissions and group ownership on some DRBD binaries for use with heartbeat:

chgrp haclient /sbin/drbdsetup
chmod o-x /sbin/drbdsetup
chmod u+s /sbin/drbdsetup
chgrp haclient /sbin/drbdmeta
chmod o-x /sbin/drbdmeta
chmod u+s /sbin/drbdmeta

Edit /etc/drbd.conf and define two resources:

  1. The DRBD device that will contain iscsitarget configuration files.
  2. The DRBD device that will become the test LUN.
global {
usage-count no;
}

resource iscsi.config {
protocol C;

handlers {
pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
}

startup {
degr-wfc-timeout 120;
}

disk {
on-io-error detach;
}

net {
cram-hmac-alg sha1;
shared-secret "password";
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}

syncer {
rate 100M;
verify-alg sha1;
al-extents 257;
}

on san01 {
device  /dev/drbd0;
disk    /dev/drbddev01/iscsi-config;
address 10.50.40.21:7788; # Use DRBD dedicated network
meta-disk /dev/drbddev01/drbd-metadata[0];
}

on san01 {
device  /dev/drbd0;
disk    /dev/drbddev01/iscsi-config;
address 10.50.40.22:7788; # Use DRBD dedicated network
meta-disk /dev/drbddev01/drbd-metadata[0];
}
}

resource iscsi.lun.test {
protocol C;

handlers {
pri-on-incon-degr "echo o > /proc/sysrq-trigger ; halt -f";
pri-lost-after-sb "echo o > /proc/sysrq-trigger ; halt -f";
local-io-error "echo o > /proc/sysrq-trigger ; halt -f";
outdate-peer "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
}

startup {
degr-wfc-timeout 120;
}

disk {
on-io-error detach;
}

net {
cram-hmac-alg sha1;
shared-secret "password";
after-sb-0pri disconnect;
after-sb-1pri disconnect;
after-sb-2pri disconnect;
rr-conflict disconnect;
}

syncer {
rate 100M;
verify-alg sha1;
al-extents 257;
}

on san01 {
device  /dev/drbd1;
disk    /dev/drbddev01/lun.test;
address 10.50.40.21:7789; # Use private inter-node address
meta-disk /dev/drbddev01/drbd-metadata[1];
}

on san01{
device  /dev/drbd1;
disk    /dev/drbddev01/lun.test;
address 10.50.40.22:7789; # Use private inter-node address
meta-disk /dev/drbddev01/drbd-metadata[1];
}
}

Reboot nodes. Test connectivity (both networks) between nodes.

Initialise DRBD meta data discs for the DRBD resources. This needs to be done on both nodes:

drbdadm create-md iscsi.config
drbdadm create-md iscsi.lun.test

Restart DRBD service.

service drbd restart

Decide which node will act as the primary for the DRBD device that will contain the iSCSI configuration files (/dev/drbd0) and initiate the first full sync between the nodes. Run the following on the primary:

[node1] #

drbdadm -- --overwrite-data-of-peer primary iscsi.config
cat /proc/drbd

You can wait until the initial sync completes but it’s not a requirement.

Testing

Create a filesystem on /dev/drbd0 (iSCSI configs) and mount it:

[node1] #

mkfs.ext4 /dev/drbd0
mkdir -p /srv/iscsi-config
mount /dev/drbd0 /srv/iscsi-config

Create the /srv/iscsi-config mount point on node 2.

Ensure replication is working as expected. On the primary node:

[node1] #

dd if=/dev/zero of=/srv/iscsi-config/test.bin bs=1M count=9
umount /srv/iscsi-config
drbdadm secondary iscsi.config

On node 2:

[node2] #

drbdadm primary iscsi.config
mount /dev/drbd0 /srv/iscsi-config
ls -l /srv/iscsi-config

Test replication the other way by deleting the file:

[node2] #

rm /srv/iscsi-config/test.bin
umount /srv/iscsi-config
drbdadm secondary iscsi.config

Make node 1 the primary and mount /srv/iscsi-config (/dev/drbd0) and ensure the file has gone:

[node1] #

drbdadm primary iscsi.config
mount /dev/drbd0 /srv/iscsi-config
ls -l /srv/iscsi-config

Decide which node will act as the primary for the DRBD device that contains the test LUN (/dev/drbd1) and initiate the first full sync between the nodes. Run the following on the primary:

[node1] #

drbdadm -- --overwrite-data-of-peer primary iscsi.lun.test

Configuring iScsi

Install the iscsitarget package. By default, iscsitarget (ietd) will not start.

apt-get install -y iscsitarget

Edit /etc/defaults/iscsitarget and set ISCSITARGET_ENABLE to true.

Heartbeat will be used to control the iscsitarget service so remove it from init:

update-rc.d -f iscsitarget remove

Relocate iscsitarget config to DRBD device. Make sure that node 1 is the primary and that /srv/iscsi-config is mounted:

[node1] #

drbdadm primary iscsi.config
mount /dev/drbd0 /srv/iscsi-config
mv /etc/iet/ietd.conf /srv/iscsi-config
ln -s /srv/iscsi-config/ietd.conf /etc/iet/ietd.conf

[node2] #

rm /etc/iet/ietd.conf
ln -s /srv/iscsi-config/ietd.conf /etc/iet/ietd.conf

Create iscsitarget config on node 1. Example:

vim /etc/iet/ietd.conf

and put this block

Target iqn.1998-04.com.domain:lun.test
Lun 0 Path=/dev/drbd1,Type=blockio,ScsiSN=291109213201
Alias lun.test
HeaderDigest None
DataDigest None
MaxConnections 1
InitialR2T Yes
ImmediateData No
MaxRecvDataSegmentLength 8192
MaxXmitDataSegmentLength 8192
MaxBurstLength 262144
FirstBurstLength 65536
DefaultTime2Wait 2
DefaultTime2Retain 20
MaxOutstandingR2T 8
DataPDUInOrder Yes
DataSequenceInOrder Yes
ErrorRecoveryLevel 0

Configuring Heartbeat

Configure heartbeat to control virtual IP address of cluster and to failover iscsitarget when a node fails. The following should be completed on node 1:

vim /etc/ha.d/ha.cf
logfacility     local0

autojoin        none # All nodes are defined explicitly.
auto_failback   no # Prevents nodes from flapping.

keepalive       2
deadtime        10
warntime        5
initdead        120

mcast           eth0 239.0.0.1 694 1 0 # Shared network, so multicast heartbeats.
bcast           eth1 # DRBD network is private, so we can use broadcasts.

node            san01
node            san02

respawn         hacluster /usr/lib/heartbeat/ipfail
ping            192.168.0.220 # Ping a core network device to assist in determining network link status.
vim /etc/ha.d/authkeys
auth 3
3 md5 password
vim /etc/ha.d/haresources
san01 drbddisk::iscsi.config Filesystem::/dev/drbd0::/srv/iscsi-config::ext4
san01 IPaddr2::192.168.0.245/24/eth0 drbddisk::iscsi.lun.test portblock::tcp::3260::block iscsitarget portblock::tcp::3260::unblock
chmod 600 /etc/ha.d/authkeys

Copy ha.cf, authkeys and haresources to node 2:

[node1] #

scp /etc/ha.d/ha.cf root@san02:/etc/ha.d
scp /etc/ha.d/authkeys root@san02:/etc/ha.d
scp /etc/ha.d/haresources root@san02:/etc/ha.d

Note: At the time of writing, the portblock resource agent script (/etc/ha.d/resource.d/portblock) is broken. Ubuntu bug #489719 has been filed, along with Debian bug #538987. Apply the following patch to both nodes:

vim /tmp/portblock.patch

Paste this block:

--- portblock.orig    2009-11-28 20:03:57.964375908 +0000
+++ portblock    2009-11-28 20:04:13.264550812 +0000
@@ -17,14 +17,14 @@
exit 1
}

-if [ $# != 3 ]; then
+if [ $# != 4 ]; then
usage
fi

OCF_RESKEY_protocol=$1
OCF_RESKEY_portno=$2
OCF_RESKEY_action=$3
-export OCF_RESKEY_action OCF_RESKEY_portno OCF_RESKEY_action
+export OCF_RESKEY_action OCF_RESKEY_portno OCF_RESKEY_protocol

OCF_TYPE=portblock
OCF_RESOURCE_INSTANCE=${OCF_TYPE}_$1_$2_$3

And then patch the file:

patch /etc/ha.d/resource.d/portblock /tmp/portblock.patch

Finally, reboot both nodes and test failover. The best way to do this is to connect the test LUN to a server, copy on a movie and play it. Fail one of the nodes either by pulling the power or via ”/etc/init.d/heartbeat stop”. The movie will freeze for a few seconds but should resume. Also tail /var/log/syslog.

 

Reference Links:

  • https://0wned.it/geek-bits/guides/high-availability-iscsi-target-using-linux/
Print Friendly, PDF & Email

Pablo Javier Furnari

Linux System Administrator at La Plata Linux
I'm a Linux Sysadmin with 8 years of experience. I work with several clients as a consulter here in Argentina and oversea (I have clients in the United States, Mexico, Pakistan and Germany).

I know my strengths and weaknesses. I'm a quick learner, I know how to work with small and big teams. I'm hard worker, proactive and I achieve everything I propose.

Leave a Reply

Your email address will not be published. Required fields are marked *


CAPTCHA Image
Reload Image