ISCSI Initiator with Linux

iSCSI with jumbo-frames and multipathing

Configure networking

LACP-bonding for the frontend

/etc/netplan/bond0.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    eno1:
      dhcp4: false
      dhcp6: false
      optional: true
    eno2:
      dhcp4: false
      dhcp6: false
      optional: true
  bonds:
    bond0:
     interfaces:
     - eno1
     - eno2
     parameters:
       lacp-rate: slow
       mode: 802.3ad
       transmit-hash-policy: layer2
     addresses:
     - 10.71.112.135/16
     gateway4: 10.71.101.1 
     nameservers:
       addresses:
       - 10.71.111.11
       - 10.71.111.12
       search:
       - domain.de

Two dedicated 10GE interfaces with jumbo-frames for the backend

/etc/netplan/iscsi.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    enp132s0f0:
      dhcp4: false
      dhcp6: false
      mtu: 9000
      addresses:
        - 10.250.71.32/24
      set-name: iscsi0
      match:
        macaddress: a0:36:9f:d4:cd:1a
    enp132s0f1:
      dhcp4: false
      dhcp6: false
      mtu: 9000
      addresses:
        - 10.251.71.32/24
      set-name: iscsi1
      match:
        macaddress: a0:36:9f:d4:cd:18

Apply the parameters and check settings

# netplan apply
# ip a sh iscsi0
7: iscsi0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default
qlen 1000
link/ether a0:36:9f:d4:cd:1a brd ff:ff:ff:ff:ff:ff
inet 10.250.71.32/24 brd 10.250.71.255 scope global iscsi0
valid_lft forever preferred_lft forever
inet6 fe80::a236:9fff:fed4:cd1a/64 scope link
valid_lft forever preferred_lft forever

# ip a sh iscsi1
5: iscsi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default
qlen 1000
link/ether a0:36:9f:d4:cd:18 brd ff:ff:ff:ff:ff:ff
inet 10.251.71.32/24 brd 10.251.71.255 scope global iscsi1
valid_lft forever preferred_lft forever
inet6 fe80::a236:9fff:fed4:cd18/64 scope link
valid_lft forever preferred_lft forever

# ip a sh bond0
12: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
group default qlen 1000
link/ether 32:2d:f2:c0:e2:3f brd ff:ff:ff:ff:ff:ff
inet 10.71.112.135/16 brd 10.71.255.255 scope global bond0
valid_lft forever preferred_lft forever
inet6 fe80::302d:f2ff:fed0:e23f/64 scope link
valid_lft forever preferred_lft forever

Check if all components are configured right for jumbo-frames

# ping -c 3 -M do -s 8972 -I iscsi0 10.250.71.1
PING 10.250.71.1 (10.250.71.1) from 10.250.71.32 iscsi0: 8972(9000) bytes of data.
8980 bytes from 10.250.71.1: icmp_seq=1 ttl=64 time=0.227 ms
8980 bytes from 10.250.71.1: icmp_seq=2 ttl=64 time=0.187 ms
8980 bytes from 10.250.71.1: icmp_seq=3 ttl=64 time=0.198 ms
--- 10.250.71.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2045ms
rtt min/avg/max/mdev = 0.187/0.204/0.227/0.016 ms

# ping -c 3 -M do -s 8972 -I iscsi1 10.251.71.1
PING 10.251.71.1 (10.251.71.1) from 10.251.71.32 iscsi1: 8972(9000) bytes of data.
8980 bytes from 10.251.71.1: icmp_seq=1 ttl=64 time=0.202 ms
8980 bytes from 10.251.71.1: icmp_seq=2 ttl=64 time=0.195 ms
8980 bytes from 10.251.71.1: icmp_seq=3 ttl=64 time=0.191 ms
--- 10.251.71.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2055ms
rtt min/avg/max/mdev = 0.191/0.196/0.202/0.004 ms

If it does not has success, one of the switches on the way or the iSCSI-storage misses jumbo-frame settings.

Configure iSCSI

Setup initiator iqn

Setup a new iqn:

# /sbin/iscsi-iname

The result is in /etc/iscsi/initiatorname.iscsi

/etc/iscsi/initiatorname.iscsi

## DO NOT EDIT OR REMOVE THIS FILE!
## If you remove this file, the iSCSI daemon will not start.
## If you change the InitiatorName, existing access control lists
## may reject this initiator.  The InitiatorName must be unique
## for each iSCSI initiator.  Do NOT duplicate iSCSI InitiatorNames.
InitiatorName=iqn.1993-08.org.debian:01:4efdaa48c123

Setup iSCSI-Interfaces

# iscsiadm -m iface -I iscsi0 -o new
# iscsiadm -m iface -I iscsi0 --op=update -n iface.net_ifacename -v iscsi0
# iscsiadm -m iface -I iscsi0
# BEGIN RECORD 2.0-874
iface.iscsi_ifacename = iscsi0
iface.net_ifacename = iscsi0
iface.ipaddress = <empty>
iface.hwaddress = <empty>
iface.transport_name = tcp
...
# END RECORD

# iscsiadm -m iface -I iscsi1 -o new
# iscsiadm -m iface -I iscsi1 --op=update -n iface.net_ifacename -v iscsi1
# iscsiadm -m iface -I iscsi1
# BEGIN RECORD 2.0-874
iface.iscsi_ifacename = iscsi1
iface.net_ifacename = iscsi1
iface.ipaddress = <empty>
iface.hwaddress = <empty>
iface.transport_name = tcp
...
# END RECORD

Discover LUNs that are offered by the storage

# iscsiadm -m discovery -t st -p 10.250.71.1
iscsiadm: cannot make connection to 10.250.71.1: No route to host
iscsiadm: cannot make connection to 10.250.71.1: No route to host
iscsiadm: cannot make connection to 10.250.71.1: No route to host
iscsiadm: cannot make connection to 10.250.71.1: No route to host
iscsiadm: cannot make connection to 10.250.71.1: No route to host
iscsiadm: cannot make connection to 10.250.71.1: No route to host
iscsiadm: connection login retries (reopen_max) 5 exceeded
10.250.71.1:3260,1 iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1

# iscsiadm -m discovery -t st -p 10.251.71.1
iscsiadm: cannot make connection to 10.251.71.1: No route to host
iscsiadm: cannot make connection to 10.251.71.1: No route to host
iscsiadm: cannot make connection to 10.251.71.1: No route to host
iscsiadm: cannot make connection to 10.251.71.1: No route to host
iscsiadm: cannot make connection to 10.251.71.1: No route to host
iscsiadm: cannot make connection to 10.251.71.1: No route to host
iscsiadm: connection login retries (reopen_max) 5 exceeded
10.251.71.1:3260,2 iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1

Login to discovered LUNs

# iscsiadm -m node -T  iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 --login
Logging in to [iface: iscsi0, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1, portal: 10.250.71.1,3260] (multiple)
Login to [iface: iscsi0, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1, portal: 10.250.71.1,3260] successful.

# iscsiadm -m node -T  iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 --login
Logging in to [iface: iscsi1, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1, portal: 10.251.71.1,3260] (multiple)
Login to [iface: iscsi1, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1, portal: 10.251.71.1,3260] successful.

Take a look at the running session

# iscsiadm -m session -P 1
Target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 (non-flash)
        Current Portal: 10.250.71.1:3260,1
        Persistent Portal: 10.250.71.1:3260,1
                **********
                Interface:
                **********
                Iface Name: iscsi0
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:4efdaa48c123
                Iface IPaddress: 10.250.71.32
                Iface HWaddress: <empty>
                Iface Netdev: iscsi0
                SID: 1
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE
Target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 (non-flash)
        Current Portal: 10.251.71.1:3260,2
        Persistent Portal: 10.251.71.1:3260,2
                **********
                Interface:
                **********
                Iface Name: iscsi1
                Iface Transport: tcp
                Iface Initiatorname: iqn.1993-08.org.debian:01:4efdaa48c123
                Iface IPaddress: 10.251.71.32
                Iface HWaddress: <empty>
                Iface Netdev: iscsi1
                SID: 2
                iSCSI Connection State: LOGGED IN
                iSCSI Session State: LOGGED_IN
                Internal iscsid Session State: NO CHANGE

Check the session is still ok after a restart of iscsid.service

# systemctl status iscsid.service
# systemctl restart iscsid.service
# systemctl status iscsid.service
# iscsiadm -m session -o show
tcp: [1] 10.251.71.1:3260,2 iqn.2006-
08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 (non-flash)
tcp: [2] 10.250.71.1:3260,1 iqn.2006-
08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 (non-flash)

Enable automatic startup of connection

# iscsiadm -m node --op=update -n node.conn[0].startup -v automatic
# iscsiadm -m node --op=update -n node.startup -v automatic

Check timeout parameter

# iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 |  grep node.session.timeo.replacement_timeout
node.session.timeo.replacement_timeout = 120

# iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 |  grep node.session.timeo.replacement_timeout
node.session.timeo.replacement_timeout = 120

Adjust timeout values to your needs

# iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1  -o update -n  node.session.timeo.replacement_timeout -v 10
# iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1  -o update -n  node.session.timeo.replacement_timeout -v 10

Configure multipathing

List SCSI devices

# lsscsi
[0:2:0:0]    disk    DELL     PERC H730 Mini   4.30  /dev/sda   <--- this is our internal disk / raid
[11:0:0:0]   cd/dvd  HL-DT-ST DVD+-RW GTA0N    A3C0  /dev/sr0
[12:0:0:1]   disk    HUAWEI   XSG1             4305  /dev/sdb   <--- this is our iSCSI-storage
[13:0:0:1]   disk    HUAWEI   XSG1             4305  /dev/sdc   <--- this is our iSCSI-storage

Get wwids for devices

# /lib/udev/scsi_id --whitelisted --device=/dev/sda
361866da075bdee001f9a2ede2705b9ba
# /lib/udev/scsi_id --whitelisted --device=/dev/sdb
3628dee5100f846b5243be07d00000004
# /lib/udev/scsi_id --whitelisted --device=/dev/sdc
3628dee5100f846b5243be07d00000004

Setup multipathing configuration

/etc/multipath.conf

defaults {
    user_friendly_names yes
}
devices {
     device {
               vendor                  "HUAWEI"
               product                 "XSG1"
               path_grouping_policy    failover
               path_checker            tur
               prio                    const
               path_selector           "round-robin 0"
               failback                immediate
               no_path_retry           15
               dev_loss_tmo            30
               fast_io_fail_tmo        5
     }
}
blacklist {
  # devnode "^sd[a]$"
  # I highly recommend you blacklist by wwid instead of device name
     
  # blacklist /dev/sda by wwid
  wwid 361866da075bdee001f9a2ede2705b9ba
}
multipaths {
  multipath {
    wwid 3628dee5100f846b5243be07d00000004
    # alias here can be anything descriptive for your LUN
    alias data
  }
}

# systemctl restart multipathd.service
# joutnalctl -lfu multipathd.service

# multipathd show config | less
<find HUAWEI and check parameters>

Startup multipathing

From the multipath(1) man page:

-r Force a reload of all existing multipath maps. This command is delegated to the
multipathd daemon if it's running. In this case, other command line switches of the multipath
command have no effect.
-ll Show ("list") the current multipath topology from all available information (sysfs, the
device mapper, path checkers ...).

# multipath -r

# multipath -ll
data (3628dee5100f846b5243be07d00000004) dm-0 HUAWEI,XSG1
size=10T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw
`-+- policy='round-robin 0' prio=50 status=active
  |- 12:0:0:1 sdb 8:16 active ready running
  `- 13:0:0:1 sdc 8:32 active ready running

# ls -al /dev/mapper/data
lrwxrwxrwx 1 root root 7 Okt 18 14:46 /dev/mapper/data -> ../dm-0

Create a systemd unit to mount it at the right time during boot

# systemctl edit --force --full data.mount

/etc/systemd/system/data.mount

[Unit]
Before=remote-fs.target
After=iscsi.service
Requires=iscsi.service
After=blockdev@dev-mapper-data.target

[Mount]
Where=/data
What=/dev/mapper/data
Type=xfs
Options=defaults

Enable your unit on next reboot and start it for now

# systemctl enable data.mount
# systemctl start  data.mount

Check for success

# df -h /dev/mapper/data
Filesystem       Size Used Avail Use% Mounted on
/dev/mapper/data  10T  72G   10T   1% /data

Further reading

External link collection:

   • https://linux.dell.com/files/whitepapers/iSCSI_Multipathing_in_Ubuntu_Server.pdf
   • https://www.suse.com/support/kb/doc/?id=000019648
   • https://ubuntu.com/server/docs/service-iscsi
   •