ISCSI Initiator with Linux

From Lolly's Wiki
Revision as of 15:27, 25 November 2021 by Lollypop (talk | contribs) (Text replacement - "</source" to "</syntaxhighlight")
Jump to navigationJump to search


iSCSI with jumbo-frames and multipathing

Configure networking

LACP-bonding for the frontend

/etc/netplan/bond0.yaml

<source lang=yaml> network: 

 version: 2
 renderer: networkd
 ethernets:
   eno1:
     dhcp4: false
     dhcp6: false
     optional: true
   eno2:
     dhcp4: false
     dhcp6: false
     optional: true
 bonds:
   bond0:
    interfaces:
    - eno1
    - eno2
    parameters:
      lacp-rate: slow
      mode: 802.3ad
      transmit-hash-policy: layer2
    addresses:
    - 10.71.112.135/16
    gateway4: 10.71.101.1 
    nameservers:
      addresses:
      - 10.71.111.11
      - 10.71.111.12
      search:
      - domain.de

</syntaxhighlight>

Two dedicated 10GE interfaces with jumbo-frames for the backend

/etc/netplan/iscsi.yaml

<source lang=yaml> network: 

 version: 2
 renderer: networkd
 ethernets:
   enp132s0f0:
     dhcp4: false
     dhcp6: false
     mtu: 9000
     addresses:
       - 10.250.71.32/24
     set-name: iscsi0
     match:
       macaddress: a0:36:9f:d4:cd:1a
   enp132s0f1:
     dhcp4: false
     dhcp6: false
     mtu: 9000
     addresses:
       - 10.251.71.32/24
     set-name: iscsi1
     match:
       macaddress: a0:36:9f:d4:cd:18

</syntaxhighlight>

Apply the parameters and check settings

<source lang=bash>

  1. netplan apply
  2. ip a sh iscsi0

7: iscsi0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000 link/ether a0:36:9f:d4:cd:1a brd ff:ff:ff:ff:ff:ff inet 10.250.71.32/24 brd 10.250.71.255 scope global iscsi0 valid_lft forever preferred_lft forever inet6 fe80::a236:9fff:fed4:cd1a/64 scope link valid_lft forever preferred_lft forever

  1. ip a sh iscsi1

5: iscsi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000 link/ether a0:36:9f:d4:cd:18 brd ff:ff:ff:ff:ff:ff inet 10.251.71.32/24 brd 10.251.71.255 scope global iscsi1 valid_lft forever preferred_lft forever inet6 fe80::a236:9fff:fed4:cd18/64 scope link valid_lft forever preferred_lft forever

  1. ip a sh bond0

12: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 32:2d:f2:c0:e2:3f brd ff:ff:ff:ff:ff:ff inet 10.71.112.135/16 brd 10.71.255.255 scope global bond0 valid_lft forever preferred_lft forever inet6 fe80::302d:f2ff:fed0:e23f/64 scope link valid_lft forever preferred_lft forever </syntaxhighlight>

Check if all components are configured right for jumbo-frames

<source lang=bash>

  1. ping -c 3 -M do -s 8972 -I iscsi0 10.250.71.1

PING 10.250.71.1 (10.250.71.1) from 10.250.71.32 iscsi0: 8972(9000) bytes of data. 8980 bytes from 10.250.71.1: icmp_seq=1 ttl=64 time=0.227 ms 8980 bytes from 10.250.71.1: icmp_seq=2 ttl=64 time=0.187 ms 8980 bytes from 10.250.71.1: icmp_seq=3 ttl=64 time=0.198 ms --- 10.250.71.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2045ms rtt min/avg/max/mdev = 0.187/0.204/0.227/0.016 ms

  1. ping -c 3 -M do -s 8972 -I iscsi1 10.251.71.1

PING 10.251.71.1 (10.251.71.1) from 10.251.71.32 iscsi1: 8972(9000) bytes of data. 8980 bytes from 10.251.71.1: icmp_seq=1 ttl=64 time=0.202 ms 8980 bytes from 10.251.71.1: icmp_seq=2 ttl=64 time=0.195 ms 8980 bytes from 10.251.71.1: icmp_seq=3 ttl=64 time=0.191 ms --- 10.251.71.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2055ms rtt min/avg/max/mdev = 0.191/0.196/0.202/0.004 ms </syntaxhighlight> If it does not has success, one of the switches on the way or the iSCSI-storage misses jumbo-frame settings.

Configure iSCSI

Setup initiator iqn

Setup a new iqn: <source>

  1. /sbin/iscsi-iname

</syntaxhighlight>

The result is in /etc/iscsi/initiatorname.iscsi

/etc/iscsi/initiatorname.iscsi

<source>

    1. DO NOT EDIT OR REMOVE THIS FILE!
    2. If you remove this file, the iSCSI daemon will not start.
    3. If you change the InitiatorName, existing access control lists
    4. may reject this initiator. The InitiatorName must be unique
    5. for each iSCSI initiator. Do NOT duplicate iSCSI InitiatorNames.

InitiatorName=iqn.1993-08.org.debian:01:4efdaa48c123 </syntaxhighlight>

Setup iSCSI-Interfaces

<source lang=bash>

  1. iscsiadm -m iface -I iscsi0 -o new
  2. iscsiadm -m iface -I iscsi0 --op=update -n iface.net_ifacename -v iscsi0
  3. iscsiadm -m iface -I iscsi0
  4. BEGIN RECORD 2.0-874

iface.iscsi_ifacename = iscsi0 iface.net_ifacename = iscsi0 iface.ipaddress = <empty> iface.hwaddress = <empty> iface.transport_name = tcp ...

  1. END RECORD

</syntaxhighlight> <source lang=bash>

  1. iscsiadm -m iface -I iscsi1 -o new
  2. iscsiadm -m iface -I iscsi1 --op=update -n iface.net_ifacename -v iscsi1
  3. iscsiadm -m iface -I iscsi1
  4. BEGIN RECORD 2.0-874

iface.iscsi_ifacename = iscsi1 iface.net_ifacename = iscsi1 iface.ipaddress = <empty> iface.hwaddress = <empty> iface.transport_name = tcp ...

  1. END RECORD

</syntaxhighlight>

Discover LUNs that are offered by the storage

<source lang=bash>

  1. iscsiadm -m discovery -t st -p 10.250.71.1

iscsiadm: cannot make connection to 10.250.71.1: No route to host iscsiadm: cannot make connection to 10.250.71.1: No route to host iscsiadm: cannot make connection to 10.250.71.1: No route to host iscsiadm: cannot make connection to 10.250.71.1: No route to host iscsiadm: cannot make connection to 10.250.71.1: No route to host iscsiadm: cannot make connection to 10.250.71.1: No route to host iscsiadm: connection login retries (reopen_max) 5 exceeded 10.250.71.1:3260,1 iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 </syntaxhighlight> <source lang=bash>

  1. iscsiadm -m discovery -t st -p 10.251.71.1

iscsiadm: cannot make connection to 10.251.71.1: No route to host iscsiadm: cannot make connection to 10.251.71.1: No route to host iscsiadm: cannot make connection to 10.251.71.1: No route to host iscsiadm: cannot make connection to 10.251.71.1: No route to host iscsiadm: cannot make connection to 10.251.71.1: No route to host iscsiadm: cannot make connection to 10.251.71.1: No route to host iscsiadm: connection login retries (reopen_max) 5 exceeded 10.251.71.1:3260,2 iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 </syntaxhighlight>


Login to discovered LUNs

<source lang=bash>

  1. iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 --login

Logging in to [iface: iscsi0, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1, portal: 10.250.71.1,3260] (multiple) Login to [iface: iscsi0, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1, portal: 10.250.71.1,3260] successful.

  1. iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 --login

Logging in to [iface: iscsi1, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1, portal: 10.251.71.1,3260] (multiple) Login to [iface: iscsi1, target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1, portal: 10.251.71.1,3260] successful. </syntaxhighlight>

Take a look at the running session

<source lang=bash>

  1. iscsiadm -m session -P 1

Target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 (non-flash) 

       Current Portal: 10.250.71.1:3260,1
       Persistent Portal: 10.250.71.1:3260,1
               **********
               Interface:
               **********
               Iface Name: iscsi0
               Iface Transport: tcp
               Iface Initiatorname: iqn.1993-08.org.debian:01:4efdaa48c123
               Iface IPaddress: 10.250.71.32
               Iface HWaddress: <empty>
               Iface Netdev: iscsi0
               SID: 1
               iSCSI Connection State: LOGGED IN
               iSCSI Session State: LOGGED_IN
               Internal iscsid Session State: NO CHANGE

Target: iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 (non-flash) 

       Current Portal: 10.251.71.1:3260,2
       Persistent Portal: 10.251.71.1:3260,2
               **********
               Interface:
               **********
               Iface Name: iscsi1
               Iface Transport: tcp
               Iface Initiatorname: iqn.1993-08.org.debian:01:4efdaa48c123
               Iface IPaddress: 10.251.71.32
               Iface HWaddress: <empty>
               Iface Netdev: iscsi1
               SID: 2
               iSCSI Connection State: LOGGED IN
               iSCSI Session State: LOGGED_IN
               Internal iscsid Session State: NO CHANGE

</syntaxhighlight>


Check the session is still ok after a restart of iscsid.service

<source lang=bash>

  1. systemctl status iscsid.service
  2. systemctl restart iscsid.service
  3. systemctl status iscsid.service
  4. iscsiadm -m session -o show

tcp: [1] 10.251.71.1:3260,2 iqn.2006- 08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 (non-flash) tcp: [2] 10.250.71.1:3260,1 iqn.2006- 08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 (non-flash) </syntaxhighlight>

Enable automatic startup of connection

<source lang=bash>

  1. iscsiadm -m node --op=update -n node.conn[0].startup -v automatic
  2. iscsiadm -m node --op=update -n node.startup -v automatic

</syntaxhighlight>

Check timeout parameter

<source lang=bash>

  1. iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 | grep node.session.timeo.replacement_timeout

node.session.timeo.replacement_timeout = 120

  1. iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 | grep node.session.timeo.replacement_timeout

node.session.timeo.replacement_timeout = 120 </syntaxhighlight>

Adjust timeout values to your needs

<source lang=bash>

  1. iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20000:10.250.71.1 -o update -n node.session.timeo.replacement_timeout -v 10
  2. iscsiadm -m node -T iqn.2006-08.com.huawei:oceanstor:210028def5f846b5::20001:10.251.71.1 -o update -n node.session.timeo.replacement_timeout -v 10

</syntaxhighlight>

Configure multipathing

List SCSI devices

<source lang=bash>

  1. lsscsi

[0:2:0:0] disk DELL PERC H730 Mini 4.30 /dev/sda <--- this is our internal disk / raid [11:0:0:0] cd/dvd HL-DT-ST DVD+-RW GTA0N A3C0 /dev/sr0 [12:0:0:1] disk HUAWEI XSG1 4305 /dev/sdb <--- this is our iSCSI-storage [13:0:0:1] disk HUAWEI XSG1 4305 /dev/sdc <--- this is our iSCSI-storage </syntaxhighlight>

Get wwids for devices

<source lang=bash>

  1. /lib/udev/scsi_id --whitelisted --device=/dev/sda

361866da075bdee001f9a2ede2705b9ba

  1. /lib/udev/scsi_id --whitelisted --device=/dev/sdb

3628dee5100f846b5243be07d00000004

  1. /lib/udev/scsi_id --whitelisted --device=/dev/sdc

3628dee5100f846b5243be07d00000004 </syntaxhighlight>

Setup multipathing configuration

/etc/multipath.conf

<source> defaults { 

   user_friendly_names yes

} devices { 

 device {
   vendor                  "HUAWEI"
   product                 "XSG1"
   path_grouping_policy    multibus
   path_checker            tur
   prio                    const
   path_selector           "round-robin 0"
   failback                immediate
   no_path_retry           15
 }

} blacklist { 

 # devnode "^sd[a]$"
 # I highly recommend you blacklist by wwid instead of device name
    
 # blacklist /dev/sda by wwid
 wwid 361866da075bdee001f9a2ede2705b9ba

} multipaths { 

 multipath {
   wwid 3628dee5100f846b5243be07d00000004
   # alias here can be anything descriptive for your LUN
   alias data
 }

} </syntaxhighlight>

Startup multipathing

From the multipath(1) man page: 

-r Force a reload of all existing multipath maps. This command is delegated to the
multipathd daemon if it's running. In this case, other command line switches of the multipath
command have no effect.
-ll Show ("list") the current multipath topology from all available information (sysfs, the
device mapper, path checkers ...).

<source lang=bash>

  1. multipath -r

</syntaxhighlight> <source lang=bash>

  1. multipath -ll

data (3628dee5100f846b5243be07d00000004) dm-0 HUAWEI,XSG1 size=10T features='1 queue_if_no_path' hwhandler='1 alua' wp=rw `-+- policy='round-robin 0' prio=50 status=active 

 |- 12:0:0:1 sdb 8:16 active ready running
 `- 13:0:0:1 sdc 8:32 active ready running

</syntaxhighlight>

<source lang=bash>

  1. ls -al /dev/mapper/data

lrwxrwxrwx 1 root root 7 Okt 18 14:46 /dev/mapper/data -> ../dm-0 </syntaxhighlight>

Create a systemd unit to mount it at the right time during boot

<source lang=bash>

  1. systemctl edit --force --full data.mount

</syntaxhighlight>

/etc/systemd/system/data.mount

<source lang=inifile> [Unit] Before=remote-fs.target After=iscsi.service Requires=iscsi.service After=blockdev@dev-mapper-data.target

[Mount] Where=/data What=/dev/mapper/data Type=xfs Options=defaults </syntaxhighlight>

Enable your unit on next reboot and start it for now

<source lang=bash>

  1. systemctl enable data.mount
  2. systemctl start data.mount

</syntaxhighlight>

Check for success

<source lang=bash>

  1. df -h /dev/mapper/data

Filesystem Size Used Avail Use% Mounted on /dev/mapper/data 10T 72G 10T 1% /data </syntaxhighlight>

Further reading

External link collection: 

   • https://linux.dell.com/files/whitepapers/iSCSI_Multipathing_in_Ubuntu_Server.pdf
   • https://www.suse.com/support/kb/doc/?id=000019648
   • https://ubuntu.com/server/docs/service-iscsi
   •
Retrieved from "https://lars.timmann.de/wiki/index.php?title=ISCSI_Initiator_with_Linux&oldid=2228"