NetApp and Solaris: Difference between revisions

From Lolly's Wiki
Jump to navigationJump to search
No edit summary
m (Text replacement - "[[Kategorie:" to "[[Category:")
 
(2 intermediate revisions by the same user not shown)
Line 1: Line 1:
[[Kategorie:NetApp|Solaris]]
[[Category:NetApp|Solaris]]
[[Kategorie:Solaris|NetApp]]
[[Category:Solaris|NetApp]]
'''Just some unsorted lines...'''
'''Just some unsorted lines...'''
'''Working on it... don't believe what you can read here! It is not proofed for now.'''
'''Working on it... don't believe what you can read here! It is not proofed for now.'''
Line 8: Line 8:
Settings for MPxIO over FC:
Settings for MPxIO over FC:
===/kernel/drv/ssd.conf===
===/kernel/drv/ssd.conf===
<source lang=bash>
<syntaxhighlight lang=bash>
###### START changes by host_config #####
###### START changes by host_config #####
ssd-config-list="NETAPP  LUN", "physical-block-size:4096, retries-busy:30, retries-reset:30, retries-notready:300, retriestimeout:10, throttle-max:64, throttle-min:8";
ssd-config-list="NETAPP  LUN", "physical-block-size:4096, retries-busy:30, retries-reset:30, retries-notready:300, retriestimeout:10, throttle-max:64, throttle-min:8";
###### END changes by host_config ####
###### END changes by host_config ####
</source>
</syntaxhighlight>


===Check it out===
===Check it out===
<source lang=bash>
<syntaxhighlight lang=bash>
# iostat -Er | /opt/sfw/bin/gawk 'BEGIN{command="echo ::ssd_state | mdb -k"; while(command|getline){if(/^un [0-9]+:/ && $NF != "0"){ssd=$2;gsub(/:$/,"",ssd);while(!/^}/){command|getline;if(/un_phy_blocksize/){un_phy_blocksize[ssd]=strtonum($NF);}}}};close(command);}/ssd/{ssd=$1;gsub(/^ssd/,"",ssd);getline;split($0,vendor,",");printf "ssd: %s\tun_phy_blocksize: %d\t%s\t%s\n",ssd,un_phy_blocksize[ssd],vendor[1],vendor[4];}'
# iostat -Er | /opt/sfw/bin/gawk 'BEGIN{command="echo ::ssd_state | mdb -k"; while(command|getline){if(/^un [0-9]+:/ && $NF != "0"){ssd=$2;gsub(/:$/,"",ssd);while(!/^}/){command|getline;if(/un_phy_blocksize/){un_phy_blocksize[ssd]=strtonum($NF);}}}};close(command);}/ssd/{ssd=$1;gsub(/^ssd/,"",ssd);getline;split($0,vendor,",");printf "ssd: %s\tun_phy_blocksize: %d\t%s\t%s\n",ssd,un_phy_blocksize[ssd],vendor[1],vendor[4];}'
</source>
</syntaxhighlight>


==Alignment and ZFS==
==Alignment and ZFS==
Line 25: Line 25:
If you have 4k as block size in your storage use ashift=12 (alignment shift exponent).
If you have 4k as block size in your storage use ashift=12 (alignment shift exponent).
===Status of alignment===
===Status of alignment===
<source lang=bash>
<syntaxhighlight lang=bash>
# ssh filer01 "priv set -q diag ; lun show -v all; priv set"
# ssh filer01 "priv set -q diag ; lun show -v all; priv set"
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Line 60: Line 60:
                 Read-Only: disabled
                 Read-Only: disabled
...
...
</source>
</syntaxhighlight>
Or use "lun alignment show":
Or use "lun alignment show":
<source lang=bash>
<syntaxhighlight lang=bash>
# ssh filer01 "priv set -q diag ; lun alignment show; priv set"
# ssh filer01 "priv set -q diag ; lun alignment show; priv set"
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
Line 85: Line 85:
                 Partial writes percentage: 0
                 Partial writes percentage: 0
                 Partial reads percentage: 14
                 Partial reads percentage: 14
</source>
</syntaxhighlight>


Or "stats show lun":
Or "stats show lun":
<source lang=bash>
<syntaxhighlight lang=bash>
filer01*> stats show -e lun:/vol/TEMP201:.*_align_histo.*
filer01*> stats show -e lun:/vol/TEMP201:.*_align_histo.*
</source>
</syntaxhighlight>


===ashift=12? Why 12?===
===ashift=12? Why 12?===
<source lang=bash>
<syntaxhighlight lang=bash>
# echo "2^12" | bc -l
# echo "2^12" | bc -l
4096
4096
</source>
</syntaxhighlight>
OK... 4k... I see.
OK... 4k... I see.


===What ashift do I have?===
===What ashift do I have?===
<source lang=bash>
<syntaxhighlight lang=bash>
# zdb | egrep ' name|ashift'
# zdb | egrep ' name|ashift'
     name: 'apache_pool'
     name: 'apache_pool'
Line 107: Line 107:
             ashift: 9
             ashift: 9
...
...
</source>
</syntaxhighlight>


===Create ZPools on NetApp LUNs with this syntax===
===Create ZPools on NetApp LUNs with this syntax===
<source lang=bash>
<syntaxhighlight lang=bash>
# zpool create -o ashift=12 <mypool> mirror <vdev1> <vdev2>
# zpool create -o ashift=12 <mypool> mirror <vdev1> <vdev2>
</source>
</syntaxhighlight>


===Solaris Cluster===
===Solaris Cluster===
<source lang=bash>
<syntaxhighlight lang=bash>
# /opt/NTAP/SANToolkit/bin/sanlun lun show | nawk '$3 ~ /^\/dev\//{line=$0;gsub(/s[0-9]+$/,"",$3);command="/usr/cluster/bin/cldev list "$3; command | getline; close(command); print line,$1; next;}NR==2{print $0,"DID";next;}NR==3{print $0"-------";next}{print;}'
# /opt/NTAP/SANToolkit/bin/sanlun lun show | nawk '$3 ~ /^\/dev\//{line=$0;gsub(/s[0-9]+$/,"",$3);command="/usr/cluster/bin/cldev list "$3; command | getline; close(command); print line,$1; next;}NR==2{print $0,"DID";next;}NR==3{print $0"-------";next}{print;}'
controller(7mode)/                                          device                                            host                  lun     
controller(7mode)/                                          device                                            host                  lun     
Line 124: Line 124:
ncl01-iscsi-svm1    /vol/vol_cyrus01/lun_tz_cyrus01_3      /dev/rdsk/c0t600A0980383033777B244834556D4867d0s2 iscsi0    iSCSI      500.1g  C    d7
ncl01-iscsi-svm1    /vol/vol_cyrus01/lun_tz_cyrus01_3      /dev/rdsk/c0t600A0980383033777B244834556D4867d0s2 iscsi0    iSCSI      500.1g  C    d7
...
...
</source>
</syntaxhighlight>


==Links==
==Links==
* [http://wiki.illumos.org/display/illumos/List+of+sd-config-list+entries+for+Advanced-Format+drives Illumos: List of sd-config-list entries for Advanced-Format drives]
* [http://wiki.illumos.org/display/illumos/List+of+sd-config-list+entries+for+Advanced-Format+drives Illumos: List of sd-config-list entries for Advanced-Format drives]
* [https://kb.netapp.com/index?page=content&id=3011193 NetApp: What is an unaligned I/O?]
* [https://kb.netapp.com/index?page=content&id=3011193 NetApp: What is an unaligned I/O?]

Latest revision as of 16:50, 25 November 2021

Just some unsorted lines... Working on it... don't believe what you can read here! It is not proofed for now.


Settings in Solaris

Settings for MPxIO over FC:

/kernel/drv/ssd.conf

###### START changes by host_config #####
ssd-config-list="NETAPP  LUN", "physical-block-size:4096, retries-busy:30, retries-reset:30, retries-notready:300, retriestimeout:10, throttle-max:64, throttle-min:8";
###### END changes by host_config ####

Check it out

# iostat -Er | /opt/sfw/bin/gawk 'BEGIN{command="echo ::ssd_state | mdb -k"; while(command|getline){if(/^un [0-9]+:/ && $NF != "0"){ssd=$2;gsub(/:$/,"",ssd);while(!/^}/){command|getline;if(/un_phy_blocksize/){un_phy_blocksize[ssd]=strtonum($NF);}}}};close(command);}/ssd/{ssd=$1;gsub(/^ssd/,"",ssd);getline;split($0,vendor,",");printf "ssd: %s\tun_phy_blocksize: %d\t%s\t%s\n",ssd,un_phy_blocksize[ssd],vendor[1],vendor[4];}'

Alignment and ZFS

First read ZFS zpools create misaligned I/O in Solaris 11 and Solaris 10 Update 8 and later (407376).

If you have 4k as block size in your storage use ashift=12 (alignment shift exponent).

Status of alignment

# ssh filer01 "priv set -q diag ; lun show -v all; priv set"
-------------------------------------------------------------------------------
LUN for ZFS
-------------------------------------------------------------------------------
        /vol/ZoneLUNs/Zone01.lun      50g (53687091200)   (r/w, online, mapped)
                Serial#: 800KP+EpO-33
                Share: none
                Space Reservation: enabled
                Multiprotocol Type: solaris_efi
                Maps: SUN_SERVER01_SERVER02=40
                Occupied Size:   46.2g (49583595520)  
                Creation Time: Wed Jan  7 11:37:58 CET 2015
--->            Alignment: partial-writes
                Cluster Shared Volume Information: 0x0 
                Space_alloc: disabled
                report-physical-size: enabled
                Read-Only: disabled
-------------------------------------------------------------------------------
LUN for Oracle Database
-------------------------------------------------------------------------------
        /vol/TEMP201/TEMP201          25g (26843545600)   (r/w, online, mapped)
                Serial#: 800KP+EpO-2t
                Share: none
                Space Reservation: enabled
                Multiprotocol Type: solaris_efi
                Maps: SUN_SERVER01_SERVER02=35
                Occupied Size:   21.6g (23195856896)  
                Creation Time: Fri Jul  4 11:02:34 CEST 2014
--->            Alignment: misaligned
                Cluster Shared Volume Information: 0x0 
                Space_alloc: disabled
                report-physical-size: enabled
                Read-Only: disabled
...

Or use "lun alignment show": 

# ssh filer01 "priv set -q diag ; lun alignment show; priv set"
-------------------------------------------------------------------------------
LUN for ZFS
Wide spread reads. I think the ashift is not correct.
-------------------------------------------------------------------------------
        /vol/ZoneLUNs/Zone01.lun 
                Multiprotocol type: solaris_efi
                Alignment: partial-writes
                Write alignment histogram percentage: 5, 5, 4, 6, 4, 6, 14, 5
                Read alignment histogram percentage: 8, 7, 10, 7, 7, 8, 36, 5
                Partial writes percentage: 47
                Partial reads percentage: 9
-------------------------------------------------------------------------------
LUN for Oracle Database
-------------------------------------------------------------------------------
        /vol/TEMP201/TEMP201     
                Multiprotocol type: solaris_efi
                Alignment: misaligned
                Write alignment histogram percentage: 0, 0, 0, 0, 0, 0, 99, 0
                Read alignment histogram percentage: 0, 0, 8, 0, 0, 0, 77, 0
                Partial writes percentage: 0
                Partial reads percentage: 14

Or "stats show lun": 

filer01*> stats show -e lun:/vol/TEMP201:.*_align_histo.*

ashift=12? Why 12?

# echo "2^12" | bc -l
4096

OK... 4k... I see.

What ashift do I have?

# zdb | egrep ' name|ashift'
    name: 'apache_pool'
            ashift: 9
    name: 'mysql_pool'
            ashift: 9
...

Create ZPools on NetApp LUNs with this syntax

# zpool create -o ashift=12 <mypool> mirror <vdev1> <vdev2>

Solaris Cluster

# /opt/NTAP/SANToolkit/bin/sanlun lun show | nawk '$3 ~ /^\/dev\//{line=$0;gsub(/s[0-9]+$/,"",$3);command="/usr/cluster/bin/cldev list "$3; command | getline; close(command); print line,$1; next;}NR==2{print $0,"DID";next;}NR==3{print $0"-------";next}{print;}'
controller(7mode)/                                           device                                            host                  lun    
vserver(Cmode)       lun-pathname                            filename                                          adapter    protocol   size    mode  DID
--------------------------------------------------------------------------------------------------------------------------------------------------------
ncl01-iscsi-svm1     /vol/vol_cyrus01/lun_tz_cyrus01_1       /dev/rdsk/c0t600A0980383033777B244834556D4865d0s2 iscsi0     iSCSI      500.1g  C     d5
ncl01-iscsi-svm1     /vol/vol_cyrus01/lun_tz_cyrus01_2       /dev/rdsk/c0t600A0980383033777B244834556D4866d0s2 iscsi0     iSCSI      500.1g  C     d6
ncl01-iscsi-svm1     /vol/vol_cyrus01/lun_tz_cyrus01_3       /dev/rdsk/c0t600A0980383033777B244834556D4867d0s2 iscsi0     iSCSI      500.1g  C     d7
...

Links

Retrieved from "https://lars.timmann.de/wiki/index.php?title=NetApp_and_Solaris&oldid=2274"