NetApp and Solaris: Difference between revisions
From Lolly's Wiki
Jump to navigationJump to search
m (Text replacement - "</source" to "</syntaxhighlight") |
m (Text replacement - "<source" to "<syntaxhighlight") |
||
| Line 8: | Line 8: | ||
Settings for MPxIO over FC: | Settings for MPxIO over FC: | ||
===/kernel/drv/ssd.conf=== | ===/kernel/drv/ssd.conf=== | ||
< | <syntaxhighlight lang=bash> | ||
###### START changes by host_config ##### | ###### START changes by host_config ##### | ||
ssd-config-list="NETAPP LUN", "physical-block-size:4096, retries-busy:30, retries-reset:30, retries-notready:300, retriestimeout:10, throttle-max:64, throttle-min:8"; | ssd-config-list="NETAPP LUN", "physical-block-size:4096, retries-busy:30, retries-reset:30, retries-notready:300, retriestimeout:10, throttle-max:64, throttle-min:8"; | ||
| Line 15: | Line 15: | ||
===Check it out=== | ===Check it out=== | ||
< | <syntaxhighlight lang=bash> | ||
# iostat -Er | /opt/sfw/bin/gawk 'BEGIN{command="echo ::ssd_state | mdb -k"; while(command|getline){if(/^un [0-9]+:/ && $NF != "0"){ssd=$2;gsub(/:$/,"",ssd);while(!/^}/){command|getline;if(/un_phy_blocksize/){un_phy_blocksize[ssd]=strtonum($NF);}}}};close(command);}/ssd/{ssd=$1;gsub(/^ssd/,"",ssd);getline;split($0,vendor,",");printf "ssd: %s\tun_phy_blocksize: %d\t%s\t%s\n",ssd,un_phy_blocksize[ssd],vendor[1],vendor[4];}' | # iostat -Er | /opt/sfw/bin/gawk 'BEGIN{command="echo ::ssd_state | mdb -k"; while(command|getline){if(/^un [0-9]+:/ && $NF != "0"){ssd=$2;gsub(/:$/,"",ssd);while(!/^}/){command|getline;if(/un_phy_blocksize/){un_phy_blocksize[ssd]=strtonum($NF);}}}};close(command);}/ssd/{ssd=$1;gsub(/^ssd/,"",ssd);getline;split($0,vendor,",");printf "ssd: %s\tun_phy_blocksize: %d\t%s\t%s\n",ssd,un_phy_blocksize[ssd],vendor[1],vendor[4];}' | ||
</syntaxhighlight> | </syntaxhighlight> | ||
| Line 25: | Line 25: | ||
If you have 4k as block size in your storage use ashift=12 (alignment shift exponent). | If you have 4k as block size in your storage use ashift=12 (alignment shift exponent). | ||
===Status of alignment=== | ===Status of alignment=== | ||
< | <syntaxhighlight lang=bash> | ||
# ssh filer01 "priv set -q diag ; lun show -v all; priv set" | # ssh filer01 "priv set -q diag ; lun show -v all; priv set" | ||
------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ||
| Line 62: | Line 62: | ||
</syntaxhighlight> | </syntaxhighlight> | ||
Or use "lun alignment show": | Or use "lun alignment show": | ||
< | <syntaxhighlight lang=bash> | ||
# ssh filer01 "priv set -q diag ; lun alignment show; priv set" | # ssh filer01 "priv set -q diag ; lun alignment show; priv set" | ||
------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- | ||
| Line 88: | Line 88: | ||
Or "stats show lun": | Or "stats show lun": | ||
< | <syntaxhighlight lang=bash> | ||
filer01*> stats show -e lun:/vol/TEMP201:.*_align_histo.* | filer01*> stats show -e lun:/vol/TEMP201:.*_align_histo.* | ||
</syntaxhighlight> | </syntaxhighlight> | ||
===ashift=12? Why 12?=== | ===ashift=12? Why 12?=== | ||
< | <syntaxhighlight lang=bash> | ||
# echo "2^12" | bc -l | # echo "2^12" | bc -l | ||
4096 | 4096 | ||
| Line 100: | Line 100: | ||
===What ashift do I have?=== | ===What ashift do I have?=== | ||
< | <syntaxhighlight lang=bash> | ||
# zdb | egrep ' name|ashift' | # zdb | egrep ' name|ashift' | ||
name: 'apache_pool' | name: 'apache_pool' | ||
| Line 110: | Line 110: | ||
===Create ZPools on NetApp LUNs with this syntax=== | ===Create ZPools on NetApp LUNs with this syntax=== | ||
< | <syntaxhighlight lang=bash> | ||
# zpool create -o ashift=12 <mypool> mirror <vdev1> <vdev2> | # zpool create -o ashift=12 <mypool> mirror <vdev1> <vdev2> | ||
</syntaxhighlight> | </syntaxhighlight> | ||
===Solaris Cluster=== | ===Solaris Cluster=== | ||
< | <syntaxhighlight lang=bash> | ||
# /opt/NTAP/SANToolkit/bin/sanlun lun show | nawk '$3 ~ /^\/dev\//{line=$0;gsub(/s[0-9]+$/,"",$3);command="/usr/cluster/bin/cldev list "$3; command | getline; close(command); print line,$1; next;}NR==2{print $0,"DID";next;}NR==3{print $0"-------";next}{print;}' | # /opt/NTAP/SANToolkit/bin/sanlun lun show | nawk '$3 ~ /^\/dev\//{line=$0;gsub(/s[0-9]+$/,"",$3);command="/usr/cluster/bin/cldev list "$3; command | getline; close(command); print line,$1; next;}NR==2{print $0,"DID";next;}NR==3{print $0"-------";next}{print;}' | ||
controller(7mode)/ device host lun | controller(7mode)/ device host lun | ||
Revision as of 15:27, 25 November 2021
Solaris NetApp Just some unsorted lines... Working on it... don't believe what you can read here! It is not proofed for now.
Settings in Solaris
Settings for MPxIO over FC:
/kernel/drv/ssd.conf
###### START changes by host_config #####
ssd-config-list="NETAPP LUN", "physical-block-size:4096, retries-busy:30, retries-reset:30, retries-notready:300, retriestimeout:10, throttle-max:64, throttle-min:8";
###### END changes by host_config ####
Check it out
# iostat -Er | /opt/sfw/bin/gawk 'BEGIN{command="echo ::ssd_state | mdb -k"; while(command|getline){if(/^un [0-9]+:/ && $NF != "0"){ssd=$2;gsub(/:$/,"",ssd);while(!/^}/){command|getline;if(/un_phy_blocksize/){un_phy_blocksize[ssd]=strtonum($NF);}}}};close(command);}/ssd/{ssd=$1;gsub(/^ssd/,"",ssd);getline;split($0,vendor,",");printf "ssd: %s\tun_phy_blocksize: %d\t%s\t%s\n",ssd,un_phy_blocksize[ssd],vendor[1],vendor[4];}'
Alignment and ZFS
First read ZFS zpools create misaligned I/O in Solaris 11 and Solaris 10 Update 8 and later (407376).
If you have 4k as block size in your storage use ashift=12 (alignment shift exponent).
Status of alignment
# ssh filer01 "priv set -q diag ; lun show -v all; priv set"
-------------------------------------------------------------------------------
LUN for ZFS
-------------------------------------------------------------------------------
/vol/ZoneLUNs/Zone01.lun 50g (53687091200) (r/w, online, mapped)
Serial#: 800KP+EpO-33
Share: none
Space Reservation: enabled
Multiprotocol Type: solaris_efi
Maps: SUN_SERVER01_SERVER02=40
Occupied Size: 46.2g (49583595520)
Creation Time: Wed Jan 7 11:37:58 CET 2015
---> Alignment: partial-writes
Cluster Shared Volume Information: 0x0
Space_alloc: disabled
report-physical-size: enabled
Read-Only: disabled
-------------------------------------------------------------------------------
LUN for Oracle Database
-------------------------------------------------------------------------------
/vol/TEMP201/TEMP201 25g (26843545600) (r/w, online, mapped)
Serial#: 800KP+EpO-2t
Share: none
Space Reservation: enabled
Multiprotocol Type: solaris_efi
Maps: SUN_SERVER01_SERVER02=35
Occupied Size: 21.6g (23195856896)
Creation Time: Fri Jul 4 11:02:34 CEST 2014
---> Alignment: misaligned
Cluster Shared Volume Information: 0x0
Space_alloc: disabled
report-physical-size: enabled
Read-Only: disabled
...
Or use "lun alignment show":
# ssh filer01 "priv set -q diag ; lun alignment show; priv set"
-------------------------------------------------------------------------------
LUN for ZFS
Wide spread reads. I think the ashift is not correct.
-------------------------------------------------------------------------------
/vol/ZoneLUNs/Zone01.lun
Multiprotocol type: solaris_efi
Alignment: partial-writes
Write alignment histogram percentage: 5, 5, 4, 6, 4, 6, 14, 5
Read alignment histogram percentage: 8, 7, 10, 7, 7, 8, 36, 5
Partial writes percentage: 47
Partial reads percentage: 9
-------------------------------------------------------------------------------
LUN for Oracle Database
-------------------------------------------------------------------------------
/vol/TEMP201/TEMP201
Multiprotocol type: solaris_efi
Alignment: misaligned
Write alignment histogram percentage: 0, 0, 0, 0, 0, 0, 99, 0
Read alignment histogram percentage: 0, 0, 8, 0, 0, 0, 77, 0
Partial writes percentage: 0
Partial reads percentage: 14
Or "stats show lun":
filer01*> stats show -e lun:/vol/TEMP201:.*_align_histo.*
ashift=12? Why 12?
# echo "2^12" | bc -l
4096
OK... 4k... I see.
What ashift do I have?
# zdb | egrep ' name|ashift'
name: 'apache_pool'
ashift: 9
name: 'mysql_pool'
ashift: 9
...
Create ZPools on NetApp LUNs with this syntax
# zpool create -o ashift=12 <mypool> mirror <vdev1> <vdev2>
Solaris Cluster
# /opt/NTAP/SANToolkit/bin/sanlun lun show | nawk '$3 ~ /^\/dev\//{line=$0;gsub(/s[0-9]+$/,"",$3);command="/usr/cluster/bin/cldev list "$3; command | getline; close(command); print line,$1; next;}NR==2{print $0,"DID";next;}NR==3{print $0"-------";next}{print;}'
controller(7mode)/ device host lun
vserver(Cmode) lun-pathname filename adapter protocol size mode DID
--------------------------------------------------------------------------------------------------------------------------------------------------------
ncl01-iscsi-svm1 /vol/vol_cyrus01/lun_tz_cyrus01_1 /dev/rdsk/c0t600A0980383033777B244834556D4865d0s2 iscsi0 iSCSI 500.1g C d5
ncl01-iscsi-svm1 /vol/vol_cyrus01/lun_tz_cyrus01_2 /dev/rdsk/c0t600A0980383033777B244834556D4866d0s2 iscsi0 iSCSI 500.1g C d6
ncl01-iscsi-svm1 /vol/vol_cyrus01/lun_tz_cyrus01_3 /dev/rdsk/c0t600A0980383033777B244834556D4867d0s2 iscsi0 iSCSI 500.1g C d7
...
