Storage
area networking
Storage
area networking
A storage area network (SAN) can address several challenges faced by system administrators. Unlike direct-attached storage (DAS), sans allow the administrator to manage a central pool of storage and allocate it to individual hosts as needed. Furthermore, the optical nature of sans provide flexibility not available with direct-attached storage which typically uses electrical signaling. For example, one can unplug a fibre cable without having to worry about "blowing" an hba.
However, their size and complexity can be daunting, particularly when faced with the terminology:
In this exercise, we're going to configure a simple san composed of three nodes and one storage array. The three nodes are Solaris servers with Emulex or JNI fibre host bus adapters (hba). As we'll see, the configuration process for these hba models differs slightly, and the concepts are very similar. Our example uses Brocade switches and an EMC Symmetrix array, but the concepts would apply equally well to different vendors' products.
The following diagram illustrates several principals common to most san topologies:
There are two separate storage area networks ("san1" and "san2"). Though most switches support inter-switch links (ISLs), many vendors suggest isolation of fabrics for simple SANs as shown here.. Each switch therefore comprises an independent "fabric".
The array provides multiple I/O paths to every disk device. In the diagram below, for example, the disk device "2ad" is accessible via both fa4b and fa13a. Depending on the array, the two paths may be simultaneously active (active/active) or only one path may be valid at a time (active/passive).
Each host has a connection into each fabric. Host-based software load-balances the traffic between the two links if the array supports active/active disk access. Furthermore, the host should adjust to a single link failure by rerouting all traffic along the surviving path.
Take a moment to examine the diagram below and consider the host "pear". Assuming that it requires highly-available access to disk 2ad, note that there are two separate paths to that device:
pear(fcaw0) -- san1(port2) -- san1(port15) -- fa13a -- 2ad
pear(fcaw1) -- san2(port2) -- san2(port14) -- fa4b -- 2ad
Our desired configuration requires the following:
After the configuration is complete, the following devices will remain unassigned and available for future use: 2b5, 2b9, 2bd, 2c1, and 2c5.
In the sequence below, we configure the host hba drivers to scan for the required luns, update the switch zoning, and finally configure the lun masking (VolumeLogix) in order to make the storage visible to the above three servers.
Determine world-wide names of all adapters in SAN configuration.
For host bus adapters, the world wide name is typically displayed in the /var/adm/messages file after the fibre card and software driver have been installed. An example of for an Emulex PCI fibre HBA is as follows. In this case, the relevant value the WWPN, the World-Wide Port Name:
Aug 26 09:26:28 banana lpfc: [ID 242157 kern.info] NOTICE:
lpfc0:031:Link Up Event received Data: 1 1 0 0
Aug 26 09:26:31 banana lpfc: [ID 129691 kern.notice] NOTICE:
lpfc0: Firmware Rev 3.20 (D2D3.20X3)
Aug 26 09:26:31 banana lpfc: [ID 664688 kern.notice] NOTICE:
lpfc0: WWPN:10:00:00:00:c9:28:22:38
WWNN:20:00:00:00:c9:28:22:38 DID 0x210913
Aug 26 09:26:31 banana lpfc: [ID 494464 kern.info] NOTICE:
lpfc1:031:Link Up Event received Data: 1 1 0 0
Aug 26 09:26:34 banana lpfc: [ID 129691 kern.notice] NOTICE:
lpfc1: Firmware Rev 3.20 (D2D3.20X3)
Aug 26 09:26:34 banana lpfc: [ID 664688 kern.notice] NOTICE:
lpfc1: WWPN:10:00:00:00:c9:28:22:56
WWNN:20:00:00:00:c9:28:22:56 DID 0x210913
Note that the HBA vendor also may provide a tool that allows the system administrator to query the wwn of the HBA (e.g. Emulex supplies the lputil application). The world-wide name of the symmetrix fibre adapters (FA) can be obtained from EMC. In the case of our sample san, the world wide names are as follows:
| Host name | port | SCSI target | World Wide Name |
|---|---|---|---|
| apple | fcaw0 | 200000e069415402 | |
| apple | fcaw1 | 200000e0694157a0 | |
| pear | fcaw0 | 200000e069415773 | |
| pear | fcaw1 | 200000e069415036 | |
| banana | lpfc0 | 10000000c9282238 | |
| banana | lpfc1 | 10000000c9282256 | |
| symmetrix | fa4a | target 20 | 500604872363ee43 |
| symmetrix | fa4b | target 21 | 500604872363ee53 |
| symmetrix | fa13a | target 22 | 500604872363ee4c |
| symmetrix | fa13b | target 23 | 500604872363ee5c |
There are several vendors of host bus adapters, and the vendor's documentation is the best reference for the configuration process. In this section, we'll highlight the process of configuring the server for two popular vendors, Emulex and JNI. Please refer to the vendor's documentation for your specific card.
Typically, there are two configuration files that need to be updated once the vendor's hba software has been installed. The hba driver's configuration file typically resides in the /kernel/drv directory, and must be updated to support persistent binding and any other configuration requirements specified by the array vendor. Secondly, the Solaris "sd" driver configuration file sd.conf must be updated to tell the operating system to scan for more than the default list of scsi disk devices. The examples below describe the process for configuring Emulex and JNI cards in to support an EMC Symmetrix array.
Configure the /kernel/drv/fcaw.conf on servers apple and pear:
fca_nport = 1; failover = 60; def_hba_binding = "fas"; fca_verbose = 1;
Also available for examination are the complete fcaw.conf files for PowerPath and non-PowerPath.
fcp-bind-WWPN="500604872363ee43:lpfc0t20",
"500604872363ee5c:lpfc1t23";
Also available for examination are the complete lpfc.conf files for PowerPath and non-PowerPath.
By default, the Solaris server will scan for a limited number of scsi devices. The administrator has to update the /kernel/drv/sd.conf file to tell the sd driver to scan for a broader range of scsi devices. In both cases, the target number associated with the WWN of the array adapter is arbitrary. In our case, we've assigned scsi targets 20, 21, 22, and 23 to the four array adapters. The following list describes the additions to the /kernel/drv/sd.conf file for each of the three hosts:
# Entries added for host apple to "see" lun 0-4 on fa4a and fa13b # fa4a = 500604872363ee43 name="sd" class="scsi" target=20 lun=0 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=1 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=2 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=3 hba="fcaw0" wwn="500604872363ee43"; name="sd" class="scsi" target=20 lun=4 hba="fcaw0" wwn="500604872363ee43"; # fa13b = 500604872363ee5c name="sd" class="scsi" target=23 lun=0 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=1 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=2 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=3 hba="fcaw1" wwn="500604872363ee5c"; name="sd" class="scsi" target=23 lun=4 hba="fcaw1" wwn="500604872363ee5c";
# Entries added for host pear to "see" lun 0-2 on fa4b and fa13a # fa4b = 500604872363ee53 name="sd" class="scsi" target=21 lun=0 hba="fcaw0" wwn="500604872363ee53"; name="sd" class="scsi" target=21 lun=1 hba="fcaw0" wwn="500604872363ee53"; name="sd" class="scsi" target=21 lun=2 hba="fcaw0" wwn="500604872363ee53"; # fa13a = 500604872363ee4c name="sd" class="scsi" target=22 lun=0 hba="fcaw1" wwn="500604872363ee4c"; name="sd" class="scsi" target=22 lun=1 hba="fcaw1" wwn="500604872363ee4c"; name="sd" class="scsi" target=22 lun=2 hba="fcaw1" wwn="500604872363ee4c";
# Entries added for host banana to "see" lun 5-7 on fa4a and fa13b # fa4a = 500604872363ee43 name="sd" parent="lpfc" target=20 lun=0 hba="lpfc0"; name="sd" parent="lpfc" target=20 lun=5 hba="lpfc0"; name="sd" parent="lpfc" target=20 lun=6 hba="lpfc0"; name="sd" parent="lpfc" target=20 lun=7 hba="lpfc0"; # fa13b = 500604872363ee5c name="sd" parent="lpfc" target=23 lun=0 hba="lpfc1"; name="sd" parent="lpfc" target=23 lun=5 hba="lpfc1"; name="sd" parent="lpfc" target=23 lun=6 hba="lpfc1"; name="sd" parent="lpfc" target=23 lun=7 hba="lpfc1";
set sd:sd_max_throttle=20 set scsi_options = 0x7F8 * Powerpath? Enter * No sd:sd_io_time=0x78 * Yes sd:sd_io_time=0x3C set sd:sd_io_time=0x3C
In the example session below, we configure the switch "san1" zoning so that one of the hbas in each host can "see" fa4a and fa13a of the Symmetrix. The switch "san2" is configured so that the other hba in each host can "see" fa4b and fa13b of the Symmetrix. We leave the configuration of san2 as an exercise for the reader.
This configuration illustrates a few principles that we have not discussed earlier. First, each host bus adapter is given only the access required to see the disks it needs. Secondly, each zone contains a single host bus adapter (i.e. a signal initiator) and a single array adapter.
Fabric OS (tm) Release v2.5.1b
login: admin
Password:
san1:admin> switchshow
switchName: san1 switchType: 6.1 switchState: Online switchRole: Principal switchDomain: 2 switchId: fffc02 switchWwn: 10:00:00:60:69:40:27:db switchBeacon: OFF port 0: id No_Light port 1: id Online F-Port 20:00:00:e0:69:41:54:02 port 2: id Online F-Port 20:00:00:e0:69:41:57:73 port 3: id Online F-Port 10:00:00:00:c9:28:22:38 port 4: id No_Light port 5: id No_Light port 6: id No_Light port 7: id No_Light port 8: id No_Light port 9: id No_Light port 10: id No_Light port 11: id No_Light port 12: id No_Light port 13: id No_Light port 14: id Online F-Port 50:06:04:87:23:63:ee:43 port 15: id Online F-Port 50:06:04:87:23:63:ee:4c
san1:admin> aliCreate "APPLE_FCAW0",
"20:00:00:e0:69:41:54:02";
san1:admin> aliCreate "PEAR_FCAW0",
"20:00:00:e0:69:41:57:73";
san1:admin> aliCreate "BANANA_LPFC0",
"10:00:00:00:c9:28:22:38";
san1:admin> aliCreate "SYMM_FA4A",
"50:06:04:87:23:63:ee:43";
san1:admin> aliCreate "SYMM_FA13A",
"50:06:04:87:23:63:ee:4c";
san1:admin> zoneCreate
"Z_APPLE_FCAW0_SYMM_FA4A", "APPLE_FCAW0;
SYMM_FA4A";
san1:admin> zoneCreate
"Z_PEAR_FCAW0_SYMM_FA13A", "PEAR_FCAW0;
SYMM_FA13A";
san1:admin> zoneCreate
"Z_BANANA_LPFC0_SYMM_FA4A", "BANANA_LPFC0;
SYMM_FA4A";
san1:admin> cfgAdd "FABRIC1",
"Z_APPLE_FCAW0_SYMM_FA4A; Z_PEAR_FCAW0_SYMM_FA13A;
Z_BANANA_LPFC0_SYMM_FA4A";
san1:admin> cfgSave
Updating flash ...
san1:admin> cfgEnable "FABRIC1";
zone config "FABRIC1" is in effect
san1:admin> zoneshow
Defined configuration:
cfg: FABRIC1
Z_APPLE_FCAW0_SYMM_FA4A; Z_PEAR_FCAW0_SYMM_FA13A;
Z_BANANA_LPFC0_SYMM_FA4A
zone: Z_APPLE_FCAW0_SYMM_FA4A
APPLE_FCAW0; SYMM_FA4A
zone: Z_BANANA_LPFC0_SYMM_FA4A
BANANA_LPFC0; SYMM_FA4A
zone: Z_PEAR_FCAW0_SYMM_FA13A
PEAR_FCAW0; SYMM_FA13A
alias: APPLE_FCAW0
20:00:00:e0:69:41:54:02
alias: BANANA_LPFC0
10:00:00:00:c9:28:22:38
alias: PEAR_FCAW0
20:00:00:e0:69:41:57:73
alias: SYMM_FA13A
50:06:04:87:23:63:ee:4c
alias: SYMM_FA4A
50:06:04:87:23:63:ee:43
Effective configuration:
cfg: FABRIC1
zone: Z_APPLE_FCAW0_SYMM_FA4A
20:00:00:e0:69:41:54:02
50:06:04:87:23:63:ee:43
zone: Z_BANANA_LPFC0_SYMM_FA4A
10:00:00:00:c9:28:22:38
50:06:04:87:23:63:ee:43
zone: Z_PEAR_FCAW0_SYMM_FA13A
20:00:00:e0:69:41:57:73
50:06:04:87:23:63:ee:4c
san1:admin> logout
The zoning configuration defined above allows the two virtual endpoints of the fibre link to "see" each other. In the zoning example shown above, however, both apple and banana are both zoned to "see" fa4a (zones Z_APPLE_FCAW0_SYMM_FA4A and Z_BANANA_LPFC0_SYMM_FA4A). Often, the administrator wants to restrict the host visibility of disks for a single fibre adapter. The process of restricting access to disks based on the host WWN is referred to as lun masking.
In the example below, the administrator uses the EMC fpath command (recently, EMC has deprecated the fpath command in favour of the symmask/symmaskdb commands) to explicitly grant disk access to host WWNs. Without this access, the host would not be able to see the disks even though the zoning and hba configuration allowed it to see the fibre adapter on the array.
# /usr/symcli/bin/fpath adddev -d /dev/rdsk/c1t20d0s2 \
-w 200000e069415402 -f 4a -r "15d 161 165 169"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
-w 200000e069415402 -n "apple/fcaw0"
# /usr/symcli/bin/fpath adddev -d /dev/rdsk/c1t20d0s2 \
-w 200000e0694157a0 -f 13b -r "15d 161 165 169"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
-w 200000e0694157a0 -n "apple/fcaw1"
# /usr/symcli/bin/fpath adddev -d /dev/rdsk/c1t20d0s2 \
-w 200000e069415773 -f 4b -r "2ad 2b1"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
-w 200000e069415773 -n "pear/fcaw0"
# /usr/symcli/bin/fpath adddev -d /dev/rdsk/c1t20d0s2 \
-w 200000e069415036 -f 13a -r "2ad 2b1"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
-w 200000e069415036 -n "pear/fcaw1"
# /usr/symcli/bin/fpath adddev -d /dev/rdsk/c1t20d0s2 \
-w 10000000c9282238 -f 4a -r "16d 171 175"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
-w 10000000c9282238 -n "banana/lpfc0"
# /usr/symcli/bin/fpath adddev -d /dev/rdsk/c1t20d0s2 \
-w 10000000c9282256 -f 13b -r "16d 171 175"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
-w 10000000c9282256 -n "banana/lpfc1"
Refresh the VolumeLogix database:
# /usr/symcli/bin/fpath refresh -d
/dev/rdsk/c1t20d0s2
Query the VolumeLogix database:
# /usr/symcli/bin/fpath lsdb -d
/dev/rdsk/c1t20d0s2 -s on
Make a backup of the VolumeLogix database:
# /usr/symcli/bin/fpath backupdb -d /dev/rdsk/c1t20d0s2 \
-o /usr/emc/VolumeLogix/backup/vcmdb.`date'+%Y%m%d'`
"reboot --
-r") on all three servers.
You should see the desired disks. Put a Sun label on them via the "format" command and the configuration is complete.
To summarize, four "layers" of configuration must be correct in order to grant a host access to SAN storage:
References: