unixway > san

Introduction

A storage area network (SAN) can address several challenges faced by system administrators. Unlike direct-attached storage (DAS), sans allow the administrator to manage a central pool of storage and allocate it to individual hosts as needed. Furthermore, the optical nature of sans provide flexibility not available with direct-attached storage which typically uses electrical signaling. For example, one can unplug a fibre cable without having to worry about "blowing" an hba.

However, their size and complexity can be daunting, particularly when faced with the terminology:

Topology of a simple SAN

In this exercise, we're going to configure a simple san composed of three nodes and one storage array. The three nodes are Solaris servers with Emulex or JNI fibre host bus adapters (hba). As we'll see, the configuration process for these hba models differs slightly, and the concepts are very similar. Our example uses Brocade switches and an EMC Symmetrix array, but the concepts would apply equally well to different vendors' products.

The following diagram illustrates several principals common to most san topologies:

Take a moment to examine the diagram below and consider the host "pear". Assuming that it requires highly-available access to disk 2ad, note that there are two separate paths to that device:

  1. pear(fcaw0) -- san1(port2) -- san1(port15) -- fa13a -- 2ad

  2. pear(fcaw1) -- san2(port2) -- san2(port14) -- fa4b -- 2ad

Figure 1: Topology of a simple SAN:
Image of simple san

 

Our desired configuration requires the following:

After the configuration is complete, the following devices will remain unassigned and available for future use: 2b5, 2b9, 2bd, 2c1, and 2c5.

Identify the WWN of devices

In the sequence below, we configure the host hba drivers to scan for the required luns, update the switch zoning, and finally configure the lun masking (VolumeLogix) in order to make the storage visible to the above three servers.

Determine world-wide names of all adapters in SAN configuration.

For host bus adapters, the world wide name is typically displayed in the /var/adm/messages file after the fibre card and software driver have been installed. An example of for an Emulex PCI fibre HBA is as follows. In this case, the relevant value the WWPN, the World-Wide Port Name:

Aug 26 09:26:28 banana lpfc: [ID 242157 kern.info] NOTICE: 
                lpfc0:031:Link Up Event received Data: 1 1 0 0
Aug 26 09:26:31 banana lpfc: [ID 129691 kern.notice] NOTICE: 
                lpfc0: Firmware Rev 3.20 (D2D3.20X3)
Aug 26 09:26:31 banana lpfc: [ID 664688 kern.notice] NOTICE: 
                lpfc0: WWPN:10:00:00:00:c9:28:22:38 
                WWNN:20:00:00:00:c9:28:22:38 DID 0x210913

Aug 26 09:26:31 banana lpfc: [ID 494464 kern.info] NOTICE: 
                lpfc1:031:Link Up Event received Data: 1 1 0 0
Aug 26 09:26:34 banana lpfc: [ID 129691 kern.notice] NOTICE: 
                lpfc1: Firmware Rev 3.20 (D2D3.20X3)
Aug 26 09:26:34 banana lpfc: [ID 664688 kern.notice] NOTICE: 
                lpfc1: WWPN:10:00:00:00:c9:28:22:56 
                WWNN:20:00:00:00:c9:28:22:56 DID 0x210913

Note that the HBA vendor also may provide a tool that allows the system administrator to query the wwn of the HBA (e.g. Emulex supplies the lputil application). The world-wide name of the symmetrix fibre adapters (FA) can be obtained from EMC. In the case of our sample san, the world wide names are as follows:

Host name port SCSI target World Wide Name
apple fcaw0 200000e069415402
apple fcaw1 200000e0694157a0
pear fcaw0 200000e069415773
pear fcaw1 200000e069415036
banana lpfc0 10000000c9282238
banana lpfc1 10000000c9282256
symmetrix fa4a target 20 500604872363ee43
symmetrix fa4b target 21 500604872363ee53
symmetrix fa13a target 22 500604872363ee4c
symmetrix fa13b target 23 500604872363ee5c

Update the host configuration

There are several vendors of host bus adapters, and the vendor's documentation is the best reference for the configuration process. In this section, we'll highlight the process of configuring the server for two popular vendors, Emulex and JNI. Please refer to the vendor's documentation for your specific card.

Typically, there are two configuration files that need to be updated once the vendor's hba software has been installed. The hba driver's configuration file typically resides in the /kernel/drv directory, and must be updated to support persistent binding and any other configuration requirements specified by the array vendor. Secondly, the Solaris "sd" driver configuration file sd.conf must be updated to tell the operating system to scan for more than the default list of scsi disk devices. The examples below describe the process for configuring Emulex and JNI cards in to support an EMC Symmetrix array.

By default, the Solaris server will scan for a limited number of scsi devices. The administrator has to update the /kernel/drv/sd.conf file to tell the sd driver to scan for a broader range of scsi devices. In both cases, the target number associated with the WWN of the array adapter is arbitrary. In our case, we've assigned scsi targets 20, 21, 22, and 23 to the four array adapters. The following list describes the additions to the /kernel/drv/sd.conf file for each of the three hosts:

Update the /etc/system file as per EMC's requirements for the Symmetrix.
set sd:sd_max_throttle=20
set scsi_options = 0x7F8

* Powerpath?   Enter
* No           sd:sd_io_time=0x78
* Yes          sd:sd_io_time=0x3C
set sd:sd_io_time=0x3C

Update the switch configuration

In the example session below, we configure the switch "san1" zoning so that one of the hbas in each host can "see" fa4a and fa13a of the Symmetrix. The switch "san2" is configured so that the other hba in each host can "see" fa4b and fa13b of the Symmetrix. We leave the configuration of san2 as an exercise for the reader.

This configuration illustrates a few principles that we have not discussed earlier. First, each host bus adapter is given only the access required to see the disks it needs. Secondly, each zone contains a single host bus adapter (i.e. a signal initiator) and a single array adapter.

Fabric OS (tm)  Release v2.5.1b
                               
login: admin
Password:   
san1:admin> switchshow
switchName:     san1  
switchType:     6.1
switchState:    Online   
switchRole:     Principal
switchDomain:   2        
switchId:       fffc02
switchWwn:      10:00:00:60:69:40:27:db
switchBeacon:   OFF                    
port  0: id  No_Light                                     
port  1: id  Online        F-Port  20:00:00:e0:69:41:54:02
port  2: id  Online        F-Port  20:00:00:e0:69:41:57:73
port  3: id  Online        F-Port  10:00:00:00:c9:28:22:38
port  4: id  No_Light      
port  5: id  No_Light      
port  6: id  No_Light      
port  7: id  No_Light      
port  8: id  No_Light      
port  9: id  No_Light      
port 10: id  No_Light      
port 11: id  No_Light      
port 12: id  No_Light                                     
port 13: id  No_Light  
port 14: id  Online        F-Port  50:06:04:87:23:63:ee:43
port 15: id  Online        F-Port  50:06:04:87:23:63:ee:4c
san1:admin> aliCreate "APPLE_FCAW0", "20:00:00:e0:69:41:54:02";
san1:admin> aliCreate "PEAR_FCAW0", "20:00:00:e0:69:41:57:73";
san1:admin> aliCreate "BANANA_LPFC0", "10:00:00:00:c9:28:22:38";
san1:admin> aliCreate "SYMM_FA4A", "50:06:04:87:23:63:ee:43";
san1:admin> aliCreate "SYMM_FA13A", "50:06:04:87:23:63:ee:4c";
san1:admin> zoneCreate "Z_APPLE_FCAW0_SYMM_FA4A", "APPLE_FCAW0; SYMM_FA4A";
san1:admin> zoneCreate "Z_PEAR_FCAW0_SYMM_FA13A", "PEAR_FCAW0; SYMM_FA13A";
san1:admin> zoneCreate "Z_BANANA_LPFC0_SYMM_FA4A", "BANANA_LPFC0; SYMM_FA4A";
san1:admin> cfgAdd "FABRIC1", "Z_APPLE_FCAW0_SYMM_FA4A; Z_PEAR_FCAW0_SYMM_FA13A; Z_BANANA_LPFC0_SYMM_FA4A";
san1:admin> cfgSave
Updating flash ...
san1:admin> cfgEnable "FABRIC1";
zone config "FABRIC1" is in effect
san1:admin> zoneshow
Defined configuration:
 cfg:   FABRIC1       
                Z_APPLE_FCAW0_SYMM_FA4A; Z_PEAR_FCAW0_SYMM_FA13A; 
                Z_BANANA_LPFC0_SYMM_FA4A        
 zone:  Z_APPLE_FCAW0_SYMM_FA4A             
                APPLE_FCAW0; SYMM_FA4A
 zone:  Z_BANANA_LPFC0_SYMM_FA4A        
                BANANA_LPFC0; SYMM_FA4A 
 zone:  Z_PEAR_FCAW0_SYMM_FA13A         
                PEAR_FCAW0; SYMM_FA13A
 alias: APPLE_FCAW0                    
                20:00:00:e0:69:41:54:02
 alias: BANANA_LPFC0                   
                10:00:00:00:c9:28:22:38
 alias: PEAR_FCAW0                     
                20:00:00:e0:69:41:57:73
 alias: SYMM_FA13A                     
                50:06:04:87:23:63:ee:4c
 alias: SYMM_FA4A                      
                50:06:04:87:23:63:ee:43
                                       
Effective configuration:
 cfg:   FABRIC1      
 zone:  Z_APPLE_FCAW0_SYMM_FA4A        
                20:00:00:e0:69:41:54:02
                50:06:04:87:23:63:ee:43
 zone:  Z_BANANA_LPFC0_SYMM_FA4A        
                10:00:00:00:c9:28:22:38 
                50:06:04:87:23:63:ee:43
 zone:  Z_PEAR_FCAW0_SYMM_FA13A         
                20:00:00:e0:69:41:57:73
                50:06:04:87:23:63:ee:4c
                                       
san1:admin> logout

Update the array's lun masking configuration

The zoning configuration defined above allows the two virtual endpoints of the fibre link to "see" each other. In the zoning example shown above, however, both apple and banana are both zoned to "see" fa4a (zones Z_APPLE_FCAW0_SYMM_FA4A and Z_BANANA_LPFC0_SYMM_FA4A). Often, the administrator wants to restrict the host visibility of disks for a single fibre adapter. The process of restricting access to disks based on the host WWN is referred to as lun masking.

In the example below, the administrator uses the EMC fpath command (recently, EMC has deprecated the fpath command in favour of the symmask/symmaskdb commands) to explicitly grant disk access to host WWNs. Without this access, the host would not be able to see the disks even though the zoning and hba configuration allowed it to see the fibre adapter on the array.

# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \
               -w 200000e069415402 -f 4a -r "15d 161 165 169"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
               -w 200000e069415402 -n "apple/fcaw0"
# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \
               -w 200000e0694157a0 -f 13b -r "15d 161 165 169"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
               -w 200000e0694157a0 -n "apple/fcaw1"

# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \
               -w 200000e069415773 -f 4b -r "2ad 2b1"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
               -w 200000e069415773 -n "pear/fcaw0"
# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \
               -w 200000e069415036 -f 13a -r "2ad 2b1"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
               -w 200000e069415036 -n "pear/fcaw1"

# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \
               -w 10000000c9282238 -f 4a -r "16d 171 175"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
               -w 10000000c9282238 -n "banana/lpfc0"
# /usr/symcli/bin/fpath adddev  -d /dev/rdsk/c1t20d0s2 \
               -w 10000000c9282256 -f 13b -r "16d 171 175"
# /usr/symcli/bin/fpath chgname -d /dev/rdsk/c1t20d0s2 \
               -w 10000000c9282256 -n "banana/lpfc1"

Refresh the VolumeLogix database:

# /usr/symcli/bin/fpath refresh -d /dev/rdsk/c1t20d0s2

Query the VolumeLogix database:

# /usr/symcli/bin/fpath lsdb -d /dev/rdsk/c1t20d0s2 -s on

Make a backup of the VolumeLogix database:

# /usr/symcli/bin/fpath backupdb -d /dev/rdsk/c1t20d0s2 \
     -o /usr/emc/VolumeLogix/backup/vcmdb.`date'+%Y%m%d'`

Reboot servers

Perform a reconfiguration reboot (e.g. "reboot -- -r") on all three servers.

You should see the desired disks. Put a Sun label on them via the "format" command and the configuration is complete.

To summarize, four "layers" of configuration must be correct in order to grant a host access to SAN storage:

  1. physical: The physical cabling linking the hba, switch, and array must function.
  2. host configuration: The host bus adapter must be configured according to the manufacturer's documentation. Our example referred to Emulex and JNI cards, but the process for other vendors is usually similar. Furthermore, The Solaris sd.conf configuration file must be updated so that the host scans for the appropriate luns.
  3. san: The san fabric must be configured so that the host bus adapter and the array fibre adapter are in the same zone.
  4. lun masking: The array lun masking configuration must be updated to give the host permission to access the required disks.

References: