<-- Please click if you found this site useful ;-)
EMC - Clariion
For initial setup
of Clariion CX-300 and CX-700
Connect a windows laptop to the clariion head unit serial port with speed of 19200 bps,
it is able to talk both serial console (standard text) and also TCP/IP over Serial (SLIP?)
so that a null modem can be setup between laptop and serial port to configure
the machine via web interface.
The EMC Storage (server side) config is all done thru a web GUI.
The CX-300 and CX-700 actually run embeded windows XP (older version run
embeded windows 2000). I have no idea what happen if there happen to be a
virus on it...
Get the CLARiiON Procedure Generator(Clariion PG). It cranks out a huge doc with step by step procedure of how to add new host to the SAN, etc. Very good stuff!! However, still check the support matrix doc, which has the ultimate say. Clariion PG and eLab are known to lag behind in its DB update of what is suportable!!!
Access Logix = SAN environment, CX will do LUN masking (select LUNs presented to select hosts)
Non Access Logix = Direct attach environment, all LUN presented to the direct attached host(s).
WWN World Wide Number (MAC address equiv for the SAN world).
PWWN, WWPN Port WWN (one for each HBA port for a given host)
NWWN, WWNN Node WWN (a slight diff from PWWN, but still multiple WWN per machine/host) so what does it do??
When creating LUN, best to create them of slightly different size if
they do not present other asymetry problems on the host side.
eg 48, 49, 50, 51, 52 GB for 5 LUNs.
This is because it will be hard to tell which emcpowerX device is which LUN
checking WWN numbers are not readily retrievable.
(Latest release in 2006 is clearly labeling the LUN now, but solaris format
still doesn't show such info, being able to match size allows easy telling that c7t1d5s0 is the same LUN presented by emcpower5a).
Of course, if the LUN will then be stripped or mirrored by the OS, then it would not be feasible to have them of different size. But it begs the question of why do software RAID on the OS when the Clariion can do hardware RAID...
Another trick in sizing LUN is to use the Raid Group number as the last digit of the LUN size. eg:
LUN 1 created in RG1 = 101 GB
LUN 2 created in RG4 = 104 GB
This may sometime make it easier to tell which LUN is in which Raid Group.
Assign range of LUN number for given host.
eg LUN 10-39 for VCS Oracle DB storage group
LUN 60-89 for ESX cluster
NaviCLI has no security, any machine with the binary installed can send
command to EMC CX-700 to configure the box, not just snapshot, but
raid group, lun dev, etc. Only possibility is to change naviCLI to use a
different port (obscurity) and firewall up the port (assuming production vlan
is secure). By default, NaviCLI use port 6389. Additional features use port 6390 thru 6392. EMC Acknoledge this weakness, and has a security paper on the topic; see cache. NaviSecCli was supposed to fix this...
Snapshot is on one EMC LUN at a time, no atomic funtcion to take snap on multiple LUN at the same time. Thus, if host FS span across multiple EMC LUN, and EMC snapshot is desired, there is a potential race condition that can have data corruption. -- the performance gain of multiple LUN may not be worth the potential data loss. Don't do multiple LUN per FS!!
Emulex HBA on Solaris
There are two sets of drivers.
somehow no /usr/sbin/lpfc/lputil for Solaris,
but there is at least:
hbacmd List # list all HBA and their WWN
* emxamd # replaced by hbacmd above?
* HBAnyware # GUI tool for the HBA
May be downloaded from Emulex.com and likely going to be able to work with either driver.
Clariion Procedure Generator calls for the LPFC driver, which are developed
specifically for EMC. This is marked as "Solaris ready".
system lpfc Emulex LightPulse FC SCSI/IP Host Bus Adapter driver
system HBAnyware Emulex HBAnyware FC Host Bus Adapter Remote Manager
Commands used to configure the HBA with this driver are
- /usr/sbin/lputil TUI (though it doesn't seems to be installed by above package).
- /opt/HBAnyware/hbanyware GUI (req reboot after initial install to work).
- /kernel/drv/lpfc.conf Persistent binding req finding lpfc instance from /var/adm/messages,
and adding the info to this file (may have tool to do this).
- /kernel/drv/sd.conf LPFC release 6.21f don't use this to assign LUN anymore.
Was not supposed to req reboot to find new LUN either,
but may need to run some commands that I don't know,
so reboot -- -r.
## Exmaple of persistent binding config.
## wwpn of array: hba instance (lpfcX) + binding (tY)
## One approach is:
## spAx assign to t0
## spBx assign to t1
## keep the same for both HBA/FC fabric
## But since there are typically 4 HBA path, I found it more useful to
## simply assing a uniq tag tY number to each path.
## eg below assign t3, t4 to the first HBA (one for each Service Processor on the Clariion),
## then t5 and t6 to the second HBA, which connects to the second fail over fabric.
## the lpfc are 1 and 3 because the HBA are actually dual ported, albeit only 1 port is used.
## CX WWPN | tY binding number
Alternatively, there is the SFS FCA (EMLXS) driver, which is part of the
SAN Foundation Suite (SFS) from Sun (Leadville v4.4.14 install_it version).
It is included by default in Solaris 10 update 1.
EMC support this version also.
This is the only one that will work in Solaris x86 environment.
system SUNWcfcl Common Fibre Channel HBA API Library (Usr)
system SUNWcfclr Common Fibre Channel HBA API Library (Root)
system SUNWcfclx Common Fibre Channel HBA API Library (Usr) (64-bit)
system SUNWfchba Sun Fibre Channel Host Bus Adapter Library
system SUNWemlxs Emulex-Sun LightPulse FCA driver (root)
system SUNWemlxsx Emulex-Sun LightPulse FCA driver (root) (64-bit)
system SUNWemlxu Emulex-Sun LightPulse FCA Utilties (usr)
system SUNWemlxux Emulex-Sun LightPulse FCA Utilties (usr) (64-bit)
Sun Support Enginner can provide sfkcheck script to see if all desired patches
are installed. SFS install other SAN tools in addition to Emulex drivers above,
which can stay on machine even if LPFC driver is used.
Commands used to configure the HBA with this driver:
- luxadm -e port
- cfgadm -c configure c3
- mpathadm (MPxIO, Solaris native multipathing software)
Sun Dev Net has a good overview of the Solaris
Leadville Fibre Channel stack
Install the PowerPath package on Solaris (and other supported OS)
and it provides EMC Clariion commands related to disk/path access.
To add LUN, after it is presented to host, run:
/usr/sbin/lpfc/lun_scan all # script provided by emulex
powermt save # ? is save really needed??
# this save config to /etc/powermt.custom
# in some strange case when powerpath hoses, remove this file and reboot.
cfgadm -c configure c3
powermt display dev=all # list all devices
powermt display dev=emcpower0a # list specific dev (usually 4 path per LUN)
powermt check # see if there are dead path, remove them
powermt config # scan for new disk/path
powermt restore # place all LUN back to original SP,
# clear up any transpass. (useful after test HA).
emcpreg -install # add powerpath license info.
modinfo | grep emcp # see PP related kernel modules.
pkgrm EMCpower call "/etc/emc/bin/emcpmgr unconfig -e mpx" to do clean up.
If this script fails, EMCpower pkg cannot be removed.
powermt display dev=all # and check things are okay
reboot if desired to ensure spurious device don't pop up again.
powercf -q patches /kernel/etc/emcp.conf and remove any spurious devices.
It should be run during boot time; but somehow this manual procedure was needed
to remove the ghost device.
----> don't do that removal in PP 5.2, PP get confused and can't do anything.
To remap emcpowerX devices, use emcpadm.
emcpadm print_mappings # display mapping
# there are commands to import/export/check
emcpadm export_mappings -f powerpath.map # (1) create a map, saving it in a file
emcpadm check_mappings [-v] -f powerpath.map # check to see if path is the same in other node of a cluster, or check with historical settings
emcpadm import_mappings [-v] -f powerpath.map # make a second cluster node have same device path as (1)
> Before importing new mappings on a node or server:
> 1. Preview changes with emcpadm check_mappings.
> 2. Shut down all applications and database systems.
> 3. Unmount file systems.
> 4. Deport VxVM disk groups.
Older Multi-Path software may be refered and no longer work with PP. They should be removed.
ATF - Application Transparent Failover
CDE - CLARiiON Driver Extensions
The Sun Package name for them are CLatf and CLcde. If they are not on the system, safe to know they
won't clobber PowerPath.
PP & VxVM
System with Veritas, run these commands so that VxVM knows how to work with PP on the CX:
vxddladm rmjbod vid=DGC # 4.1+
vxddladm addjbod vid=DGC pagecode=0x83 offset=8 length=16 # 4.0 and older
Effective after reboot.
PP & Solstice Disk Suite (SDS) / Sun Volume Management (SVM)
Back in the days, SDS add an entry "forceload: drv/sd" into /etc/system.
The SVM version that came with Solaris 9 HW 2006 no longer does this,
the kernel was to be able to load this automatically correctly at boot.
However, PP will nonetheless put that in (within the block of all the drv that PP loads).
It is needed for system with PP to pre-load this driver for the root disk mirroring to work.
Thus, if removing PP manually, this drv/sd should be left as forceload in /etc/system
or else, will need to split the mirror of the root disk containing /
or machine won't boot!
If pkgrm EMCpower works without error, then it is okay of forceload: drv/sd is removed from /etc/system.
But if doing manual removal, need to take out all non-sun entries in /etc/system
and drivers in /kernel/drv /kernel/drv/sparcv9, + many other emc files.
refer to emc primus 81236 and 128595 doc.
PowerPath Devices in Cluster Environment
In a cluster environment, it is often desirable to see all LUNs show up
under the same /dev/emcpowerX device name. There are several ways to do this:
add 1 LUN at a time,
run the HBA vendor's scan disk utility
and configure power path.
Do this on all nodes of the cluster before adding the next LUN.
(eg when adding more node to a cluster, or multiple LUN are created
by a storage admin and handed off to the sys admin):
shutdown PP and copy files /etc/emcp_devicesDB.dat and emcp_devicesDB.idx
from "master" node to the other nodes.
emcpadm rename -s emcpowera -t emcpowerf to rename from (source) emcpowera (to) emcpowerf .
emcpadm getused can be used to view device's major and minor numbers.
NAVICLI and NAVIAGENT
optional packages to be installed on Solaris (or other OS):
/opt/Navisphere/bin/navicli -h ClariionIP naviCliCmd # reg unix machine
/nas/sbin/navicli -h spa naviClicmd # integrated NS
In the control station of an integrated NS series, spa and spb would be the head of the cx backend.
w/o naviCliCmd, it will list all commands. eg:
navicli -help # short help message
navicli foo # display list of commands
getall # retrieve all info, good for testing.
getlun 101 # get info about LUN (#101, ommit for all LUNs)
getdisk 3_1_0 # find out usage of disk in Bus 3, Enclosure 1, Disk 0 (leftmost). (rightmost could be called _E or _13)
# hotspare or part of RAID GROUP, serial number, size, etc.
# line with Prct Rebuilt indicates LunID: rebuild %
getlun 32 | egrep -A1 Rebuilt # see rebuild status of Lun #32
getlun 32 -prb # see progress of rebuild (in percent)
# RAID 5 on 750GB SATA II takes about 4 hours to rebuild.
getlun 32 -rb # see priority of rebuild
getsniffer -rg 15 # see status of backgroun verify process (eg when raid group had been rebuild and
# want to check for consistency.
navicli -h spa getcontrol -busy
# in ns80, see how busy the backend CX service processor A is
# all navicli command works from the control station even when
# it is integrated model that doesn't present navisphere to outside workd
New version has secure version called naviseccli.
GUI version navicli.jar ??
For configuration, edit this file:
Mainly, add info so that the agent know how to connect to the Clariion head
and issue commands to it:
/etc/init.d/agent start # solaris
/etc/rc.agent start # aix (spawn by inittab)
Required packages in SUN:
application NAVIAGENT Navisphere Disk Array Management Tool (AGENT)
application NAVICLI Navisphere Disk Array Management Tool (CLI)
application NAVISCLI Navisphere Disk Array Management Tool (CLI)
navicli -h cx300a getall
naviseccli -User xyz -Password pw -scope 0 -h cx300a getall
if no -Password is given, should prompt for it...
Alternatively, can create a user and it will have encrypted password in a file so that no username/password need to be given.
naviseccli -addusersecurity -scope 0 -user xyz -secfilepath /root
it will prompt for password if not entered on the cli
Shift Ctrl F2 (or F12).
Email notification support is set under host tab, template is used by both SP.
If ESRS is setup, it will be under the ESRS workstation.
Checking for host access to CX
Even when hosts are registered to the Clariion, in strange situations, it may not have access to all the service processers. It is an ACL, and this could result in single path and lack of HA for some hosts. To check it (Courtesy of R. Anderson@Dell):
1. Enter the engineering mode
2. Open the storage groups section of the navigation tree
3. Right click on the specific storage group you want to check
4. Select properties for the storage group
5. Go to the hosts tab
6. You will notice that there is an advanced button below the left side window of this tab (once after everything is loaded. Be patient!)
7. Click the advanced button
8. The hosts that have access to the storage group will have tabs presenting the initiators. Those that have check marks have been given access to the storage group. Those that are not checked do not have access but can then be checked if they are logged into the array.
Gathering info for support
To start a new SP Collect: Expand the Navisphere Manager tree so you can see the SPs. Right-click on an SP and select "SP Collect" from the menu. It will take several minutes (depending on array workload and size) for the SP Collect to be created.
To download an SP Collect: Expand the Navisphere Manager tree so you can see the SPs. Right-click on an SP and select "File Transfer Manager" from the menu. This opens the File Transfer Manager dialog. Look for files in the list that have names ending with "_data.zip". There are the SP Collect files. Choose the one with the most recent date. If you just started a SP Collect and you don't see it in the list, click the Refresh button until it appears. Then choose a destination directory by clicking the Browse button and finally click the Transfer button. Wait until the transfer is complete.
get tool from powerlink (or ask support to put it in ftp site).
just run ./emcgrab.sh. albeit all the warnings, never had problems running it on live production system.
For ESX system, run vm-support
Other EMC Software
Power Path = FC link management, fail over when more than one link exist.
(plain = full version) - complete package
Utility Kit Paower Path = stip down version, probably handles some limited fail over case when EMC LUN are used as boot disks. Porbably mainly used so that multiple HBA + Path to Clariion SP does not create multiple disks view on host side.
But it is not intented as HA solution.
Snapshot = copy on write, but copy block of the old data to separate SnapLUN.
Master disk don't get out of space problem due to snapshot. SnapLUN out of space would mean snapshot no longer valid.
A bit more writting than NetApp snapshot w/in normal data area.
BCV - Business Continuity Volume - cloned version of snapshot. Essentially, parallel write to BCV volume (same size as
master LUN). THen when snapshot taken, parallel write stops. This give backup a chance to get consistent data.
SnapMirror - multisite mirroring, based on snapshots.
SAN Switch Config
Cisco MDS 9100 series SAN switch.
EMC san switch DS-24 M2 / McData Teton 4500
This is a 24 port rebranded mcdata san switch, 2 Gbit SC connector w/ GBIC module needed.
serial port in front, 19200!! ask for password at first before doing anything.
cli commands :
telnet or ssh in to the box, only one active method, only 1 active connection at a time.
change superuser name and password. It will prompt to login right away.
everything will be shown on the screen.
Root> config security userRights administrator admin bla
Root> config security userRights administrator Administrator g..A..
All the rest of the commands assume run from Root> prompt
can enter each commandn to enter submenu, enter ? for available commands and quick description.
config security userRights show
List SuperUser and general user UserName for the switch CLI.
show feature Display additional license added to switch (extra ports enablement
config features installKey ASS9-yvZt-BH4T-Thd
Add the license key for extra 8 ports
Switch will reboot after the command is issued!!
Note that there is no simple way to remove license key once installed.
Docs actually says unremovable, so ensure which switch is addming license onto before issueing command.
ACKX-qfFU-gweH-R (some other 8 or 16 port key from EMC email of unknown source).
To reset password, hook up terminal to serial port, power on switch,
it will automatically be logged in as admin.
then issue cmd "passwd admin" to reset password.
EMC/McData DS-24 M2 switch default accounts:
serial login: admin/level-2 or password?
ews (embeded web server):
cli via telnet
http web port 80 (use ie, apperently mozilla and firefox can't login, keep looping!)
The above are case sensitive. The default are same for both places, but they are maintained as two different password
set. Upon first login, both CLI and Web prompt for new password, but same default can be used again.
Newer version switch software web and cli password are sync, at least when set from web page.
EMC/McData DS-8 and DS-16 switch default accounts:
EMC/McData DS-32 switch default accounts:
- InterOperability e-lab