Introduction
Reinitializing the Dell EMC Unity OE should be considered a last resort, and by nature is fully destructive, completely removing all data and configuration that may already exist. This procedure will reset the system to a 'back-to-factory' state based on the same OE version that was last in use.
When this procedure might apply
- If the Converged Network Adapter ports (CNA ports 4 & 5) on the SPs have been set to a protocol that is not desired (the ports can be initially set as Fibre Channel or Ethernet), then the only way of changing the protocol personality of the ports is by reinitializing the system ('back to factory state').
- If the system has been installed without the CNA ports populated (Management IP assigned, Unisphere Initialization wizard run, but is not yet in production), and you have ETHERNET SFPs, you can physically add the SFPs to the CNA ports, then perform the "Commit IO Ports" by navigating to Unisphere > System > System View > Enclosures > DPE. Committing IO ports requires multiple SP reboots.
CAUTION: If the CNA ports are empty and you need to add FC SFPs, this step will not work. The system will require reinstallation with the FC SFPs installed in the ports.
- If the wrong I/O Modules have been initialized on the system, you can use the "reinit" procedure to change the modules to a different type
- If some other condition exists whereby the System cannot be restored to Normal Mode, but has no apparent hardware issues (and is not in production).
Note 1: In all cases when changing from one SLIC type to another, or changing protocols on the CNA ports, you must change the hardware after both SPs have been placed into Service Mode, and then perform the system re-initialization. If the re-initialization process fails in any scenario, you should contact your Service Provider for further assistance.
Note 2: You cannot use this procedure to change out the system drives (drives 0-3) from one type or size to another, because the reinit process itself must access the Unity image that is stored on the system drives.
Before you begin
Before starting the reinitialization, make sure you still have your original license file. If you are unable to find it, download it from the array before starting the reinit. The license file will be located on the primary SP under /EMC/backend/CEM/licenses/. Without the license file the system cannot be initialized after the reinit.
Performing a System reinitialization from Unisphere
- If the system is accessible from Unisphere, you can perform the system "reinit", or reinitialization, by navigating to the System > Service > Service Tasks section. Here, you must first verify if the SPs are in Normal Mode, and then place each SP into Service Mode by running the "Enter Service Mode" function. If both SPs are in Normal Mode, place the non-Primary SP into Service Mode first, wait for it to completely reboot, and then place the Primary SP into Service Mode (GUI shows as "Storage Processor X (Primary)".
- If you are Changing SLIC type do the following, else go to step 3). Once both SPs are in Service Mode, you may remove SLICs from the SP slots, or SFPs from the CNA ports, and replace with different SLICs or SFP type. For example, if you needed to change the CNA port protocol from FC to Ethernet, or vice versa.
Note: It is possible that pulling out and changing IO modules will panic the SPs. Simply wait for the SPs to completely reboot back to Service Mode, then continue with the procedure.
- With both SPs in "Service Mode" (aka, Rescue Mode), you must log into Unisphere as the "service" user with service password, and then run the System > Service > Service Task called "Reinitialize", which will then reinstall the Unity Operating Environment from an image stored on the System Drives.
- Once a system has been reinitialized, which can take approximately an hour, it will require IP initialization (Connection Utility) and Unisphere Initialization wizard steps.
Note 1: Use the
SP Fault LEDs to verify when the system is ready for IP initialization (SP fault LED should blink 1 second Amber, followed by 3 seconds Blue, OR, can be at a steady Blue condition--either scenario indicates that the System is ready for IP initialization on the Management Port).
Note 2: In lieu of running the Connection Utility, you can use svc_initial_config service command to assign the Management IP address and user-friendly name. Run this from the primary SP (the SP running the ECOM service). Use "pidof ECOM" or "pgrep ECOM" to determine which SP is running the ECOM service. Enable SSH on the Management IP using svc_ssh -enable. The customer should then login to the System from a browser, use user account admin and default password Password123# , and complete the Initialization Wizard steps.
Performing a System reinitialization using CLI with SSH connection
Note: The following procedure will re-initialize the system at the version that is already on the System.
Caution: As a recommended precaution, you should always inject root to the system, when possible, before performing a reinit in the event recovery steps are required.
- Use Putty to SSH to the system management IP, login with service user credentials.
Note: If the Management IP is not available over the network, you will need local on-site access to the SPs Service Port, along with a Windows client that has the IPMITool installed. As an alternative, connect to either SP over IPMI, and assign a temporary network IP address to the SP using the svc_network service command. This will also enable SSH access by default to this IP. You can also do this for the other SP, if required. From that point, you can connect to the system remotely.
- Determine if SPs are running in "Normal Mode" vs. "Rescue Mode" (i.e., Service Mode):
spa:~$ svc_diag
======== Now executing basic state ========
* System Serial Number is: FNM001531xxxxx
* System Friendly Host Name is:
* Current Software version: upc_nextUnity_mcs_201606071611_upcBuilder-4.1.0.7769613-GNOSIS_RETAIL
* Unisphere IP address(es): xx.xx.xx.xx
* SSH Enabled: true
* FIPS mode: Disabled
* Boot Mode: Normal Mode --> Indicates that this SP is running in Normal Mode
spb:~$ ssh peer
spa:~$ svc_diag
======== Now executing basic state ========
* System Serial Number is: FNM001531xxxxx
* System Friendly Host Name is:
* Current Software version: upc_nextUnity_mcs_201606071611_upcBuilder-4.1.0.7769613-GNOSIS_RETAIL
* Unisphere IP address(es): xx.xx.xx.xx
* SSH Enabled: true
* FIPS mode: Disabled
* Boot Mode: Rescue Mode --> This SP is running in Rescue Mode (aka Service Mode) in this example
- Before reinitializing, both SPs must be in Rescue/Service Mode. Use the following example to put an SP into Service Mode:
Note: When you log into the system, by default you log into the Primary SP. It is recommended that you "ssh peer" and then place the peer SP into Service Mode first, wait for it to completely reboot, and then place the Primary SP into Service Mode.
spb:~$ svc_rescue_state -s --> This will set the Rescue State flag on the SP
spb:~$ svc_shutdown -r --> Reboot the SP and it will come back up in Rescue Mode
spb:~$ svc_diag
-----abridged--------
* Boot Mode: Rescue Mode
- Once SPs are in Service Mode, you can remove or rearrange SLICs, swap SFPs in the CNA ports to change from FC to Ethernet protocol, or vice versa (SPs may reboot after removing SLICs, if so, they will come back in Service Mode)
- Perform the system reinitialization from either SP, using the following:
spb:~$ svc_reinit
- Answer system prompt, then wait approximately an hour for the system to reset itself back to a factory fresh state.
- Verify SP Fault LEDs are showing the correct behavior before running CU and the Configuration wizard.
Note: In lieu of running the Connection Utility, you can use svc_initial_config service command to assign the Management IP address and user-friendly name. Run this from the primary SP (the SP running the ECOM service). Use "pidof ECOM" or "pgrep ECOM" to determine which SP is running the ECOM service. Enable SSH on the Management IP using svc_ssh -enable. The customer should then login to the System from a browser, use user account admin and default password Password123# , and complete the Initialization Wizard steps.
Performing a System reinitialization using the IPMITool (No IP network)
Use this method only if you cannot log into the system with Unisphere or SSH over the network.
- Download and install the IPMI Tool 1.0 on your laptop (available from support.emc.com). The IPMI Tool is a Windows command prompt utility that is used to establish a Serial-over-LAN connection to the Service Processors (SPs).
- Use the following steps to connect and login to the SPs:
a) Connect your laptop to a portable switch, and run Ethernet cables from the switch to each SP's Service LAN port
Note: If you do not have a switch, you can connect to a single Service LAN port directly from your laptop, but are limited to one SP, though you can still "ssh peer" to access the other SP from the command line.
b) Configure the laptop with an IP address for the Service LAN network: 128.221.1.249, netmask 255.255.255.0, no gateway required
c) Open two command prompts on the laptop, and use the following syntax to connect to each SP:
c:\> ipmitool.exe -I lanplus -C 3 -U console -P <unity_serial_number> -H 128.221.1.252 sol activate -->SPA Internal IP
c:\> ipmitool.exe -I lanplus -C 3 -U console -P <unity_serial_number> -H 128.221.1.253 sol activate -->SPB Internal IP
Note: For the IPMI command, the User is "console" and password is the Serial Number of the system for the SOL activation. If having issues connecting, disable Cisco AnyConnect services and your wireless network adapter.
d) Once the IPMI connection is established, press the Enter key once. You should see a login prompt. At this point, login using the "service" user account and password. If the service user password has never been changed, it is "service" by default.
Note: IPMI sessions can be fickle, and may drop back to the command prompt at times. Simply arrow up to the IPMI command to reconnect--it should not require login again.
- Once logged in, verify whether each SP is in Normal or Rescue Mode using the svc_diag command.
- Place each SP into Rescue Mode using the following methodology:
Note: As with other reinit procedures, it is recommended that you place the non-Primary SP into Service Mode first, reboot it and wait for it to completely reboot, then place the Primary SP into Service Mode. You can identify the Primary SP when the following command returns a running process for the ECOM service.
spa:~$ pidof ECOM
28771 --> The non-Primary SP does not run the ECOM service
spb:~$ svc_rescue_state -s
spb:~$ svc_shutdown -r
Note: Wait for the SP to come back up, repeat process on the other SP, as needed
- Perform the system reinitialization from either SP and wait at least an hour for it to complete.
spb:~$ svc_reinit
- Verify SP Fault LEDs are showing correct behavior before performing the IP initialization (Connection Utility) & running the Unisphere Configuration wizard.
Note: Use svc_initial_config to assign a name and Management IP to the system in lieu of the CU, then run the Configuration Wizard from a browser connection. The default login to Unisphere is with user admin and Password123# .
State of SP Fault LED after successful reinitialization
Note: Applies to all reinitialization methods shown in this KB article)
Note: Refer to the Unity Hardware Information Guide for location and descriptions of LED indicators (
http://bit.ly/unityinfohub 'Get to Know Your Storage System') or
KB 488128
- When the Unity OE installation is complete, the SP Fault LEDs should display the following, indicating that the system is ready for IP initialization:
One second Amber, followed by a 3 seconds Blue, on each SP
- Or, if DHCP is setup on the customer network, the SP fault LED may show:
A solid Blue LED condition
Note: In either case, the system is ready for IP initialization. Refer to the Unity Installation Guide for a complete chart of system LED indicator conditions.
- If the SP Fault LEDs are alternating one second Amber then Blue, the SP is in Service Mode, which would indicate that the system is NOT READY for IP initialization, and you should connect to the SPs using the IPMI Tool to investigate.
After reinitializing:
- Configuration - Once re-initializing is complete, the system will boot back up. You must run the Connection Utility to reset the Management IP address, and then go through the Configuration Wizard as if it were a new installation. The Connection Utility can be downloaded from Unity product page on the support pages.
- Restore - No one-step restore procedure is available for all Unity configurations that you had before re-initializing. Configuration settings can be backed up from the Service page. A Save Configuration button allows you to save the configuration to a file. However, there is no way to restore this configuration automatically or manually. This "save config" text file is provided to be a guide for the administrator to manually recreate the setup. By referring to this text file all configurations should be done manually by the administrator and all data should be restored from backup media.
- Licensing - You will have to reload the license keys during the reconfiguration. You can either go to the license page on Dell Online Support and download the keys or use the saved keys that you might have received from an email earlier.