Deploying Fiber Channel solution with S4810 as FSB and S5000 as FCF

The purpose of this article is to provide the best practice Fiber Channel implementation with S4810 as FSB and S5000 as FCF

The datacenter networking mainly comprise of Ethernet and Storage traffics. The Ethernet segment is called LAN (Local Area Network) and the storage segment is called SAN (Storage Area Network).

Fiber Channel (FC) is a high speed network technology used for connecting different Storage devices.

In the traditional networking environment there is dedicated links for LAN and SAN traffic. Once the FCoE (FiberChannel over Ethernet) technology emerged the Datacenter and Enterprise migrated from traditional infrastructure to converged infrastructure using the same link for carrying data and storage traffic.

Fiber Channel common terminologies

What is FIP (FCoE Initialization Protocol)?

Fiber Channel over Ethernet initialization protocol (FIP) ensures a virtual point-to-point link between Enodes and Fiberchannel Forwarded switches over transit FCoE-enabled bridges.

The below mentioned steps are carried out by FIP.

• Enode sends FIP VLAN Request to discover FCoE VLANs on All-FCoE-Forwarders MAC (01:01:18:01:00:02)
• Enode receives FIP VLAN Response from FCF.
• Enode sends FIP Solicitation to discover FCFs using well known MAC (01:01:18:01:00:02)
  • FIP Solicitation contains information regarding Enode MAC, Enode WWNN and Enode FCoE frame size.
• Enode receives FIP Advertisement from FCF
  • FIP Advertisement contains information like Priority, WWNN of FCF, Fabric Descriptor and FKA ADV Period.
• Enode sends FIP FC ELS FLOGI to address 0xFFFFFE
• Enode receives FC ELS ACC (FLOGI) from FCF.

What is FIP Snooping Bridge (FSB)?

1. It acts as a bridge between FCoE capable devices (CNAs) and the Fiber Channel Forwarder (FCF).
2. It monitors the FIP frames and dynamically configures ACLs to permit authorized point-to-point traffic.
3. It is also known as FCoE transit switch

What is FCF (Fiber Channel Forwarders)?

It is a combination of FCoE and FC switch. It accepts the connections from the FSB to pass through it and reach the Fiber Channel switch.

What is N_Port ID Virtualization (NPIV)?

                This is a feature which helps multiple N_Port IDs to share a single physical N_Port.

NPIV initialization

• N_Port sends FDISC to address 0xFFFFFE to obtain an additional address
  • Fabric Discovery (FDISC)
    • "Discover Fabric Service Parameters"
    • works just like FLOGI
• N_Port sends PLOGI to address 0xFFFFFC to register this additional address with the name server
• N_Port sends SCR to address 0xFFFFFD to register for state change notifications.
• repeat NPIV initialization for next address

What is Zoning?

• It is a mechanism that ensures that only nodes that are part of a zone can communicate with each other.
• prevents unauthorized access of storage assets.
• prevents undesired host-host communication and fabric-wide Registered State Change notification disruptions

What is Data Center Bridging (DCB)?

                The Data Center Bridging mainly comprises of three factors.

• Priority Flow Control (PFC)
• Enhanced transmission Selection (ETS)
• Data Center Bridging Exchange (DCBX)

What are the different types of ports?

N_Port (Node Port): A port on the End Node (Either Host or Storage)

F_Port (Fabric Port): A port on the FC switch which provides the necessary information to Node ports.

E_Port (Expansion Port): A port connected between FC switches to form an Inter-Switch Link.

 In this document we will discuss scenarios where Dell Datacenter products are carrying Fiberchannel traffic. This will also allow us to show the best practice configuration steps need to be followed.

Deploying the Dell S4810 FSB with Dell S5000 series switch (NPIV)

.

Steps performed to configure Dell S4810 as FSB:

NOTE: FIP snooping is only supported on S4810 Ethernet interfaces in standalone or VLT mode but not in a switch stack. FCoE is also not supported over VLT LAG.

1. Allocate blocks in cam profile for ‘FcoeAcl’ if not there.

> For example

cam-acl l2acl 4 ipv4acl 4 ipv6acl 0 ipv4qos 2 l2qos 1 l2pt 0 ipmacacl 0 vman-qos 0 ecfmacl 0 fcoeacl 2

1. Enable FIP snooping.

# feature fip-snooping

# fip-snooping enable

1. Enable lldp globally.

# protocol lldp

# no disable

1. Configure service class to honor dot1p values.

# service-class dynamic dot1p

1. Configure the default vlan for FIP process.

By default vlan 1 will be used as native vlan. It can be changed to other value from 2 – 4094.

Create a vlan by typing below commands

# interface vlan <vlan id>

# no shutdown

# exit

Make the new vlan as default vlan

#default vlan-id <vlan id>

1. Create the vlan for FCoE traffic

This vlan will only carry the FCoE traffic.

Create a vlan by typing below commands

# interface vlan <vlan id>

# no shutdown

# exit

1. Configure the interfaces connecting to the Servers/hosts (Enodes):

# portmode hybrid – configuring portmode hybrid will allow the port to communicate on the native/default VLAN untagged, and other vlans can be tagged after wards

# switchport – this will configure the interface as a layer 2 port.

# protocol lldp –

# dcbx port-role auto-downstream – This command will configure the port-role as a dcbx capable downstream port. It will push the dcbx settings to the Server CNA ports if the server CNA is configured as a willing port.

# no shutdown

# exit

1. Configure the interfaces connecting to the Uplinks (FCF):

# portmode hybrid – configuring portmode hybrid will allow the port to communicate on the native/default VLAN untagged, and other vlans can be tagged after wards

# switchport – this will configure the interface as a layer 2 port.

# fip-snooping port-mode fcf – This allows FIP frames to pass through for FLOGI & FDISC process

# protocol lldp –

# dcbx port-role auto-upstream – This command will configure the port-role as a dcbx capable upstream port. It will learn the dcbx settings from the upstream FCF port.

# no shutdown

# exit

1. Configure spanning tree protocol RSTP

# protocol spanning-tree rstp

# no disable

# bridge-priority <value>

1. Configure the server facing ports as spanning-tree edge ports

# spanning-tree rstp edge-port

1. Save the configuration

# write memory

Steps performed to configure Dell S5000 as NPIV:

1. Enable Fiber Channel capability

#feature fc

1. Create DCB MAP and configure priority-based flow control (PFC) and enhanced transmission selection (ETS) settings for LAN and SAN traffic.

Priorities are mapped to priority groups using the ‘priority-pgid’ command. In this example, priorities 0, 1, 2, 4, 5, 6, and 7 are mapped to priority group 0. Priority 3 is mapped to priority-group 1.

# dcb-map <DCBMAP>

# priority-group 0 bandwidth 60 pfc off [60% bandwidth for LAN traffic with PFC off]

# priority-group 1 bandwidth 40 pfc on [40% bandwidth for SAN traffic with PFC on]

# priority-pgid 0 0 0 1 0 0 0 0 [Map FCoE priority 3 to priority group 1]

# exit
 

1. Create the vlan for FCoE traffic

This vlan will only carry the FCoE traffic.

Create a vlan by typing below commands

# Interface vlan <vlan id>

# no shutdown

# exit

1. Create FCoE MAP.

# fcoe-map <FCMAP>

# fabric-id <ID> vlan <FCoE VLAN ID> [Link the FCoE MAP to the FCoE VLAN]

# fc-map <0efcxx> [Value ranges from 0EFC00 – 0EFCFF]

# exit

1. Apply the DCB MAP to the respective interfaces. The FCoE MAP is used to map FCoE traffic to the respective VLAN/interfaces.

The FCoE MAP needs to be applied to interface connected to FSB (S4810) and to the FC interface connecting to the FC switch.

# interface fi 0/x [Port connected to FC switch]

# fabric <FCMAP> [Apply FCoE MAP]

# no shut

# exit

# interface te 0/x[Port connected to FSB (S4810)]

# dcb-map <DCBMAP>[Apply DCB MAP]

# fcoe-map <FCMAP> [Apply FCoE MAP]

# end

1. Save the config

# write memory

Commands for the troubleshooting and what to look for

1. Show interface status

Check for the fields like status, speed, duplex and VLAN in the output.

1. Show interface <interface-id>

Check for MTU settings and make sure there is no CRC or throttles on the interfaces.

1. Show fip-snooping config

Check for FIP Snooping Feature status, Global config status, FC-MAP.

1. Show fip-snooping fcf

Check for FCF’s MAC address, connecting interface and VLAN fields.

1. Show fip-snooping enode

Check for Enode MAC address, interface on which Enode is connected, VLAN, FC-ID and FCF-MAC

1. Show fip-snooping sessions

Check for FCoE MAC in the output