Dell EMC Networking OS9 - Memory monitoring in S4810, MXL and Z9000 switches

This article explains about the list of outputs that need to be collected in the cases where memory leaking is suspected.

Memory leak in most cases is a very slow process and extensive log analysis is required on such cases.

We may require multiple samples of the outputs for our analysis and to understand which part of the memory is affected.

 Memory monitoring tool

A memory monitoring tool was incorporated into the S4810, MXL and Z9000 9.1(0.0P6+) releases.  This tool captures detailed data allocation usage valuable in troubleshooting potential memory leaks.  The memory monitoring tool can be accessed through a console or telnet session. 

1: Enable Memory Monitoring Tool

 To enable the memory monitoring tool.

 Usage e.g.:

remote-exec cp f10mc -i 0x87

 Example:

Command:

Z9000#remote-exec cp f10mc -i 0x87

System Response:

f10mc -i 0x87

f10mc: Initialized for malloc, pool, pool_cache, with filter for F10 allocations

SStk-0 #

 2: Collecting Data

2a: Collecting Periodic General Memory Data

Collect the data periodically (4 to 6 hour intervals) each day for several days.  To collect the output, execute the following command.  The following command displays general memory details of the system.  The cycle number in the output will increment by one each time the output is collected.

 Usage e.g.:

remote-exec cp f10mc -r 0

  Example:

Command:

Z9000#remote-exec cp f10mc –r 0

System Response:

f10mc -r 0

{

-----------<cycle 3>-----------

#### This cycle: 46176 bytes held by 56 allocations ####

  M:       1888(16); +2329280(18371), -2329280(18371)

  P:          0(0); +0(0), -0(0)

 PC:      44288(40); +3053056(11226), -3053056(11226)

#### Since MCInit: 708902 bytes held by 2112 allocations ####

  M:      52006(1338); +110122134(928732), -110070128(927394)

  P:          0(0); +0(0), -0(0)

 PC:     656896(774); +148369408(548411), -147712512(547637)

}

SStk-0 #

The following output may be requested in addition to the memory monitoring tool output.

 Related Commands:

remote-exec cp top -b -o res all

remote-exec cp netstat –m

remote-exec cp vmstat –s

show process memory

show process memory management

show memory

show clock

 2b: Collecting Detailed Memory Data

After the collection period for the "remote-exec cp f10mc –r 0" command has completed, the output of the following command will be requested.  The following command will provide precise memory details of the system.

 Usage e.g.:

 remote-exec cp f10mc -r 1

 Example:

Command:

Z9000#remote-exec cp f10mc -r 1

System Response:

f10mc -r 1

{

PC:mbpl:0xC71F5200:256:3

PC:mclpl:0xD7C24800:2048:3

PC:mbpl:0xC71F3500:256:3

{output truncated}

PC:mbpl:0xCAFC7F00:256:2

PC:mclpl:0xD7C4D000:2048:2

-----------<cycle 3>-----------

#### This cycle: 286144 bytes held by 266 allocations ####

  M:       3520(16); +21362(417), -20514(415)

  P:          0(0); +0(0), -0(0)

 PC:     282624(250); +1129472(2200), -1126912(2197)

#### Since MCInit: 599392 bytes held by 542 allocations ####

  M:       5216(20); +67838(1335), -62622(1315)

  P:          0(0); +0(0), -0(0)

 PC:     594176(522); +3513088(6660), -2918912(6138)

}

SStk-0 #

 Engineering will evaluate the collected data and provide details on how to collect further information that will assist in identifying source for memory leak.  Output requested will be for suspect memory allocations:

Usage e.g.:

remote-exec cp f10mc –c 0x<memory address>

 Example:

Command:

Z9000# remote-exec cp f10mc –c 0xCE0DDD00

 System Response:

f10mc -c 0xCE0DDD00

PC:mbpl:0xCE0DDD00:256:2> 0xC0100EAD 0xC033DA54 0xC0137ADE 0xC0136DD0 0xC0142682 0xC013FA17 0xC0148C38 0xC037C808 0xC037BE3F 0xC09B805A

call trace:

0xC0100EAD in Xsoftintr

0xC033DA54 in softint_dispatch

0xC0137ADE in ipintr

0xC0136DD0 in ip_input

0xC0142682 in tcp_input

0xC013FA17 in syn_cache_get

0xC0148C38 in tcp_template

0xC037C808 in m_gethdr

0xC037BE3F in m_get

0xC09B805A in f10MC_pool_cache_get_paddr

 SStk-0 #

3:Disable Memory Monitoring

 To disable the memory monitoring tool after the data review is complete..

 Usage e.g.:

remote-exec cp f10mc -i 0x0

Example:

Command:

Z9000#remote-exec cp f10mc -i 0x0

System Response:

f10mc -i 0x0

f10mc: deinitialized

SStk-0 #