Since upgrading to DDMC 6.1 or 6.2, trying to upload RPM files from the GUI fails. This could also occur for DDOS when using the GUI (DDSM)

The DataDomain Management Center (DDMC) allows administrators to easily manage and supervise dozens of remote DDs and DDVEs at a time. For DDMC 6.1.2.40 and 6.2.0.30 versions there is a code defect in the way the GUI frontend (Apache) and backend (Tomcat) communicate, which can cause requests taking more than 20 seconds to be aborted. This issue has been reported to also affect the same versions of DDOS, as both products (DDMC and DDOS) share the same fundamental code and libraries.

This is typically the case when trying to upload a RPM file through the GUI, as unless the network between the browser and the DDMC is significantly fast, the 1.5 GiB+ RPM file will take more than 20 seconds to upload, and hence result in hitting this issue and the upload failing. So depending on the file size and the speed of the network the user may see progress up to a given percentage, only to fail later on.

One way to confirm this issue is to check the Tomcat backend logs for the GUI while trying to upload the RPM file:

# log view debug/sm/ddms.info ... #### Uploading of the file is started 14 Jul 2019 05:50:35,816 com.datadomain.webf.server.auth.AuthFilter.addSkippedUrl:109 [root] INFO : Adding skipped URL: /upload 14 Jul 2019 05:50:35,817 com.datadomain.webf.server.rpc.RPCProxy.beginCall:150 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Starting request com.datadomain.ddms.server.upload.UploadReq 14 Jul 2019 05:50:35,826 com.datadomain.webf.server.rpc.RPCProxy.endCall:235 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Completed request administration.UploadReq succesfully #### Progress is being logged for the upload 14 Jul 2019 05:50:38,845 com.datadomain.ddms.server.upload.UploadProgressHandler.execute:36 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Percent=1, Status=Pending 14 Jul 2019 05:50:40,798 com.datadomain.ddms.server.upload.UploadProgressHandler.execute:36 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Percent=2, Status=Pending 14 Jul 2019 05:50:42,818 com.datadomain.ddms.server.upload.UploadProgressHandler.execute:36 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Percent=3, Status=Pending ... 14 Jul 2019 05:50:52,838 com.datadomain.ddms.server.upload.UploadProgressHandler.execute:36 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Percent=9, Status=Pending 14 Jul 2019 05:50:54,815 com.datadomain.ddms.server.upload.UploadProgressHandler.execute:36 [administration.20d8a3aa59d8159a8a546543de9b16338] INFO : Percent=10, Status=Pending #### But eventually, 20 seconds exactly after the file started being uploaded, it aborts with "end of file" 14 Jul 2019 05:50:55,816 com.datadomain.ddms.server.upload.UploadServlet.sendResponse:158 [root] ERROR: org.apache.commons.fileupload.FileUploadBase$IOFileUploadException: Processing of multipart/form-data request failed. java.io.EOFException at org.apache.commons.fileupload.FileUploadBase.parseRequest(FileUploadBase.java:367) at org.apache.commons.fileupload.servlet.ServletFileUpload.parseRequest(ServletFileUpload.java:126) at com.datadomain.ddms.server.upload.UploadServlet.sendResponse(UploadServlet.java:112) at com.datadomain.ddms.server.upload.UploadServlet.sendResponse(UploadServlet.java:32) at com.datadomain.webf.server.rpc.BaseServlet.executeRequest(BaseServlet.java:152) at com.datadomain.webf.server.rpc.BaseServlet.doPost(BaseServlet.java:108) ...

For these same DDMC versions, and while not related to the file upload issues, there is a minor issue (despite the CRITICAL tag) being reported in the logs as well, which is shown below for awareness:

SEVERE: An incompatible version [1.1.29] of the APR based Apache Tomcat Native library is installed, while Tomcat requires version [1.2.14]

This issue will be corrected (by uploading some internal libraries) at a later stage.

Root cause is because the RPM package being uploaded is bigger than 1.7 GiB. When file size is greater than 1.7 GiB the fileupload upgrade API doesn't return the right "contentLength" to calculate the percentage. This limit was 1.6 GiB in past versions, but still needed some internal library upgrade, which is missing in the mentioned releases.

 

 

This issue affecting DDMC / DDOS 6.1.2.40 and DDMC / DDOS 6.2.0.30 will be fixed in the upcoming releases:

• DDMC / DDOS 6.1.2.50 and later
• DDMC / DDOS 6.2.0.40 and later

Of course, if a customer is already on any of the defective releases, an alternate means of uploading the fixed DDMC / DDOS RPM will be necessary prior to running any subsequent upgrade. Customers not yet in DDMC 6.1.2.40 or DDMC 6.2.0.30 may upgrade to the fixed releases at any time without problems.

If the problem arises on a DDMC when it is a DDOS image the one that fails to be uploaded (DDOS RPM files for deploying to managed DDs), then the destination directory in the DDMC should be /ddr/var/ddr-releases/ instead of the one mentioned below (which is for DDMC RPMs for upgrading the DDMC itself, or DDOS).

To get the upgrade RPM uploaded to the DDMC /ddr/var/releases/ directory once in your desktop, so that the RPM shows in the GUI and you can upgrade from either the GUI or the CLI, you may use:

• SCP (SSH) : once the RPM file has been downloaded to another computer, use "scp DDMC-VERSION-BUILD.rpm sysadmin@DDMC_IP_ADDRESS:/ddr/var/releases/" (or the graphical tool of your choosing, but not WinSCP, which will not work with error "SFTP Connection Error Received unexpected end-of-file from SFTP server" error when connecting.  If you see this error, use the command line version of scp for your client OS) to send the RPM file from the computer to the appropriate directory in the DDMC VM. Using PuTTY as the CLI implementation for uploading the file, the following example command line and options will work : # pscp -scp -v -P 22 7.2.0.60-682124.rpm sysadmin@dd.example.com:/ddr/var/releases/
• FTP : you may follow the steps under " ENABLING FTP/FTPS ACCESS ON A DDR " to upload the RPM under /ddr/var/releases/, as in the Data Domain: How to collect/upload a support bundle (SUB) from a Data Domain Restorer (DDR) public KB article