Some article numbers may have changed. If this isn't what you're looking for, try searching all articles. Search articles

Article Number: 000055896

ECS: Bucket Management - Overview and Troubleshooting

Summary: This article describes how to manage buckets, considerations and troubleshooting.

This article may have been automatically translated. If you have any feedback regarding its quality, please let us know using the form at the bottom of this page.

Article Content

Symptoms

This article describes buckets, bucket management and basic troubleshooting tips.

Cause

N/A

Resolution

Bucket Management

What is a bucket?

Buckets are containers for objects created in a namespace and sometimes considered a logical container for sub-tenants. In S3, containers are called buckets and this term has been adopted by ECS. In Atmos, the equivalent of a bucket is a subtenant; in Swift, the equivalent of a bucket is a container, and for CAS, a bucket is a CAS pool. Buckets are global resources in ECS. Each bucket is created in a namespace and each namespace is created in a Replication Group (RG).

Buckets are global resources that can span multiple sites. Bucket creation involves assigning it to a namespace and a RG. The bucket level is where ownership and file or CAS access is enabled.

Buckets can be accessed via different tools at the same time, i.e. access the same bucket with GeoDrive and S3 Browser.

How to create a bucket?

ECS offers several ways to create a bucket. To create a bucket the following must be done:

Upload an ECS license.
Select a set of nodes to create at least one storage pool.
Create a VDC.
Create at least one Replication Group.
Create at least one namespace.

Bucket creation can be done via the ECS GUI or Rest API:

ECS GUI

Log in to the ECS system using your browser and select the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Getting started with ECS - 'Create a bucket'

Rest API

Refer to the ECS REST API guide for management and object operations.

What protocols are supported?

When a object user is created - 'Manage' - 'Users' - 'New Object User', the following protocol options are available:

S3
Atmos
Swift
CAS
File access via NFSv3 (S3)
File access to CIFS via GeoDrive (S3)

What bucket options are supported? - Bucket creation

By creating a new bucket - 'Manage' - 'Buckets' - 'New Bucket', the following options are available:

File system

Required for NFSv3. Required on bucket creation. Note this cannot be altered once the bucket is created.

Refer to article ECS: Best practices for mounting NFS exports for further information.

CAS

For CAS operations, this must be set when creating the bucket. CAS buckets cannot be accessed with other protocols and the CAS option must be set when creating a bucket.

Refer to article ECS: CAS User & Bucket Administration Best Practices for further information.

Metadata Search

Enable indexing of Objects based on user and system metadata, and cannot be enabled after save.

Refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Buckets' - 'Working with buckets in the ECS Portal' - 'Bucket settings' - 'Metadata search fields'

Access during outage (ADO)

ECS allows you to configure buckets to be accessible if one of your VDC fails.

Refer to article ECS: The importance of the ADO (Access During Outage) parameter for further information.

Server-side Encryption

Enable to allow data-at-rest encryption, which encrypts data on the storage disks to reduce the risk of data loss by physical theft of disks.

Refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Buckets' - 'Working with buckets in the ECS Portal' - 'Bucket settings'

Quota

Soft and Hard Quota can be assigned on either, bucket or namespace level.

Refer to article ECS: How to increase the quota set on an ECS Bucket for further information.

Bucket Tagging

Bucket tags are key-value pairs that you can associate with a bucket. Max 10 tags per bucket.

Refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Buckets' - 'Working with buckets in the ECS Portal' - 'Bucket settings' - 'Bucket tagging'

Retention

Retention can be set on bucket or namespace level. Retention applies to CAS and S3 buckets.

Refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Namespaces' - 'Working with namespaces in the ECS Portal' - 'Namespace settings' - 'Retention periods and policies'

What bucket options are supported? - Bucket edit

Once a bucket is created, there is an option to modify bucket settings via the 'Edit Bucket' drop down with 'Edit ACL' and 'Edit Policy'. Refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Manage' - 'Buckets' - 'Edit Bucket'.

The privileges a user has when accessing a bucket are set using an Access Control List (ACL). You can assign ACLs for a user, for a set of pre-defined groups, such as all users, and for a custom group.When you create a bucket and assign an owner to it, an ACL is created that assigns a default set of permissions to the bucket owner - the owner is, by default, assigned full control. You can modify the permissions assigned to the owner or you can add new permissions for a user by selecting the 'Edit ACL' operation for the bucket:

Edit ACLs

S3 allows to set the following permissions:

Read	Allows user to list the objects in the bucket.
Read ACL	Allows user to read the bucket ACL.
Write	Allows user to create or update any object in the bucket.
Write ACL	Allows user to write the ACL for the bucket.
Execute	Sets the execute permission when accessed as a file system. This permission has no effect when the object is accessed using the ECS object protocols.
Full Control	Allows user to Read, Write, Read ACL, and Write ACL. Note: Non object owners can Read, Write, Read ACL, and Write ACL if the permission has been granted or can only list the objects.
Privileged Write	Allows user to perform writes to a bucket or object when the user does not have normal write permission. Required for CAS buckets.
Delete	Allows user to delete buckets and objects. Required for CAS buckets.
None	User has no privileges on the bucket.

Multiple users can be created with different permissions. Refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Buckets' - 'Working with buckets in the ECS Portal' - 'Set ACLs'

For CAS refer to article ECS: Set up bucket ACLs for CAS for further information.

Bucket policies provide greater flexibility than ACLs and allow fine grained control over permissions for bucket operations and for operations on objects within the bucket. Policy conditions are used to assign permissions for a range of objects that match the condition and are used to automatically assign permissions to newly uploaded objects. You can modify the permissions assigned to objects or you can add new permissions by selecting the 'Edit Policy' operation for the bucket:

Edit Policy

The ECS GUI provides a Bucket Policy Editor to create a bucket policy for an existing bucket. Policies are defined in JSON format and the syntax used for policies is the same as that used for Amazon AWS:

kA23a000000GG6FCAW_1_0
You can use bucket policies in the following typical scenarios:

Grant bucket permissions to a user
Grant bucket permissions to all users
Automatically assign permissions to created objects

For examples on the above operations, refer to the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide - 'Buckets' - 'Working with buckets in the ECS Portal' - 'Set bucket policies' - 'Bucket policy scenarios'.

How to test buckets?

There are various ways to test S3 connectivity, these are a few examples for accessing them. Simplest method to access a S3 bucket via S3 Browser.

WinSCP:	ECS: Connecting ECS buckets with WINSCP
s3test.sh script:	ECS: How to test S3 create bucket, upload object, read object, delete object, delete bucket with the s3test.sh script
S3curl:	ECS: How to perform basic s3 operations on ECS 3.x using the s3curl.pl script.
S3Browser:	ECS : How to connect ECS with S3Browser?

What are the bucket limitations?

Buckets are limited by the maximum capacity configured to the system the bucket is configured on.

Buckets are multi-protocol enabled, except CAS buckets. CAS buckets cannot be shared.

Client access and API operation availability at the namespace, bucket and object levels during a TSO is governed the following ADO options set at the namespace and bucket level:

Off (default) - Strong consistency is maintained during a temporary outage.
On - Eventually consistent access is allowed during a temporary site outage.

Data consistency during a TSO is implemented at the bucket level. Configuration is set at the namespace level, which sets the default ADO setting in place for ADO during new bucket creation and can be overridden at new bucket creation; meaning TSO can be configured for some buckets and not for others.

How to delete a bucket?

To delete a bucket, make sure the bucket is entirely empty. This not only includes current objects, also, if configured, versions and delete markers need to be deleted before you can remove that bucket from.

If it is not possible to delete the bucket manually, a lifecycle policy can be set on the bucket to delete it. Refer to article ECS: How to delete a S3 bucket with many objects in it using lifecycle policies for further information.

Additional Information

Considerations

Before creating a buckets remember that some options cannot be changed later:

Replication Group
Server-side Encryption
File System
CAS
Metadata Search

Make sure the buckets are configured properly, else a migration might be required.

The following rules apply to the naming of S3 buckets in ECS:

Must be between one and 255 characters in length. (S3 requires bucket names to be 1 255 characters long)
Can include dot (.), hyphen (-), and underscore (_) characters and alphanumeric characters ([a-zA-Z0-9])
Can start with a hyphen (-) or alphanumeric character.
Cannot start with a dot (.)
Cannot contain a double dot (..)
Cannot end with a dot (.)
Must not be formatted as IPv4 address.
Bucket names must be unique within a namespace.
Namespace and bucket names should be DNS compatible since they can appear in a DNS record.

Bucket best practices:

Use buckets for specific environment, workflow, or uses. For instance: dev, test, finance, operations, etc.
In multi-site deployments, create buckets at the VDC site closest to the application accessing and updating the objects. There is overhead involved with checking the latest copy if the ownership of object is at a remote site.
For best performance, recommended to have less than 1000 buckets in a single namespace

For additional information, select the ECS Help ('?' icon) in the ECS GUI which links the ECS Administration Guide.

ECS Code bucket limitations

Refer to the latest ECS Release Notes for up to date fixes implemented on bucket limitations.

Troubleshooting articles

Article Properties

Affected Product

ECS Appliance

Last Published Date

19 Jul 2023

Version

Article Type

Solution

Welcome

Welcome to Dell