Custom Endpoints for AWS CLI: S3 and MinIO

When working with AWS S3 or S3-compatible services like MinIO, you may need to use custom endpoints instead of the default AWS endpoints. This is particularly common when you're working with a self-hosted S3 service or when you're accessing S3 services in a private network.

Here are short notes on using custom endpoints with the AWS CLI for S3 operations, specifically for MinIO:

• Setting Custom Endpoints: To communicate with an S3-compatible service like MinIO, use the --endpoint-url option in the AWS CLI to specify the service's custom endpoint URL.

Example Command:

aws --endpoint-url http://minio-service.minio.svc:9000 s3 ls s3://data-science-pipelines

This command lists the contents of the data-science-pipelines bucket on a MinIO server accessible at http://minio-service.minio.svc:9000.

• Handling Credentials: If you encounter an "Unable to locate credentials" error, configure the AWS CLI with the necessary credentials using the aws configure command. You can set the custom endpoint again and provide the access key and secret key when prompted:

aws configure

Enter the MinIO access key and secret key when prompted.

• No Region or Output Format: Since MinIO does not use AWS regions, you can leave the default region name and default output format blank during configuration.

• Successful Configuration and Usage: After configuring the credentials, you can use the AWS CLI to access the MinIO service with the custom endpoint, as shown in the previous example.

• Persistence: These settings are saved in the AWS CLI configuration files, typically located in ~/.aws/credentials and ~/.aws/config. This means that after the initial setup, you do not need to re-enter the credentials.

• Session-Based Usage: If you prefer not to save the credentials in the configuration files, you could use environment variables or pass credentials directly in the CLI commands for single sessions.

• Set the Custom Endpoint in the Profile: After configuring the profile with your credentials, you can add the endpoint URL to the profile's configuration in the ~/.aws/config file to avoid passing the endpoint URL each time in your cli.

Open the file in a text editor and add the following lines under your profile:

[default]
endpoint_url = http://minio-service.minio.svc:9000

By following these steps, you can easily interact with MinIO or any other S3-compatible storage service using the AWS CLI with custom endpoints.