> 科技资讯 > Amazon S3 对象存储Java API操作记录(Minio与S3 SDK两种实现)

Amazon S3 对象存储Java API操作记录(Minio与S3 SDK两种实现)

科技资讯 投稿 13400℃ 0 评论

Amazon S3 对象存储Java API操作记录(Minio与S3 SDK两种实现)

缘起

S3对象存储的基本概念

S3是什么？

Amazon S3(Simple Storage Service对象存储出现得比较早且使用简单的RESTful API，于是成为了对象存储服务(Object Storage Service，OSS业内的标准接口规范。

S3的逻辑模型

Minio客户端方式操作S3

详细API文档：https://min.io/docs/minio/linux/developers/java/API.html

引入依赖

Maven:

<dependency>
    <groupId>io.minio</groupId>
    <artifactId>minio</artifactId>
    <version>8.5.2</version>
</dependency>

Gradle:

dependencies {
    implementation("io.minio:minio:8.5.2"
}

初始化客户端

private static final String HTTP_PROTOCOL = "http";

private MinioClient minioClient;
private String endpoint = "http://192.168.0.8:9200";
private String accessKey = "testKey";
private String secretKey = "testSecretKey";

public void init( throws MalformedURLException {
    URL endpointUrl = new URL(endpoint;
    try {
        // url上无端口号时，识别http为80端口，https为443端口
        int port = endpointUrl.getPort( != -1 ? endpointUrl.getPort( : endpointUrl.getDefaultPort(;
        boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol( ? false : true;
        //@formatter:off
        this.minioClient = MinioClient.builder(.endpoint(endpointUrl.getHost(, port, security
            .credentials(accessKey, secretKey.build(;
        //@formatter:on
        // 忽略证书校验，防止自签名证书校验失败导致无法建立连接
        this.minioClient.ignoreCertCheck(;
    } catch (Exception e {
        e.printStackTrace(;
    }
}

建桶

public boolean createBucket(String bucket {
    try {
        minioClient.makeBucket(MakeBucketArgs.builder(.bucket(bucket.build(;
    } catch (Exception e {
        e.printStackTrace(;
        return false;
    }
    return true;
}

删桶

public boolean deleteBucket(String bucket {
    try {
        minioClient.removeBucket(RemoveBucketArgs.builder(.bucket(bucket.build(;
        logger.info("删除桶[{}]成功", bucket;
    } catch (Exception e {
        e.printStackTrace(;
        return false;
    }
    return true;
}

判断桶是否存在

public boolean bucketExists(String bucket {
    try {
        return minioClient.bucketExists(BucketExistsArgs.builder(.bucket(bucket.build(;
    } catch (Exception e {
        e.printStackTrace(;
        return false;
    }
}

上传对象

public void upload(String bucket, String objectId, InputStream input {
    try {
        //@formatter:off
        minioClient.putObject(PutObjectArgs.builder(.bucket(bucket.object(objectId
                              .stream(input, input.available(, -1
                              .build(;
        //@formatter:on
    } catch (Exception e {
        e.printStackTrace(;
    }
}

下载对象

提供两个下载方法，一个将输入流返回，另一个用参数输出流写出

public InputStream download(String bucket, String objectId {
    try {
        return minioClient.getObject(GetObjectArgs.builder(.bucket(bucket.object(objectId.build(;
    } catch (Exception e {
        e.printStackTrace(;
    }
    return null;
}

public void download(String bucket, String objectId, OutputStream output {
    //@formatter:off
    try (InputStream input = minioClient.getObject(
        GetObjectArgs.builder(.bucket(bucket.object(objectId.build( {
        IOUtils.copyLarge(input, output;
    } catch (Exception e {
        e.printStackTrace(;
    }
    //@formatter:on
}

删除对象

public boolean deleteObject(String bucket, String objectId {
    //@formatter:off
    try {
        minioClient.removeObject(RemoveObjectArgs.builder(
                                 .bucket(bucket.object(objectId.build(;
    } catch (Exception e {
        e.printStackTrace(;
    }
    //@formatter:on
    return true;
}

判断对象是否存在

public boolean objectExists(String bucket, String key {
    //@formatter:off
    try {
        // minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在
        minioClient.statObject(StatObjectArgs.builder(
                               .bucket(bucket.object(key.build(;
    } catch (Exception e {
        return false;
    }
    //@formatter:on
    return true;
}

完整代码

import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;

import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import io.minio.BucketExistsArgs;
import io.minio.GetObjectArgs;
import io.minio.MakeBucketArgs;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import io.minio.RemoveBucketArgs;
import io.minio.RemoveObjectArgs;
import io.minio.StatObjectArgs;

public class S3MinioClientDemo {
    private static final Logger logger = LoggerFactory.getLogger(S3MinioClientDemo.class;
    private static final String HTTP_PROTOCOL = "http";

    private MinioClient minioClient;
    private String endpoint = "http://192.168.0.8:9200";
    private String accessKey = "testKey";
    private String secretKey = "testSecretKey";

    public void init( throws MalformedURLException {
        URL endpointUrl = new URL(endpoint;
        try {
            // url上无端口号时，识别http为80端口，https为443端口
            int port = endpointUrl.getPort( != -1 ? endpointUrl.getPort( : endpointUrl.getDefaultPort(;
            boolean security = HTTP_PROTOCOL.equals(endpointUrl.getProtocol( ? false : true;
            //@formatter:off
            this.minioClient = MinioClient.builder(.endpoint(endpointUrl.getHost(, port, security
                    .credentials(accessKey, secretKey.build(;
            //@formatter:on
            // 忽略证书校验，防止自签名证书校验失败导致无法建立连接
            this.minioClient.ignoreCertCheck(;
        } catch (Exception e {
            e.printStackTrace(;
        }
    }

    public boolean createBucket(String bucket {
        try {
            boolean found = minioClient.bucketExists(BucketExistsArgs.builder(.bucket(bucket.build(;
            if (found {
                logger.info("桶名[{}]已存在", bucket;
                return false;
            }
            minioClient.makeBucket(MakeBucketArgs.builder(.bucket(bucket.build(;
        } catch (Exception e {
            e.printStackTrace(;
        }
        return true;
    }

    public boolean deleteBucket(String bucket {
        try {
            minioClient.removeBucket(RemoveBucketArgs.builder(.bucket(bucket.build(;
            logger.info("删除桶[{}]成功", bucket;
        } catch (Exception e {
            e.printStackTrace(;
            return false;
        }
        return true;
    }

    public boolean bucketExists(String bucket {
        try {
            return minioClient.bucketExists(BucketExistsArgs.builder(.bucket(bucket.build(;
        } catch (Exception e {
            e.printStackTrace(;
            return false;
        }
    }

    public void upload(String bucket, String objectId, InputStream input {
        try {
          //@formatter:off
          minioClient.putObject(PutObjectArgs.builder(.bucket(bucket.object(objectId
                     .stream(input, input.available(, -1
                     .build(;
          //@formatter:on
        } catch (Exception e {
            e.printStackTrace(;
        }
    }

    public InputStream download(String bucket, String objectId {
        try {
            return minioClient.getObject(GetObjectArgs.builder(.bucket(bucket.object(objectId.build(;
        } catch (Exception e {
            e.printStackTrace(;
        }
        return null;
    }

    public void download(String bucket, String objectId, OutputStream output {
        //@formatter:off
        try (InputStream input = minioClient.getObject(
                GetObjectArgs.builder(.bucket(bucket.object(objectId.build( {
            IOUtils.copyLarge(input, output;
        } catch (Exception e {
            e.printStackTrace(;
        }
        //@formatter:on
    }

    public boolean objectExists(String bucket, String objectId {
        //@formatter:off
        try {
            // minio客户端未提供判断对象是否存在的方法，此方法中调用出现异常时说明对象不存在
            minioClient.statObject(StatObjectArgs.builder(
                    .bucket(bucket.object(objectId.build(;
        } catch (Exception e {
            return false;
        }
        //@formatter:on
        return true;
    }

    public boolean deleteObject(String bucket, String objectId {
        //@formatter:off
        try {
            minioClient.removeObject(RemoveObjectArgs.builder(
                    .bucket(bucket.object(objectId.build(;
        } catch (Exception e {
            e.printStackTrace(;
        }
        //@formatter:on
        return true;
    }

    public void close( {
        minioClient = null;
    }

}

Amazon S3 SDK方式操作S3

官方API文档：https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html

引入依赖

Maven:

<dependency>
    <groupId>com.amazonaws</groupId>
    <artifactId>aws-java-sdk-s3</artifactId>
    <version>1.11.300</version>
</dependency>

Gradle:

dependencies {   
    implementation 'com.amazonaws:aws-java-sdk-s3:1.11.300'
}

初始化客户端

private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class;

private AmazonS3 s3client;
private String endpoint = "http://192.168.0.8:9200";
private String accessKey = "testKey";
private String secretKey = "testSecretKey";

public void init( throws MalformedURLException {
    URL endpointUrl = new URL(endpoint;
    String protocol = endpointUrl.getProtocol(;
    int port = endpointUrl.getPort( == -1 ? endpointUrl.getDefaultPort( : endpointUrl.getPort(;

    ClientConfiguration clientConfig = new ClientConfiguration(;
    clientConfig.setSignerOverride("S3SignerType";
    clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase(;

    // 禁用证书检查，避免https自签证书校验失败
    System.setProperty("com.amazonaws.sdk.disableCertChecking", "true";
    // 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题
    System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true";
    AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey;
    // 创建 S3Client 实例
    AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig;
    s3client.setEndpoint(endpointUrl.getHost( + ":" + port;
    s3client.setS3ClientOptions(S3ClientOptions.builder(.setPathStyleAccess(true.build(;
    this.s3client = s3client;
}

建桶

public boolean createBucket(String bucket  {
    String bucketName = parseBucketName(bucket;
    try {
        if (s3client.doesBucketExist(bucketName {
            logger.warn("bucket[{}]已存在", bucketName;
            return false;
        }
        s3client.createBucket(bucketName;
    } catch (Exception e {
        e.printStackTrace(;
    }
    return true;
}

删桶

public boolean deleteBucket(String bucket  {
    try {
        s3client.deleteBucket(bucket;
        logger.info("删除bucket[{}]成功", bucket;
    } catch (Exception e {
        e.printStackTrace(;
        return false;
    }
    return true;
}

判断桶是否存在

public boolean bucketExists(String bucket  {
    try {
        return s3client.doesBucketExist(bucket;
    } catch (Exception e {
        e.printStackTrace(;
    }
    return false;
}

上传对象

public void upload(String bucket, String objectId, InputStream input  {
    try {
        // 创建文件上传的元数据
        ObjectMetadata meta = new ObjectMetadata(;
        // 设置文件上传长度
        meta.setContentLength(input.available(;
        // 上传
        s3client.putObject(bucket, objectId, input, meta;
    } catch (Exception e {
        e.printStackTrace(;
    }
}

下载对象

public InputStream download(String bucket, String objectId  {
    try {
        S3Object o = s3client.getObject(bucket, objectId;
        return o.getObjectContent(;
    } catch (Exception e {
        e.printStackTrace(;
    }
    return null;
}

public void download(String bucket, String objectId, OutputStream out  {
    S3Object o = s3client.getObject(bucket, objectId;
    try (InputStream in = o.getObjectContent( {
        IOUtils.copyLarge(in, out;
    } catch (Exception e {
        e.printStackTrace(;
    }
}

删除对象

public boolean deleteObject(String bucket, String objectId  {
    try {
        s3client.deleteObject(bucket, objectId;
    } catch (Exception e {
        e.printStackTrace(;
        return false;
    }
    return true;
}

判断对象是否存在

public boolean existObject(String bucket, String objectId  {
    try {
        return s3client.doesObjectExist(bucket, objectId;
    } catch (Exception e {
        e.printStackTrace(;
        return false;
    }
}

完整代码

import java.io.InputStream;
import java.io.OutputStream;
import java.net.MalformedURLException;
import java.net.URL;

import org.apache.tomcat.util.http.fileupload.IOUtils;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import com.amazonaws.ClientConfiguration;
import com.amazonaws.Protocol;
import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3Client;
import com.amazonaws.services.s3.S3ClientOptions;
import com.amazonaws.services.s3.model.ObjectMetadata;
import com.amazonaws.services.s3.model.S3Object;

/**
 * S3对象存储官方SDK实现
 *
 * @author ZhangChenguang
 * @date 2023年2月2日
 */
@SuppressWarnings("deprecation"
public class S3SdkDemo {
    private static final Logger logger = LoggerFactory.getLogger(S3SdkDemo.class;

    private AmazonS3 s3client;
    private String endpoint = "http://192.168.0.8:9200";
    private String accessKey = "testKey";
    private String secretKey = "testSecretKey";
    
    public void init( throws MalformedURLException {
        URL endpointUrl = new URL(endpoint;
        String protocol = endpointUrl.getProtocol(;
        int port = endpointUrl.getPort( == -1 ? endpointUrl.getDefaultPort( : endpointUrl.getPort(;

        ClientConfiguration clientConfig = new ClientConfiguration(;
        clientConfig.setSignerOverride("S3SignerType";
        clientConfig.setProtocol(Protocol.valueOf(protocol.toUpperCase(;

        // 禁用证书检查，避免https自签证书校验失败
        System.setProperty("com.amazonaws.sdk.disableCertChecking", "true";
        // 屏蔽 AWS 的 MD5 校验，避免校验导致的下载抛出异常问题
        System.setProperty("com.amazonaws.services.s3.disableGetObjectMD5Validation", "true";
        AWSCredentials awsCredentials = new BasicAWSCredentials(accessKey, secretKey;
        // 创建 S3Client 实例
        AmazonS3 s3client = new AmazonS3Client(awsCredentials, clientConfig;
        s3client.setEndpoint(endpointUrl.getHost( + ":" + port;
        s3client.setS3ClientOptions(S3ClientOptions.builder(.setPathStyleAccess(true.build(;
        this.s3client = s3client;
    }

    public boolean createBucket(String bucket  {
        try {
            s3client.createBucket(bucket;
        } catch (Exception e {
            e.printStackTrace(;
        }
        return true;
    }

    public boolean deleteBucket(String bucket  {
        try {
            s3client.deleteBucket(bucket;
            logger.info("删除bucket[{}]成功", bucket;
        } catch (Exception e {
            e.printStackTrace(;
            return false;
        }
        return true;
    }
    
    public boolean bucketExists(String bucket  {
        try {
            return s3client.doesBucketExist(bucket;
        } catch (Exception e {
           e.printStackTrace(;
        }
        return false;
    }

    public void upload(String bucket, String objectId, InputStream input  {
        try {
            // 创建文件上传的元数据
            ObjectMetadata meta = new ObjectMetadata(;
            // 设置文件上传长度
            meta.setContentLength(input.available(;
            // 上传
            s3client.putObject(bucket, objectId, input, meta;
        } catch (Exception e {
            e.printStackTrace(;
        }
    }

    public InputStream download(String bucket, String objectId  {
        try {
            S3Object o = s3client.getObject(bucket, objectId;
            return o.getObjectContent(;
        } catch (Exception e {
            e.printStackTrace(;
        }
        return null;
    }

    public void download(String bucket, String objectId, OutputStream out  {
        S3Object o = s3client.getObject(bucket, objectId;
        try (InputStream in = o.getObjectContent( {
            IOUtils.copyLarge(in, out;
        } catch (Exception e {
            e.printStackTrace(;
        }
    }

    public boolean existObject(String bucket, String objectId  {
        try {
            return s3client.doesObjectExist(bucket, objectId;
        } catch (Exception e {
            e.printStackTrace(;
            return false;
        }
    }

    public boolean deleteObject(String bucket, String objectId  {
        try {
            s3client.deleteObject(bucket, objectId;
        } catch (Exception e {
            e.printStackTrace(;
            return false;
        }
        return true;
    }

    public void close(  {
        s3client = null;
    }
}

遇到的问题

1、bucket名称必须是小写，不支持下划线

    处理方式：写方法转换下bucket名称，将大写转小写，将下划线替换为中划线。

2、minio客户端下载非官方S3存储的文件时，如果响应头的Content-Length与实际文件大小不符，会导致minio客户端包装的okhttp3报错

Caused by: java.net.ProtocolException: unexpected end of stream
        at okhttp3.internal.http1.Http1ExchangeCodec$FixedLengthSource.read(Http1ExchangeCodec.java:430 ~[okhttp-3.14.9.jar:?]
        at okhttp3.internal.connection.Exchange$ResponseBodySource.read(Exchange.java:286 ~[okhttp-3.14.9.jar:?]
        at okio.RealBufferedSource$1.read(RealBufferedSource.java:447 ~[okio-1.17.2.jar:?]
        at com.jiuqi.nr.file.utils.FileUtils.writeInput2Output(FileUtils.java:83 ~[nr.file-2.5.7.jar:?]
        at com.jiuqi.nr.file.impl.FileAreaServiceImpl.download(FileAreaServiceImpl.java:395 ~[nr.file-2.5.7.jar:?]
        ... 122 more

抓包发现问题的图：

PS：客户现场部署的S3是浪潮公司提供的，如果现场遇到这个情况，就不要固执去找对方对线了，完全没用。

总结

希望对读者有所用处，觉得写得不错和有帮到你，欢迎点个赞，您的支持就是我的鼓励！

编程笔记 » Amazon S3 对象存储Java API操作记录(Minio与S3 SDK两种实现)

赞同 (50) or 分享 (0)
游客 发表我的评论   换个身份
取消评论

表情
(0)个小伙伴在吐槽