如何用 Minio SDK 访问滴滴云对象存储

日期： 2018-12-19 分类：个人收藏 363次阅读

一、前言

滴滴云对象存储 S3 定位分布式海量数据存取服务。S3 提供标准的对象存储服务，您可以通过网络随时随地方便地存储和获取图片/音频/视频/文本/大文件等各类数据文件。

Minio 是一个开源的对象存储项目，它兼容亚马逊 S3 云存储服务接口，GitHub 地址是：https://github.com/minio。

Minio SDK 是一个非常轻量 SDK，本文将以 Python ( minio-py )、Java ( minio-java ) 和 Golang ( minio-go ) 三种语言，在外网 Centos6.4 版本环境上介绍如何用 Minio SDK 来访问滴滴云对象存储 S3。

二、密匙获取

在用 Minio SDK 前，我们先要获取自己对象存储 S3 的密匙 AK 和 SK，注册后先登陆滴滴云官网：https://www.didiyun.com ，进入“控制台”然后按下图示例申请 AK 和 SK（即图中的 SecretID 和 SecretKey）：
在这里插入图片描述

三、域名选择

可以根据不同环境使用不同的域名。目前可使用域名如下：

外网域名：s3.didiyunapi.com
DC2 域名: s3-internal.didiyunapi.com
内网域名：s3-gz.didiyunapi.com

后面将以外网域名为例来讲解。

四、 Minio SDK 访问滴滴云对象存储 S3 示例

1. minio-py 访问

minio-py 支持 Python2.7 及以上的版本，我们以 Centos7.4 自带的 2.7.5 版本为例。

1）用 PIP 安装 minio-py

pip install minio

2）例子代码

vi example.py

#!/usr/bin/python
# Import Minio library.

from minio import Minio
from minio.error import (ResponseError, NoSuchBucket, TooManyBuckets, BucketAlreadyOwnedByYou,
                         BucketAlreadyExists)
import time

minioClient = Minio('s3.didiyunapi.com',
                    access_key='AKDD002E38WCJIT6HXTHDTDZCMBCX',
                    secret_key='ASDD2QQb41K9MpxrXs5kcxmaRQ7MQyPCMh8qKMP9',
                    secure=True)

try:
    # bucket exists?
    if minioClient.bucket_exists("mybucket9") == False:
        # Creates a new bucket.
        minioClient.make_bucket("mybucket9", location="gz")
    else:
        print("Bucket exist!")

    # Put a file
    print(minioClient.fput_object('mybucket9', 'myobject', '/tmp/object'))

    # List all object paths in bucket `
    objects = minioClient.list_objects('mybucket9', prefix='',
                                  recursive=True)
    for obj in objects:
        print(obj.bucket_name + " | "+ str(obj.object_name.encode('utf-8')) + " | " + str(obj.is_dir) + " | " + str(obj.last_modified) + " | "
                  + obj.etag + " | " + str(obj.size) + " | " + str(obj.content_type))

except BucketAlreadyOwnedByYou as err:
    print("BucketAlreadyOwnedByYou: " + str(err))
except BucketAlreadyExists as err:
    print("BucketAlreadyExists: " + str(err))
except TooManyBuckets as err:
    print("TooManyBuckets: " + str(err))
except ResponseError as err:
    print("ResponseError: " + str(err))
    raise
except Exception as e:
    print("Unkonw exception: " + str(e))

3）执行

chmod a+x example.py

./example.py

2. minio-java 访问

1）下载 minio-java 的 JAR 包

先安装 JDK 后，再下载 minio-java：

wget http://repo1.maven.org/maven2/io/minio/minio/5.0.3/minio-5.0.3-all.jar

2）例子代码

在文件“minio-5.0.3-all.jar”同目录下：

vi Example.java

import java.io.IOException;
import java.security.NoSuchAlgorithmException;
import java.security.InvalidKeyException;
import java.io.InputStream;
import java.io.ByteArrayInputStream;

import org.xmlpull.v1.XmlPullParserException;

import io.minio.MinioClient;
import io.minio.errors.MinioException;

public class Example {
    public static void main(String[] args) throws NoSuchAlgorithmException, IOException, InvalidKeyException, XmlPullParserException {
        String endpoint = "s3.didiyunapi.com";
        String accessKeyID = "AKDD002E38WCJIT6HXTHDTDZCMBCX";
        String secretAccessKey = "ASDD2QQb41K9MpxrXs5kcxmaRQ7MQyPCMh8qKMP9";
        try {
            // Create a minioClient with the Endpoint, Access key and Secret key.
            MinioClient minioClient = new MinioClient(endpoint, accessKeyID, secretAccessKey);

            String myBucket = "mybucket9";
            // Check whether bucket exists or not.
            boolean found = minioClient.bucketExists(myBucket);
            if (found) {
                System.out.println("Bucket exists");

                // Get input stream to have content of 'my-objectname' from 'my-bucketname'
                InputStream stream = minioClient.getObject(myBucket, "myobject");

                // Read the input stream and print to the console till EOF.
                byte[] buf = new byte[16384];
                int bytesRead;
                while ((bytesRead = stream.read(buf, 0, buf.length)) >= 0) {
                    System.out.println(new String(buf, 0, bytesRead));
                }

                minioClient.removeObject(myBucket, "myobject");
                minioClient.removeBucket(myBucket);
            } else {
                System.out.println("Bucket does not exist");
            }
            
                        String newBucket = "mybucket10";
            found = minioClient.bucketExists(newBucket);
            if (found) {
                System.out.println("Bucket: " + newBucket + " exists");
            } else {
                System.out.println("Bucket: " + newBucket + " not exists");
                minioClient.makeBucket(newBucket);
                System.out.println("Bucket:" + newBucket + " is created successfully");

                StringBuilder builder = new StringBuilder();
                for (int i = 0; i < 3; i++) {
                    builder.append("Sphinx of black quartz, judge my vow: Used by Adobe InDesign to display font samples. ");
                    builder.append("(29 letters)\n");
                    builder.append("brave ghost pled; Watch Jeopardy!, Alex Trebeks fun TV quiz game.\n");
                    builder.append("- --\n");
                }
                ByteArrayInputStream bais = new
                ByteArrayInputStream(builder.toString().getBytes("UTF-8"));
                // Create an object
                minioClient.putObject(newBucket, "myobjcet1", bais, bais.available(), "application/octet-stream");
                bais.close();
                System.out.println("Object is uploaded successfully");
            }
        } catch(MinioException e) {
            System.out.println("Error occurred: " + e);
        }
    }
}

3）编译和运行

编译：

javac -cp "minio-5.0.3-all.jar"  Example.java

运行：

java -cp "minio-5.0.3-all.jar:." Example

3. minio-go 访问

1）安装

Golang 环境搭好和 GOPATH 环境变量配置好后：

go get -u github.com/minio/minio-go

cd $GOPATH/src

mkdir -p golang.org/x

cd $GOPATH/src/golang.org/x

git clone https://github.com/golang/crypto.git

git clone https://github.com/golang/net.git

git clone https://github.com/golang/sys.git

git clone https://github.com/golang/text.git

2）例子代码

vi example.go

package main

import (
    "github.com/minio/minio-go"
    "log"
)

func main() {
    endpoint := "s3.didiyunapi.com"
    accessKeyID := "AKDD002E38WCJIT6HXTHDTDZCMBCX"
    secretAccessKey := "ASDD2QQb41K9MpxrXs5kcxmaRQ7MQyPCMh8qKMP9"
    useSSL := true

    // Initialize minio client object.
    minioClient, err := minio.New(endpoint, accessKeyID, secretAccessKey, useSSL)
    if err != nil {
        log.Fatalln(err)
        return
    }

    myBucket := "mybucket10"
    myObject := "myobjcet1"
    found, err := minioClient.BucketExists(myBucket)
    if err != nil {
        log.Fatalln(err)
        return
    }
    if found {
        log.Println("Bucket ", myBucket, "found")
        // stat
        objInfo, err := minioClient.StatObject(myBucket, myObject, minio.StatObjectOptions{})
        if err != nil {
            log.Fatalln(err)
            return
        }
        log.Println(objInfo)

        // get object
        err = minioClient.FGetObject(myBucket, myObject, "/tmp/myobject1", minio.GetObjectOptions{})
        if err != nil {
            log.Fatalln(err)
            return
        }

        err = minioClient.RemoveObject(myBucket, myObject)
        if err != nil {
            log.Fatalln(err)
            return
        }

        err = minioClient.RemoveBucket(myBucket)
        if err != nil {
            log.Fatalln(err)
            return
        }
    } else {
        log.Println("Bucket not found")
    }
}

3）编译和运行

编译：