Ceph安装radosgw和Python连接访

Ceph安装radosgw和Python连接访问

1、安装一台CentOS7，并且安装Ceph到这台机器，具体安装在前面的博客中已经提到过（http://blog.csdn.net/chuan_day/article/details/60577049）。

      radosgw节点的主机名为rgw-node1.up.com,短名称：rgw-node1。

我的hosts

[root@ceph-admin ~]# cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.0.50 ceph-admin

192.168.0.51 ceph1

192.168.0.52 ceph2

192.168.0.53 ceph3

192.168.0.54 ceph4

192.168.0.60 rgw-node1.up.com rgw-node1

192.168.0.80 ceph-client

2、安装完后将原先的Ceph集群的admin节点执行以下命令，目的是将ceph.conf等文件传递到rgw-node1节点上：

      ceph-deploy config push rgw-node1

在ceph-admin上执行以上命令。

3、在ceph-admin(管理节点上创建对象网关密钥)

[dgb@ceph-admin ceph]$ sudo ceph-authtool --create-keyring /etc/ceph/ceph.client.radosgw.keyring

执行完后会在/etc/ceph生成相应文件。

相应的还需执行赋予权限的操作：

[dgb@ceph-admin ceph]$ sudo chmod +r ceph.client.radosgw.keyring

4、为rgw实例生成网关用户和密码，rgw实例名是gateway(这个实例名肯定可以改，因为ceph的对象网关还能多网关部署)

[dgb@ceph-admin ceph]$ sudo ceph-authtool /etc/ceph/ceph.client.radosgw.keyring -n client.radosgw.gateway --gen-key

5、给密钥添加权限

[dgb@ceph-admin ceph]$ sudo ceph-authtool  -n client.radosgw.gateway --cap osd 'allow rwx' --cap mon 'allow rwx' /etc/ceph/ceph.client.radosgw.keyring

6、将密钥添加到集群中

[dgb@ceph-admin ceph]$ ceph auth add client.radosgw.gateway -i /etc/ceph/ceph.client.radosgw.keyring

7、将密钥分配给rgw节点

[dgb@ceph-admin ceph]$ sudo scp /etc/ceph/ceph.client.radosgw.keyring  rgw-node1:/etc/ceph/ceph.client.radosgw.keyring

这里所谓的分配其实就是把文件拷到rgw-node1节点上

8、在rgw-node1中的ceph.conf中添加如下内容

[client.radosgw.gateway]

host = rgw-node1

keyring = /etc/ceph/ceph.client.radosgw.keyring

rgw socket path = /var/run/ceph/ceph.radosgw.gateway.fastcgi.sock

log file = /var/log/ceph/client.radosgw.gateway.log

rgw dns name = rgw-node1.up.com

rgw print continue = false

那么如果实例名不是gateway的话，这里就要相应的修改。

9、在ceph-admin节点执行创建radosgw，安装在rgw-node1

[dgb@ceph-admin my-cluster]$ ceph-deploy --overwrite-conf rgw create rgw-node1

这里的--overwrite-conf得加。

10、判断rgw-node1节点上是否安装radosgw成功

[root@rgw-node1 ~]# netstat -nlp | grep -i 7480

tcp        0      0 0.0.0.0:7480            0.0.0.0:*               LISTEN      850/radosgw

或者

[dgb@rgw-node1 ~]$ curl http://rgw-node1:7480
<?xml version="1.0" encoding="UTF-8"?>

<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">

  <Owner>

<ID>anonymous</ID>

<DisplayName></DisplayName>

</Owner>

        <Buckets></Buckets>

</ListAllMyBucketsResult>

到此对象网关已经安装完成。

下面介绍通过s3 API来访问对象存储

1、确保rgw-node1可以访问Ceph集群

[dgb@rgw-node1 ~]$ sudo ceph -s -k /etc/ceph/ceph.client.radosgw.keyring  --name client.radosgw.gateway

    cluster bbdf2c52-8f97-46ae-b78a-e512a58d5b8e

     health HEALTH_OK

     monmap e1: 4 mons at {ceph-admin=192.168.0.50:6789/0,ceph1=192.168.0.51:6789/0,ceph2=192.168.0.52:6789/0,ceph3=192.168.0.53:6789/0}

            election epoch 96, quorum 0,1,2,3 ceph-admin,ceph1,ceph2,ceph3

     osdmap e35: 3 osds: 3 up, 3 in

            flags sortbitwise,require_jewel_osds

      pgmap v122: 112 pgs, 7 pools, 1588 bytes data, 171 objects

            109 MB used, 45937 MB / 46046 MB avail

                 112 active+clean

[dgb@rgw-node1 ~]$ radosgw-admin user create --uid=dgb --display-name="dgb" --email=dinggaob@126.com -k /etc/ceph/ceph.client.radosgw.keyring --name client.radosgw.gateway

{

    "user_id": "dgb",

    "display_name": "dgb",

    "email": "dinggaob@126.com",

    "suspended": 0,

    "max_buckets": 1000,

    "auid": 0,

    "subusers": [],

    "keys": [

        {

            "user": "mona",

            "access_key": "2O3HASV93KW7QS920QF6",

            "secret_key": "VK7kkbukQEu5Z2bRRy7kgIBa0rHsaRfsEZVvBRXH"

        }

    ],

    "swift_keys": [],

    "caps": [],

    "op_mask": "read, write, delete",

    "default_placement": "",

    "placement_tags": [],

    "bucket_quota": {

        "enabled": false,

        "max_size_kb": -1,

        "max_objects": -1

    },

    "user_quota": {

        "enabled": false,

        "max_size_kb": -1,

        "max_objects": -1

    },

    "temp_url_keys": []

}

此时，最好将两个密钥记录一下：

 "access_key": "2O3HASV93KW7QS920QF6",

  "secret_key": "VK7kkbukQEu5Z2bRRy7kgIBa0rHsaRfsEZVvBRXH"

3、搭建下DNS服务器，我就搭建在rgw-node1上

所以一下操作都在rgw-node1上

yum install bind* -y

vi /etc/resolve.conf

search up.com

nameserver 192.168.0.60

[root@rgw-node1 ~]# cat /etc/named.conf

options {
listen-on port 53 { 127.0.0.1;192.168.0.60; };
listen-on-v6 port 53 { ::1; };
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
memstatistics-file "/var/named/data/named_mem_stats.txt";
allow-query     { localhost;192.168.0.0/24; };

zone "up.com" IN {

        type master;

        file "db.up.com";

        allow-update  { none; };

};

以上标红的全部是需要新加的，如果配置过DNS服务器的话  这个就比较容易的。接下去创建db.up.com

[root@rgw-node1 ~]# cat /var/named/db.up.com

@ 86400 IN SOA up.com. root.up.com. (
20091028 ; serial yyyy-mm-dd
10800 ; refresh every 15 min
3600 ; retry every hour
3600000 ; expire after 1 month +
86400 ); min ttl of 1 day

@ 86400 IN NS up.com.

@ 86400 IN A 192.168.0.60

* 86400 IN CNAME @

好了准备工作算是做好了，验证下dns服务器：

nslookup rgw-node1.up.com

输出相关的IP，主机名就为正确。

在安装一台ceph-client用来当做客户端

1、安装ceph软件

2、安装s3cmd软件

在这个http://s3tools.org/download下载

然后安装：

解压s3cmd-master.zip，然后进入目录后又setup.py文件在同一个目录下执行：

python setup.py install

执行s3cmd --configure

[root@rgw-node1 ~]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.

Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.

Access Key [2O3HASV93KW7QS920QF6]: 

Secret Key [VK7kkbukQEu5Z2bRRy7kgIBa0rHsaRfsEZVvBRXH]: 

Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.

S3 Endpoint [rgw-node1.up.com:7480]: 

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used

if the target S3 system supports dns based buckets.

DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.rgw-node1.up.com:7480]: 

Encryption password is used to protect your files from reading

by unauthorized persons while in transfer to S3

Encryption password [oracle]: 

Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3

servers is protected from 3rd party eavesdropping. This method is

slower than plain HTTP, and can only be proxied with Python 2.7 or newer

Use HTTPS protocol [No]: 

On some networks all internet access must go through a HTTP proxy.

Try setting it here if you can't connect to S3 directly

HTTP Proxy server name: 

New settings:

  Access Key: 2O3HASV93KW7QS920QF6

  Secret Key: VK7kkbukQEu5Z2bRRy7kgIBa0rHsaRfsEZVvBRXH

  Default Region: US

  S3 Endpoint: rgw-node1.up.com:7480

  DNS-style bucket+hostname:port template for accessing a bucket: %(bucket)s.rgw-node1.up.com:7480

  Encryption password: oracle

  Path to GPG program: /usr/bin/gpg

  Use HTTPS protocol: False

  HTTP Proxy server name: 

  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] n

Save settings? [y/N] y

Configuration saved to '/root/.s3cfg'

因为我是已经执行过s3cmd --configure所以一些值已经有了，这里需要注意的是：

Use HTTPS protocol [No]: 这个选项选择NO，默认是YES，不然会连接不上毕竟HTTPS，比较简单。

然后安装就完成了。

来检验是否真的使用s3cmd命令行能操作ceph对象存储。

以上都在ceph-client节点上执行，下面的命令也一样：

[root@ceph-client s3cmd-master]# s3cmd mb s3://test1

ERROR: S3 error: 405 (MethodNotAllowed)

我执行以上命令但是出现了错误，不知道为什么，我又执行了s3cmd ls  并没有错误，那么我大胆的得出结论我的s3cmd是可以访问，经过一大堆的查询资料，我看到

一个帖子有类似s3cmd mb s3://bucketX，突然觉得是不是得有大写啊，尝试：

[root@ceph-client s3cmd-master]# s3cmd mb s3://Test1

成功。但是还是奇怪为什么一定要这样 难道是规定的，这不怎么科学。

通过python程序连接访问ceph对象存储

1、在ceph-client节点上安装boto
下载boto-develop.zip
通过python setup.py install
很快就安装好。
2、创建connCeph.py,如下：
import boto

import boto.s3.connection

access_key = '2O3HASV93KW7QS920QF6'

secret_key = 'VK7kkbukQEu5Z2bRRy7kgIBa0rHsaRfsEZVvBRXH'

conn = boto.connect_s3(

        aws_access_key_id = access_key,

        aws_secret_access_key = secret_key,

        host = 'rgw-node1.up.com',port = 7480,

        is_secure=False,               # uncomment if you are not using ssl

        calling_format = boto.s3.connection.OrdinaryCallingFormat(),

        )

for bucket in conn.get_all_buckets():

        print "{name}\t{created}".format(

                name = bucket.name,

                created = bucket.creation_date,

        )

bucket = conn.create_bucket('my-new-bucket1')
让人奇怪的是，当我使用python创建bucket时，命名为“My-new-bucket1”，然后就报错了，提示我不能使用大写字母作为bucket名字，
而使用了小写后顺利通过了，也只能这样了，程序执行结果正常：
[root@ceph-client home]# python connCeph.py 

Test1

2017-05-10T10:08:39.459Z

bucketX

2017-05-10T10:08:16.293Z

my-new-bucket1

2017-05-10T10:50:42.377Z

my-new-bucket2

2017-05-10T10:56:44.754Z

以上便是全部的初步搭建radosgw和python程序访问的步骤和代码。