《利用Python 进行数据分析》 - 笔记(5)

问题导读：

1.合并数据集

2.重塑和轴向旋转

3.数据转换（待续）

解决方案：

合并数据集

（1）数据库风格的DataFrame合并

pandas的merge 函数 将通过一个或多个键将行连接起来
如果没有指定列，merge 就会直接依据相同列名的那一列进行连接

In [3]: df1 = pd.DataFrame(
...: {'key':['b','b','a','c','a','a','b'],
...: 'data1':range(7)}
...: )

In [4]: df1
Out[4]:
data1 key
0      0   b
1      1   b
2      2   a
3      3   c
4      4   a
5      5   a
6      6   b

[7 rows x 2 columns]

In [5]: df2 = pd.DataFrame(
...: {'key':['a','b','d'],
...: 'data2':range(3)}
...: )

In [6]: df2
Out[6]:
data2 key
0      0   a
1      1   b
2      2   d

[3 rows x 2 columns]

In [7]: pd.merge(df1,df2)
Out[7]:
data1 key  data2
0      0   b      1
1      1   b      1
2      6   b      1
3      2   a      0
4      4   a      0
5      5   a      0

[6 rows x 3 columns]

In [8]: pd.merge(df1,df2, on='key')
Out[8]:
data1 key  data2
0      0   b      1
1      1   b      1
2      6   b      1
3      2   a      0
4      4   a      0
5      5   a      0

[6 rows x 3 columns]

如果两个对象的列名不同，也可以在指定之后，进行合并

In [10]: df3 = pd.DataFrame({'lkey':['b','b','a','c','a','a','b'],
....: 'data1':range(7)})

In [11]: df4 = pd.DataFrame({'rkey':['a','b','d'],'data2':range(3)})

In [12]: pd.merge(df3,df4,left_on='lkey',right_on='rkey')
Out[12]:
data1 lkey  data2 rkey
0      0    b      1    b
1      1    b      1    b
2      6    b      1    b
3      2    a      0    a
4      4    a      0    a
5      5    a      0    a

[6 rows x 4 columns]

merge 做的是交集，其他方式有“left”，“right”，“outer”
outer 外连接，求取的是键的并集

In [13]: pd.merge(df1,df2,how='outer')
Out[13]:
data1 key  data2
0      0   b      1
1      1   b      1
2      6   b      1
3      2   a      0
4      4   a      0
5      5   a      0
6      3   c    NaN
7    NaN   d      2

[8 rows x 3 columns]

多对多

In [15]: df1 = pd.DataFrame({'key':['b','b','a','c','a','b'],
....: 'data1':range(6)})

In [16]: df2 = pd.DataFrame({'key':['a','b','a','b','d'],
....: 'data2':range(5)})

In [17]: pd.merge(df1,df2,on='key', how = 'left')
Out[17]:
data1 key  data2
0       0   b      1
1       0   b      3
2       1   b      1
3       1   b      3
4       5   b      1
5       5   b      3
6       2   a      0
7       2   a      2
8       4   a      0
9       4   a      2
10      3   c    NaN

[11 rows x 3 columns]

In [18]: pd.merge(df1,df2,on='key', how = 'right')
Out[18]:
data1 key  data2
0       0   b      1
1       1   b      1
2       5   b      1
3       0   b      3
4       1   b      3
5       5   b      3
6       2   a      0
7       4   a      0
8       2   a      2
9       4   a      2
10    NaN   d      4

[11 rows x 3 columns]

In [19]: pd.merge(df1,df2,on='key',how='inner')
Out[19]:
data1 key  data2
0      0   b      1
1      0   b      3
2      1   b      1
3      1   b      3
4      5   b      1
5      5   b      3
6      2   a      0
7      2   a      2
8      4   a      0
9      4   a      2

[10 rows x 3 columns]

In [21]: pd.merge(df1,df2,on='key',how='outer')
Out[21]:
data1 key  data2
0       0   b      1
1       0   b      3
2       1   b      1
3       1   b      3
4       5   b      1
5       5   b      3
6       2   a      0
7       2   a      2
8       4   a      0
9       4   a      2
10      3   c    NaN
11    NaN   d      4

[12 rows x 3 columns]

根据多个键进行合并，并传入一个由列名组成的列表即可

In [27]: left = pd.DataFrame({'key1':['foo','foo','bar'],
'key2':['one','two','one'],
'key3':[1,2,3]})

In [28]: right = pd.DataFrame({'key1':['foo','foo','foo','bar'],
'key2':['one','one','one','two'],
'rval':[4,5,6,7]})

In [29]: pd.merge(left,right,on=['key1','key2'],how='outer')
Out[29]:
key1 key2  key3  rval
0  foo  one     1     4
1  foo  one     1     5
2  foo  one     1     6
3  foo  two     2   NaN
4  bar  one     3   NaN
5  bar  two   NaN     7

[6 rows x 4 columns]

merge 的suffixes选项，用于附加到左右两个Dataframe对象的重叠列名上的字符串
用来解决为合并的相同列名的区分

In [30]: pd.merge(left,right,on='key1')
Out[30]:
key1 key2_x  key3 key2_y  rval
0  foo    one     1    one     4
1  foo    one     1    one     5
2  foo    one     1    one     6
3  foo    two     2    one     4
4  foo    two     2    one     5
5  foo    two     2    one     6
6  bar    one     3    two     7

[7 rows x 5 columns]

In [31]: pd.merge(left,right,on='key1',suffixes=('_left','_right'))
Out[31]:
key1 key2_left  key3 key2_right  rval
0  foo       one     1        one     4
1  foo       one     1        one     5
2  foo       one     1        one     6
3  foo       two     2        one     4
4  foo       two     2        one     5
5  foo       two     2        one     6
6  bar       one     3        two     7

[7 rows x 5 columns]

merge 函数的参数

(2）索引上的合并

DataFrame 的索引也可以被用作连接键
传入 left_index = True / right_index = True

In [37]: left1 = pd.DataFrame({'key':['a','b','a','a','b','c'],
'value':range(6)})

In [38]: right1 = pd.DataFrame({'group_val':[3.5,7]},index = ['a','b'])

In [39]: pd.merge(left1,right1,left_on='key',right_index=True)
Out[39]:
key  value  group_val
0   a      0        3.5
2   a      2        3.5
3   a      3        3.5
1   b      1        7.0
4   b      4        7.0

[5 rows x 3 columns]

In [40]: right1
Out[40]:
group_val
a        3.5
b        7.0

[2 rows x 1 columns]

In [41]: left1
Out[41]:
key  value
0   a      0
1   b      1
2   a      2
3   a      3
4   b      4
5   c      5

[6 rows x 2 columns]

对于层次化的索引，我们要以列表的形式指明用作合并键的多个列

In [48]: lefth = pd.DataFrame({'key1':['Ohio','Ohio','Ohio','Nevada','Nevada'],
'key2':[2000,2001,2002,2001,2002],
'data':np.arange(5.)})

In [49]: righth = pd.DataFrame(np.arange(12).reshape((6,2)),
index = [['Nevada','Nevada','Ohio','Ohio','Ohio','Ohio'],
[2001,2000,2000,2000,2001,2002]],
columns = ['event1','event2'])

In [50]: lefth
Out[50]:
data    key1  key2
0     0    Ohio  2000
1     1    Ohio  2001
2     2    Ohio  2002
3     3  Nevada  2001
4     4  Nevada  2002

[5 rows x 3 columns]

In [52]: righth
Out[52]:
event1  event2
Nevada 2001       0       1
2000       2       3
Ohio   2000       4       5
2000       6       7
2001       8       9
2002      10      11

[6 rows x 2 columns]

In [53]: pd.merge(lefth,righth,left_on=['key1','key2'],right_index=True)
Out[53]:
data    key1  key2  event1  event2
0     0    Ohio  2000       4       5
0     0    Ohio  2000       6       7
1     1    Ohio  2001       8       9
2     2    Ohio  2002      10      11
3     3  Nevada  2001       0       1

[5 rows x 5 columns]

In [54]: pd.merge(lefth,righth,left_on=['key1','key2'],right_index=True,how='outer')
Out[54]:
data    key1  key2  event1  event2
0     0    Ohio  2000       4       5
0     0    Ohio  2000       6       7
1     1    Ohio  2001       8       9
2     2    Ohio  2002      10      11
3     3  Nevada  2001       0       1
4     4  Nevada  2002     NaN     NaN
4   NaN  Nevada  2000       2       3

[7 rows x 5 columns]

同时使用合并双方的索引

In [55]: left2 = pd.DataFrame([[1.,2.],[3.,4.],[5.,6.]], index = ['a','c','e'],
....: columns= ['Ohio','Nevada'])

In [56]: right2 = pd.DataFrame([[7.,8.],[9.,10.],[11.,12],[13,14]],
....: index = ['b','c','d','e'],columns=['Missouri','Alabama'])

In [57]: left2
Out[57]:
Ohio  Nevada
a     1       2
c     3       4
e     5       6

[3 rows x 2 columns]

In [58]: right2
Out[58]:
Missouri  Alabama
b         7        8
c         9       10
d        11       12
e        13       14

[4 rows x 2 columns]

In [59]: pd.merge(left2,right2,how='outer',left_index=True,right_index=True)
Out[59]:
Ohio  Nevada  Missouri  Alabama
a     1       2       NaN      NaN
b   NaN     NaN         7        8
c     3       4         9       10
d   NaN     NaN        11       12
e     5       6        13       14

[5 rows x 4 columns]

DataFrame有一个join 实例方法，他可以更为方便地实现按索引合并
join 方法默认作的是左连接
该方法还可以使调用者的某个列跟参数DataFrame的索引进行连接

In [60]: left2.join(right2,how='outer')
Out[60]:
Ohio  Nevada  Missouri  Alabama
a     1       2       NaN      NaN
b   NaN     NaN         7        8
c     3       4         9       10
d   NaN     NaN        11       12
e     5       6        13       14

[5 rows x 4 columns]

In [61]: left2.join(right2)
Out[61]:
Ohio  Nevada  Missouri  Alabama
a     1       2       NaN      NaN
c     3       4         9       10
e     5       6        13       14

[3 rows x 4 columns]

In [62]: left1
Out[62]:
key  value
0   a      0
1   b      1
2   a      2
3   a      3
4   b      4
5   c      5

[6 rows x 2 columns]

In [63]: right1
Out[63]:
group_val
a        3.5
b        7.0

[2 rows x 1 columns]

In [64]: left1.join(right1,on='key')
Out[64]:
key  value  group_val
0   a      0        3.5
1   b      1        7.0
2   a      2        3.5
3   a      3        3.5
4   b      4        7.0
5   c      5        NaN

[6 rows x 3 columns]

join 一组DataFrame(后面会了解 concat 函数）

In [65]: another = pd.DataFrame([[7.,8.],[9.,10.],[11.,12.],[16.,17.]],
....: index=['a','c','e','f'],columns=['New York','Oregon'])

In [66]: left2.join([right2,another])
Out[66]:
Ohio  Nevada  Missouri  Alabama  New York  Oregon
a     1       2       NaN      NaN         7       8
c     3       4         9       10         9      10
e     5       6        13       14        11      12

[3 rows x 6 columns]

（3）轴向连接

轴向连接实际上可以理解为数据的连接、绑定、或堆叠

# coding=utf-8
import numpy as np
import pandas as pd

"""
numpy 的concatenate 函数：该函数用于合并原始的numpy 的数组
"""
arr = np.arange(12).reshape((3, 4))
print arr
'''
[[ 0  1  2  3]
[ 4  5  6  7]
[ 8  9 10 11]]
'''

print np.concatenate([arr, arr], axis=1)
'''
[[ 0  1  2  3  0  1  2  3]
[ 4  5  6  7  4  5  6  7]
[ 8  9 10 11  8  9 10 11]]
'''

"""
对于pandas 对象 Series 和 DataFrame 他们的带有标签的轴使你能够进一步推广数组的连接运算
"""

"""
（1） 各对象其他轴上的索引不同时，那些轴做的交集还是并集?
答案：默认是并集
"""

# concat 默认在axis=0 上工作
s1 = pd.Series([0,1], index=['a','b'])
# s1 = pd.Series([1,2,2], index=['d','b','f'])
s2 = pd.Series([2,3,4], index=['c','d','e'])
s3 = pd.Series([5,6], index=['f','g'])

print pd.concat([s1,s2,s3])

'''
a    0
b    1
c    2
d    3
e    4
f    5
g    6
dtype: int64
'''
# 如果要求在axis=1 进行操作则结果变成一个DataFrame

print pd.concat([s1,s2,s3], axis=1)

'''
0   1   2
a   0 NaN NaN
b   1 NaN NaN
c NaN   2 NaN
d NaN   3 NaN
e NaN   4 NaN
f NaN NaN   5
g NaN NaN   6
'''

# 传入的如果是join=‘inner’ 即可得到他们的交集

s4 = pd.concat([s1 * 5, s3])

print pd.concat([s1,s4], axis=1)

'''
dtype: int64
0  1
a   0  0
b   1  5
f NaN  5
g NaN  6

[4 rows x 2 columns]
'''

print pd.concat([s1,s4],axis=1,join='inner')

'''
0  1
a  0  0
b  1  5

[2 rows x 2 columns]
'''

# 指定轴进行连接操作，如果连接的两个Series 都没有该轴，则值为NaN

print pd.concat([s1,s4], axis=1, join_axes=[['a','c','b','e']])

'''
0   1
a   0   0
c NaN NaN
b   1   5
e NaN NaN

[4 rows x 2 columns]
'''

# 参与连接的片段的在结果中区分不开，这时候我们使用keys 参数建立一个层次化索引

result = pd.concat([s1,s2,s3], keys=['one','two','three'])

print result

'''
one    a    0
b    1
two    c    2
d    3
e    4
three  f    5
g    6
dtype: int64
'''
"""

s1 = pd.Series([1,2,6], index=['a','b','f'])
s2 = pd.Series([3,4], index=['c','d'])
s3 = pd.Series([5,6], index=['e','f'])

result = pd.concat([s1,s2,s3], keys=['one','two','three'])

print result

'''
one    a    1
b    2
f    6
two    c    3
d    4
three  e    5
f    6
dtype: int64
'''

"""

# print result.unstack()

# 沿axis=1 对series 进行合并，则keys 就会成为DataFrame的列头

print pd.concat([s1,s2,s3], axis=1, keys=['one','two','three'])

print pd.concat([s1,s2,s3], axis=1)

'''
dtype: int64
one  two  three
a    0  NaN    NaN
b    1  NaN    NaN
c  NaN    2    NaN
d  NaN    3    NaN
e  NaN    4    NaN
f  NaN  NaN      5
g  NaN  NaN      6

[7 rows x 3 columns]
0   1   2
a   0 NaN NaN
b   1 NaN NaN
c NaN   2 NaN
d NaN   3 NaN
e NaN   4 NaN
f NaN NaN   5
g NaN NaN   6

[7 rows x 3 columns]

'''

以上对于Series 的逻辑对于DataFrame也一样

# coding=utf-8
import pandas as pd
import numpy as np

df1 = pd.DataFrame(np.arange(6).reshape(3,2), index=['a','b','c'], columns=['one','two'])
df2 = pd.DataFrame(5+np.arange(4).reshape(2,2), index=['a','c'], columns=['three','four'])

print pd.concat([df1,df2],axis=1, keys=['level1','levle2'])

'''
level1       levle2
one  two   three  four
a       0    1       5     6
b       2    3     NaN   NaN
c       4    5       7     8

[3 rows x 4 columns]
'''

# 如果传入的不是列表而是一个字典，则字典的建就会被当做keys选项的值

dic = {'level1':df1, 'level2':df2}

print pd.concat(dic,axis=1)

'''
level1       level2
one  two   three  four
a       0    1       5     6
b       2    3     NaN   NaN
c       4    5       7     8

[3 rows x 4 columns]
'''

concat 函数的参数列表

# coding=utf-8
import pandas as pd
import  numpy as np

df1 = pd.DataFrame(np.random.randn(3,4), columns=['a','b','c','d'])
df2 = pd.DataFrame(np.random.randn(2,3), columns=['b','d','a'])

print pd.concat([df1,df2])

# 不保留连接轴的索引，创建新索引
print pd.concat([df1,df2], ignore_index=True)

print pd.concat([df1,df2], ignore_index=False)

（4）合并重叠数据

# coding=utf-8

import pandas as pd
import numpy as np

a = pd.Series([np.nan, 2.5, np.nan, 3.5, 4.5, np.nan], index=['f','e','d','c','b','a'])

b = pd.Series(np.arange(len(a), dtype=np.float64), index=['f','e','d','c','b','a'])

b[-1] = np.nan

print np.where(pd.isnull(a), b, a)

print b[:-2].combine_first(a[2:])

'''
[ 0.   2.5  2.   3.5  4.5  nan]
a    NaN
b    4.5
c    3.0
d    2.0
e    1.0
f    0.0
dtype: float64
'''

# 对于DataFrame， combine_first 自然也是会在列上做同样的事情，我们可以把这个动作当成是参数对象数据对调用对象的数据打补丁

df1 = pd.DataFrame({
'a':[1.,np.nan, 5., np.nan],
'b':[np.nan, 2., np.nan, 6.],
'c':range(2,18,4)
})

df2 = pd.DataFrame({
'a':[5., 4., np.nan, 3., 7.],
'b':[np.nan, 3., 4., 6., 8.]
})

print df1.combine_first(df2)

重塑和轴向旋转

（1）重塑层次化索引

# coding=utf-8

import pandas as pd
import numpy as np

"""
层次化索引为DataFrame 数据的重排任务提供了一种具有良好一致性的方式
（1） stack 将数据的列“旋转”为行
（2） unstack 将数据的行“旋转”为列
"""

data = pd.DataFrame(np.arange(6).reshape((2,3)),
index=pd.Index(['Ohio', 'Colorado'], name='state'),
columns=pd.Index(['one','two','three'], name='number')
)
print data

# 我们使用stack方法将DataFrame的列转换成行，得到一个Series

result = data.stack()

print result

# 同样使用unstack也可以将一个层次化索引的Series 转化得到一个DataFrame

print result.unstack()

'''
number    one  two  three
state
Ohio        0    1      2
Colorado    3    4      5

[2 rows x 3 columns]
state     number
Ohio      one       0
two       1
three     2
Colorado  one       3
two       4
three     5
dtype: int64
number    one  two  three
state
Ohio        0    1      2
Colorado    3    4      5

[2 rows x 3 columns]
'''

# unstack操作的是最内层的（stack也是）
# 当我们传入的分层级别的编号或名称，同样可以对其他级别进行unstack 操作

print result.unstack(0) == result.unstack('state')
'''
state   Ohio Colorado
number
one     True     True
two     True     True
three   True     True

[3 rows x 2 columns]
'''

当不是所有级别值都能在各分组中找到，unstack 操作可能会引入缺失数据
stack 默认会滤除缺失数据，因此该运算是可逆的

# coding=utf-8
import pandas as pd
import numpy as np

s1 = pd.Series([0, 1, 2, 3], index=['a', 'b', 'c', 'd'])
s2 = pd.Series([4, 5, 6], index=['c', 'd', 'e'])

data2 = pd.concat([s1, s2], keys=['one','two'])

print data2

# 连接 -> 层次化索引的Series -> DataFrame(行：每个Series；列：Series的索引）
print data2.unstack()

'''
one  a    0
b    1
c    2
d    3
two  c    4
d    5
e    6
dtype: int64
a   b  c  d   e
one   0   1  2  3 NaN
two NaN NaN  4  5   6

[2 rows x 5 columns]
'''

print data2.unstack().stack()
print data2.unstack().stack(dropna=False)
'''
one  a    0
b    1
c    2
d    3
two  c    4
d    5
e    6
dtype: float64
one  a     0
b     1
c     2
d     3
e   NaN
two  a   NaN
b   NaN
c     4
d     5
e     6
dtype: float64
'''

转转轴级别最低

# 对DataFrame进行操作，旋转轴 的级别将会成为结果中最低级别

df = pd.DataFrame(
{'left': result, 'right': result + 5},
columns=pd.Index(['left', 'right'], name='side')
)

print df.unstack('state')

print df.unstack('state').stack('side')

（2）将“长格式”旋转为“宽格式”

将原始数据转换成我们需要的数据格式



关系型数据库中存储结构

# coding=utf-8
import pandas as pd
import numpy as np

data = pd.read_csv("/home/peerslee/py/pydata/pydata-book-master/ch07/macrodata.csv")
frame01 = pd.DataFrame(data, columns=['year', 'realgdp', 'infl', 'unemp'])

path01 = '/home/peerslee/py/pydata/pydata-book-master/ch07/macrodata01.csv'
frame01.to_csv(path01, index=False, header=False) # 将去除index 和 header

names02 = ['year', 'realgdp', 'infl', 'unemp']
frame02 = pd.read_table(path01, sep=',', names=names02, index_col='year') # 将'year'列设置为行索引

path02 = '/home/peerslee/py/pydata/pydata-book-master/ch07/macrodata02.csv'
frame02.stack().to_csv(path02) # 轴向旋转之后写入文件中，year列会自动向后填充

names03 = ['date','item','value']
frame03 = pd.read_table(path02, sep=',', names=names03,)
print frame03
'''
date     item     value
0   1959  realgdp  2710.349
1   1959     infl     0.000
2   1959    unemp     5.800
3   1959  realgdp  2778.801
4   1959     infl     2.340
5   1959    unemp     5.100
6   1959  realgdp  2775.488
7   1959     infl     2.740
8   1959    unemp     5.300
9   1959  realgdp  2785.204
10  1959     infl     0.270
'''

result_path = '/home/peerslee/py/pydata/pydata-book-master/ch07/result_data.csv'
frame03.to_csv(result_path) # 将数据保存起来

关系型数据库（例如mysql）中就是这样存储数据的

但是我在进行数据pivot 的时候，出现了错误：
raise ValueError('Index contains duplicate entries, '
ValueError: Index contains duplicate entries, cannot reshape

所以我们要将月份也加进来，但是我不知道，怎么搞，先放在这，以后再说（如果朋友哪位朋友有思路欢迎交流学习）

我们在操作上面的数据的时候没有那么轻松，我们常常要将数据转换为DataFrame 的格式，不同的item 值分别形成一列

# coding=utf-8
import pandas as pd
import numpy as np

"""
DataFrame 中的pivot方法可以将“长格式”旋转为“宽格式”
"""
# 因为没有数据所以我们只能自己写点
ldata = pd.DataFrame({'date':['03-31','03-31','03-31','06-30','06-30','06-30'],
'item':['real','infl','unemp','real','infl','unemp'],
'value':['2710.','000.','5.8','2778.','2.34','5.1']
})

print ldata

# 将date作为行索引的名字，item为列索引的名字，将value填充进去
pivoted = ldata.pivot('date','item','value')
print pivoted
'''
item   infl   real unemp
date
03-31  000.  2710.   5.8
06-30  2.34  2778.   5.1

[2 rows x 3 columns]
'''

# 将需要重塑的列扩为两列
ldata['value2'] = np.random.randn(len(ldata))
print ldata

# 忽略pivot的最后一个参数,会得到一个带有层次化的列
pivoted = ldata.pivot('date','item')
print pivoted
'''
value                 value2
item   infl   real unemp      infl      real     unemp
date
03-31  000.  2710.   5.8  1.059406  0.437246  0.106987
06-30  2.34  2778.   5.1 -1.087665 -0.811100 -0.579266

[2 rows x 6 columns]
'''

print pivoted['value'][:5]
'''
item   infl   real unemp
date
03-31  000.  2710.   5.8
06-30  2.34  2778.   5.1

[2 rows x 3 columns]
'''

# 这个操作是完整的pivot 操作
unstacked = ldata.set_index(['date','item']).unstack('item')

print unstacked
'''
value                 value2
item   infl   real unemp      infl      real     unemp
date
03-31  000.  2710.   5.8 -1.018416 -1.476397  1.579151
06-30  2.34  2778.   5.1  0.863437  1.606538 -1.147549

[2 rows x 6 columns]
'''

数据转换：

1.移除重复数据

# coding=utf-8

import pandas as pd
import numpy as np

data = pd.DataFrame({'k1': ['one'] * 3 + ['two'] * 4,
'k2': [1, 1, 2, 3, 3, 4, 4]})

print data

'''
k1  k2
0  one   1
1  one   1
2  one   2
3  two   3
4  two   3
5  two   4
6  two   4

[7 rows x 2 columns]
'''

"""
DataFrame 的duplicated 方法返回一个布尔型Series，表示各行是否是重复行
"""

print data.duplicated()

'''
0    False
1     True
2    False
3    False
4     True
5    False
6     True
dtype: bool
'''

"""
DataFrame 的drop_duplicated 方法用于移除了重复行的DataFrame
"""

print data.drop_duplicates()

'''
k1  k2
0  one   1
2  one   2
3  two   3
5  two   4

[4 rows x 2 columns]
'''

"""
具体根据 某一列 来判断是否有重复值
"""

data['v1'] = range(7)

print data.drop_duplicates(['k1'])

'''
k1  k2  v1
0  one   1   0
3  two   3   3

[2 rows x 3 columns]
'''

"""
默认保留的是第1个值，如果想保留最后一个值则传入 take_last=True
"""

print data.drop_duplicates(['k1','k2'], take_last=True)

'''
k1  k2  v1
1  one   1   1
2  one   2   2
4  two   3   4
6  two   4   6

[4 rows x 3 columns]
'''

2.利用函数或映射进行数据转换

# coding=utf-8

import pandas as pd
import numpy as np

# 一张食物和重量的表格
data = pd.DataFrame({'food':['bacon','pulled pork','bacon','Pastrami','corned beef',
'Bacon','pastrami','honey ham','nova lox'],
'ounces':[4,3,12,6,7.5,8,3,5,6]})

# 食物和动物的表格映射
meat_to_ainmal = {
'bacon':'pig',
'pulled pork':'pig',
'pastrami':'cow',
'corned beef':'pig',
'nova lox':'salmon'
}

"""
Series 的map 方法接受一个函数或含有映射关系的字典型对象
"""

# 这里我们先将表1 中的所有食物转换为小写，再做一个map

data['animal'] = data['food'].map(str.lower).map(meat_to_ainmal)

print data

'''
food  ounces  animal
0        bacon     4.0     pig
1  pulled pork     3.0     pig
2        bacon    12.0     pig
3     Pastrami     6.0     cow
4  corned beef     7.5     pig
5        Bacon     8.0     pig
6     pastrami     3.0     cow
7    honey ham     5.0     NaN
8     nova lox     6.0  salmon

[9 rows x 3 columns]
'''
"""
我们可以传入一个能够完成全部这些工作的函数
"""

3.替换值

# coding=utf-8

import pandas as pd
import numpy as np

data = pd.Series([1., -999., 2.,  -999., -1000., 3.])

"""
我们要使用pandas 将-999 这样的数据替换成 NA 值
"""

data.replace(-999,np.nan)

"""
如果希望一次性替换多个值，可以传入一个由待替换值组成的列表以及一个替换值
"""
data.replace([-999,-1000],np.nan)

"""
如果希望对不同的值进行不同的替换，则传入一个由替换关系组成的列表即可
"""

data.replace([-999,-1000],[np.nan,0])

"""
传入的参数也可以是字典
"""

data.replace({-999:np.nan, -1000:0})

4.重命名轴索引

# coding=utf-8
import pandas as pd
import numpy as np

"""
轴标签可以通过函数或映射进行转换，从而得到一个新对象，轴还可以被就地修改
"""

data = pd.DataFrame(np.arange(12).reshape((3,4)),
index=['Ohio','Colorado','New York',],
columns=['one','two','three','four'])

"""
轴标签也有一个map 方法
"""
print data.index.map(str.upper)

"""
直接赋值给index
"""
data.index = data.index.map(str.upper)

print data

"""
创建数据集的转换版本，而不是修改原始数据
"""

data.rename(index=str.title, columns=str.upper)

"""
rename 可以结合字典型对象实现对部分轴标签的更新
"""

data.rename(index={'OHIO':'INDIANA'},
columns={'three','peekaboo'})

"""
rename 也可以就地对DataFrame 进行修改，inplace=True
"""

_=data.rename(index={'OHIO':'INDIANA'},inplace=True)