kaggle预测房价

kaggle房价预测比赛官方地址：https://www.kaggle.com/c/house-prices-advanced-regression-techniqueskaggle数据集描述：https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data

Step 1：引入相关的包

# coding:utf-8
# 注意读取文件时，Windows系统的\\和Linux系统的/的区别

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestRegressor

Step 2：读取数据 

文件的组织形式是house price文件夹下面放house_price.py和input文件夹。input文件夹下面放的是从https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data下载的train.csv test.csv sample_submission.csv 和 data_description.txt 四个文件。

# 将csv数据转换为DataFrame数据，方便用pandas进行数据预处理
# 注意将print的注释打开，可以查看输出结果

#不要让pandas自己给数据加编号，这样ID就成为index了
train_df = pd.read_csv(".\\input\\train.csv",index_col = 0)
test_df = pd.read_csv('.\\input\\test.csv',index_col = 0)
# print train_df.shape
# print test_df.shape
# print train_df.head()  # 默认展示前五行 这里是5行,80列
# print test_df.head()   # 这里是5行,79列

Step 3：合并数据 ：特征工程的工作！！！

这么做主要是为了用DF进行数据预处理的时候更加方便。等所有的需要的预处理进行完之后，我们再把他们分隔开。实际项目中，不会这样做。首先，SalePrice作为我们的训练目标，只会出现在训练集中，不会在测试集中。所以，我们先把SalePrice这一列给拿出来，不让它碍事儿。

# 看SalePrice的形状和用log1p处理后的形状

%matplotlib inline
prices = pd.DataFrame({'price':train_df['SalePrice'],'log(price+1)':np.log1p(train_df['SalePrice'])})
ps = prices.hist()
# plt.plot()
# plt.show()

# log1p即log(1+x)，可以让label平滑化，将数据变为正态分布，目的在于使数据的呈现方式接近我们所希望的前提假设，从而进行更好的统计推断
y_train = np.log1p(train_df.pop('SalePrice'))  #提出和test数据不一致的price，马上进行train和test的合并
all_df = pd.concat((train_df,test_df),axis = 0) #合并
# print all_df.shape #查看all_df  (2919,79)
# print y_train.head()  #查看处理后的标记预测值

Step 4：变量转化：特征工程和数据清洗的工作！！！

正确化变量属性：MSSubClass 的值其实应该是一个category（等级的划分），虽然是数字，但是代表多类别，Pandas是不会懂这些。使用DF的时候，这类数字符号会被默认记成数字。这种东西就很有误导性，我们需要把它变回成string

print all_df['MSSubClass'].dtypes   #dtype('int64')
all_df['MSSubClass'] = all_df['MSSubClass'].astype(str)  #转为string，便于查看他的分布情况
print all_df['MSSubClass'].dtypes
print all_df['MSSubClass'].value_counts()

把category的变量转变成numerical表达形式：当我们用numerical来表达categorical的时候，要注意，数字本身有大小的含义，所以乱用数字会给之后的模型学习带来麻烦。于是我们可以用One-Hot的方法来表达category。pandas自带的get_dummies方法，可以帮你一键做到One-Hot。

#不能被计算机理解的变量（字符串，离散型变量等）

print pd.get_dummies(all_df['MSSubClass'],prefix = 'MSSubClass'#处理离散型变量的方法get_dummies,即就是one-hot).head()
all_dummy_df = pd.get_dummies(all_df)  #pandas自动选择那些事离散型变量，省去了我们做选择
print all_dummy_df.head()

清洗第二步：处理numerical变量：比如，有一些数据是缺失的

print all_dummy_df.isnull().sum().sort_values(ascending = False).head(11)  #查看缺失情况，按照缺失情况排序
# 注意：处理缺失情况时要看数据描述，确实值得处理方式工具意义和缺失情况有很大不同，有时确实本身就有意义，我们要把他当

#做一个类型，其他时候要将其补上或者删除这个特征

#我们这里用mean填充
mean_cols = all_dummy_df.mean()
print mean_cols.head(10)
all_dummy_df = all_dummy_df.fillna(mean_cols)  #fillna填充
print all_dummy_df.isnull().sum().sum()  #输出0

标准化numerical数据：这一步并不是必要，但是得看你想要用的分类器是什么。一般来说，regression的分类器都需要这一步，最好是把源数据给放在一个标准分布内，不要让数据间的差距太大。我们不需要把One-Hot的那些0/1数据给标准化，因为只有0和1，我们的目标应该是那些本来就是numerical的数据型的特征。

numeric_cols = all_df.columns[all_df.dtypes != 'object']  #查看那些是numerical数据，本来就是数字化的数据
print numeric_cols

#标准化numerical数据，让数据更加平滑，更加便于计算：如z-score标准化：（x-x’）/s 【x：原数据；x'：平均数；s：标准差】
numeric_col_means = all_dummy_df.loc[:,numeric_cols].mean()  #均值
numeric_col_std = all_dummy_df.loc[:,numeric_cols].std()  #标准差
all_dummy_df.loc[:,numeric_cols] = (all_dummy_df.loc[:,numeric_cols] - numeric_col_means) / numeric_col_std

Step 5-1: 建立模型【房价预测/Ridge/RandomForest/cross_validation】

# 把数据处理之后，分回训练集和测试集（起初在数据处理时将train和test数据结合了）
dummy_train_df = all_dummy_df.loc[train_df.index]
dummy_test_df = all_dummy_df.loc[test_df.index]
print dummy_train_df.shape,dummy_test_df.shape    #输出（（1460,303），（1459,303））

# 将DF数据转换成Numpy Array的形式，更好地配合sklearn
X_train = dummy_train_df.values
X_test = dummy_test_df.values

Ridge Regression（回归模型的一种：对于多因子的数据集，可以直接把所有的特征都放进去，不用考虑特征提取）

from sklearn.linear_model import Ridge

from sklearn.model_selection import cross_val_score  #交叉验证来测试模型

#不是很必要，知识吧DataFrame转换成Numpy Array格式数据

X_train = dummy_train_df.values

X_test =  dummy_test_df.values

#用Sklearn自带的cross_calidation来测试模型

alphas = np.logspace(-3,2,50) #创建等比梳理与，如：10^-3至10^2其中的50个数test_scores = []  #交叉验证的得分，最后找到最好的参数for alpha in alphas:clf = Ridge(alpha)test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))test_scores.append(np.mean(test_score))

plt.plot(alphas,test_scores)  #可视化参数与分数

plt.title('Alpha vs CV Error')

plt.show()

存下所有的cv值，看看那个alpha值更好【调参数】大概alpha=10~20的时候，可以把score达到0.135左右。

Random Forest

from sklearn.ensemble import RandomForestRegressorRF

max_features = [.1,.3,.5,.7,.9,.99]test_scores = []for max_feat in max_features:clf = RandomForestRegressor(n_estimators = 200,max_features = max_feat)test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 5,scoring = 'neg_mean_squared_error'))test_scores.append(np.mean(test_score))plt.plot(max_features,test_scores)plt.title('Max Features vs CV Error')plt.show()

max_features=0.3时，RF达到了最优0.137

Step 5-2: 建立模型 【进阶版/bagging/boosting/AdaBoost/XGBoost】

从模型的角度考虑，用了bagging、boosting（AdaBoost）、XGBoost三个模型（模型框架）。 把数据集分回 训练/测试集

dummy_train_df = all_dummy_df.loc[train_df.index]dummy_test_df = all_dummy_df.loc[test_df.index]print dummy_train_df.shape,dummy_test_df.shape# 将DF数据转换成Numpy Array的形式，更好地配合sklearnX_train = dummy_train_df.valuesX_test = dummy_test_df.values

1、bagging： 

单个分类器的效果真的是很有限。我们会倾向于把N多的分类器合在一起，做一个“综合分类器”以达到最好的效果。我们从刚刚的试验中得知，Ridge(alpha=15)给了我们最好的结果ridge = Ridge(alpha=15)# bagging 把很多小的分类器放在一起，每个train随机的一部分数据，然后把它们的最终结果综合起来（多数投票）# bagging 算是一种算法框架params = [1, 10, 15, 20, 25, 30, 40] # 多少个弱分类器test_scores = []for param in params:clf = BaggingRegressor(n_estimators=param，base_estimator = ridge) # #base_estimator = ridge是弱分类器0.132（params=25时）#clf = BaggingRegressor(n_estimators = param)#用Bagging自带的DecisionTree，最好0.140test_score = np.sqrt(-cross_val_score(clf, X_train, y_train, cv=10, scoring='neg_mean_squared_error'))test_scores.append(np.mean(test_score))plt.plot(params, test_scores)plt.title('n_estimators vs CV Error')plt.show()br = BaggingRegressor(base_estimator=ridge, n_estimators=25)br.fit(X_train, y_train)y_final = np.expm1(br.predict(X_test))

2、boosting 

Boosting比Bagging理论上更高级点，它也是揽来一把的分类器。但是把他们线性排列。下一个分类器把上一个分类器分类得不好的地方加上更高的权重，这样下一个分类器就能在这个部分学得更加“深刻”。from sklearn.ensemble import AdaBoostRegressorms = [10,15,20,25,30,35,40,45,50]test_scores = []for param in params:clf = AdaBoostRegressor(base_estimator = ridge,n_estimators = param) #ms=25时，0.132，但是不稳定，需要更多的参数或者更多小分类器test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))test_scores.append(np.mean(test_score))plt.plot(params,test_scores)plt.title('n_estimators vs CV Error')plt.show()

3、XGBoost （kaggle神器）

这依旧是一款Boosting框架的模型，但是却做了很多的改进。 

from xgboost import XGBRegressorparams = [1,2,3,4,5,6]test_scores = []for param in params:clf = XGBRegressor(max_depth = param)  #深度params=5时，错误率达到0.127test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))test_scores.append(np.mean(test_score))plt.plot(params,test_scores)plt.title('max_depth vs CV Error')plt.show()xgb = XGBRegressor(max_depth = 5)xgb.fit(X_train, y_train)y_final = np.expm1(xgb.predict(X_test))

Step 6: Ensemble 这里我们用一个Stacking的思维来汲取两种或者多种模型的优点 ;首先，我们把最好的parameter拿出来，做成我们最终的model;

ridge = Ridge(alpha = 15)rf = RandomForestRegressor(n_estimators = 500,max_features = .3)ridge.fit(X_train,y_train)rf.fit(X_train,y_train)

#最前面个label做了一个log(1+x),这里需要把predit的值给exp回去，并且戒掉那个‘1’

y_ridge = np.expm1(ridge.predict(X_test))y_rf = np.expm1(rf.predict(X_test))#把所有的model的预测结果作为新的输入，最简单的就是不下直接【平均化】y_final = (y_ridge + y_rf) / 2

Step 7: 提交结果 

注意提交的格式！包括大小写、索引、列头等小细节。

submission_df = pd.DataFrame(data = {'Id':test_df.index,'SalePrice':y_final})print submission_df.head(10)submission_df.to_csv('.\\input\\submission.csv',columns = ['Id','SalePrice'],index = False)

Step5-1版完整练习代码：

# coding:utf-8# 注意Windows系统的\\和Linux系统的/的区别import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn.linear_model import Ridgefrom sklearn.model_selection import cross_val_scorefrom sklearn.ensemble import RandomForestRegressor# 文件的组织形式是house price文件夹下面放house_price.py和input文件夹# input文件夹下面放的是从https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data下载的train.csv  test.csv  sample_submission.csv 和 data_description.txt 四个文件# step1 检查源数据集，读入数据，将csv数据转换为DataFrame数据train_df = pd.read_csv(".\\input\\train.csv",index_col = 0)test_df = pd.read_csv('.\\input\\test.csv',index_col = 0)# print train_df.shape# print test_df.shape# print train_df.head()  # 默认展示前五行 这里是5行,80列# print test_df.head()   # 这里是5行,79列# step2 合并数据，进行数据预处理prices = pd.DataFrame({'price':train_df['SalePrice'],'log(price+1)':np.log1p(train_df['SalePrice'])})# ps = prices.hist()# plt.plot()# plt.show()y_train = np.log1p(train_df.pop('SalePrice'))all_df = pd.concat((train_df,test_df),axis = 0)# print all_df.shape# print y_train.head()# step3 变量转化print all_df['MSSubClass'].dtypesall_df['MSSubClass'] = all_df['MSSubClass'].astype(str)print all_df['MSSubClass'].dtypesprint all_df['MSSubClass'].value_counts()# 把category的变量转变成numerical表达形式# get_dummies方法可以帮你一键one-hotprint pd.get_dummies(all_df['MSSubClass'],prefix = 'MSSubClass').head()all_dummy_df = pd.get_dummies(all_df)print all_dummy_df.head()# 处理好numerical变量print all_dummy_df.isnull().sum().sort_values(ascending = False).head(11)# 我们这里用mean填充mean_cols = all_dummy_df.mean()print mean_cols.head(10)all_dummy_df = all_dummy_df.fillna(mean_cols)print all_dummy_df.isnull().sum().sum()# 标准化numerical数据numeric_cols = all_df.columns[all_df.dtypes != 'object']print numeric_colsnumeric_col_means = all_dummy_df.loc[:,numeric_cols].mean()numeric_col_std = all_dummy_df.loc[:,numeric_cols].std()all_dummy_df.loc[:,numeric_cols] = (all_dummy_df.loc[:,numeric_cols] - numeric_col_means) / numeric_col_std# step4 建立模型# 把数据处理之后，送回训练集和测试集dummy_train_df = all_dummy_df.loc[train_df.index]dummy_test_df = all_dummy_df.loc[test_df.index]print dummy_train_df.shape,dummy_test_df.shape# 将DF数据转换成Numpy Array的形式，更好地配合sklearnX_train = dummy_train_df.valuesX_test = dummy_test_df.values# Ridge Regression# alphas = np.logspace(-3,2,50)# test_scores = []# for alpha in alphas:#   clf = Ridge(alpha)#   test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))#   test_scores.append(np.mean(test_score))# plt.plot(alphas,test_scores)# plt.title('Alpha vs CV Error')# plt.show()# random forest# max_features = [.1,.3,.5,.7,.9,.99]# test_scores = []# for max_feat in max_features:#   clf = RandomForestRegressor(n_estimators = 200,max_features = max_feat)#   test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 5,scoring = 'neg_mean_squared_error'))#   test_scores.append(np.mean(test_score))# plt.plot(max_features,test_scores)# plt.title('Max Features vs CV Error')# plt.show()# Step 5: ensemble# 用stacking的思维来汲取两种或者多种模型的优点ridge = Ridge(alpha = 15)rf = RandomForestRegressor(n_estimators = 500,max_features = .3)ridge.fit(X_train,y_train)rf.fit(X_train,y_train)y_ridge = np.expm1(ridge.predict(X_test))y_rf = np.expm1(rf.predict(X_test))y_final = (y_ridge + y_rf) / 2# Step 6: 提交结果submission_df = pd.DataFrame(data = {'Id':test_df.index,'SalePrice':y_final})print submission_df.head(10)submission_df.to_csv('.\\input\\submission.csv',columns = ['Id','SalePrice'],index = 

Step5-2版完整练习代码：

# coding:utf-8# 注意Windows系统的\\和Linux系统的/的区别import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom sklearn.linear_model import Ridgefrom sklearn.model_selection import cross_val_scorefrom sklearn.ensemble import RandomForestRegressorfrom sklearn.ensemble import BaggingRegressorfrom sklearn.ensemble import AdaBoostRegressorfrom xgboost import XGBRegressor# 文件的组织形式是house price文件夹下面放house_price.py和input文件夹# input文件夹下面放的是从https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data下载的train.csv  test.csv  sample_submission.csv 和 data_description.txt 四个文件# step1 检查源数据集，读入数据，将csv数据转换为DataFrame数据train_df = pd.read_csv("./input/train.csv",index_col = 0)test_df = pd.read_csv('./input/test.csv',index_col = 0)# print train_df.shape# print test_df.shape# print train_df.head()  # 默认展示前五行 这里是5行,80列# print test_df.head()   # 这里是5行,79列# step2 合并数据，进行数据预处理prices = pd.DataFrame({'price':train_df['SalePrice'],'log(price+1)':np.log1p(train_df['SalePrice'])})# ps = prices.hist()# plt.plot()# plt.show()y_train = np.log1p(train_df.pop('SalePrice'))all_df = pd.concat((train_df,test_df),axis = 0)# print all_df.shape# print y_train.head()# step3 变量转化print all_df['MSSubClass'].dtypesall_df['MSSubClass'] = all_df['MSSubClass'].astype(str)print all_df['MSSubClass'].dtypesprint all_df['MSSubClass'].value_counts()# 把category的变量转变成numerical表达形式# get_dummies方法可以帮你一键one-hotprint pd.get_dummies(all_df['MSSubClass'],prefix = 'MSSubClass').head()all_dummy_df = pd.get_dummies(all_df)print all_dummy_df.head()# 处理好numerical变量print all_dummy_df.isnull().sum().sort_values(ascending = False).head(11)# 我们这里用mean填充mean_cols = all_dummy_df.mean()print mean_cols.head(10)all_dummy_df = all_dummy_df.fillna(mean_cols)print all_dummy_df.isnull().sum().sum()# 标准化numerical数据numeric_cols = all_df.columns[all_df.dtypes != 'object']print numeric_colsnumeric_col_means = all_dummy_df.loc[:,numeric_cols].mean()numeric_col_std = all_dummy_df.loc[:,numeric_cols].std()all_dummy_df.loc[:,numeric_cols] = (all_dummy_df.loc[:,numeric_cols] - numeric_col_means) / numeric_col_std# step4 建立模型# 把数据处理之后，送回训练集和测试集dummy_train_df = all_dummy_df.loc[train_df.index]dummy_test_df = all_dummy_df.loc[test_df.index]print dummy_train_df.shape,dummy_test_df.shape# 将DF数据转换成Numpy Array的形式，更好地配合sklearnX_train = dummy_train_df.valuesX_test = dummy_test_df.values# Ridge Regression# alphas = np.logspace(-3,2,50)# test_scores = []# for alpha in alphas:#   clf = Ridge(alpha)#   test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))#   test_scores.append(np.mean(test_score))# plt.plot(alphas,test_scores)# plt.title('Alpha vs CV Error')# plt.show()# random forest# max_features = [.1,.3,.5,.7,.9,.99]# test_scores = []# for max_feat in max_features:#   clf = RandomForestRegressor(n_estimators = 200,max_features = max_feat)#   test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 5,scoring = 'neg_mean_squared_error'))#   test_scores.append(np.mean(test_score))# plt.plot(max_features,test_scores)# plt.title('Max Features vs CV Error')# plt.show()# ensemble# 用stacking的思维来汲取两种或者多种模型的优点# ridge = Ridge(alpha = 15)# rf = RandomForestRegressor(n_estimators = 500,max_features = .3)# ridge.fit(X_train,y_train)# rf.fit(X_train,y_train)# y_ridge = np.expm1(ridge.predict(X_test))# y_rf = np.expm1(rf.predict(X_test))# y_final = (y_ridge + y_rf) / 2# 做一点高级的ensembleridge = Ridge(alpha = 15)# bagging 把很多小的分类器放在一起，每个train随机的一部分数据，然后把它们的最终结果综合起来（多数投票）# bagging 算是一种算法框架# params = [1,10,15,20,25,30,40]# test_scores = []# for param in params:#   clf = BaggingRegressor(base_estimator = ridge,n_estimators = param)#   test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))#   test_scores.append(np.mean(test_score))# plt.plot(params,test_scores)# plt.title('n_estimators vs CV Error')# plt.show()# br = BaggingRegressor(base_estimator = ridge,n_estimators = 25)# br.fit(X_train,y_train)# y_final = np.expm1(br.predict(X_test))# boosting 比bagging更高级，它是弄来一把分类器，把它们线性排列，下一个分类器把上一个分类器分类不好的地方加上更高的权重，这样，下一个分类器在这部分就能学习得更深刻# params = [10,15,20,25,30,35,40,45,50]# test_scores = []# for param in params:#   clf = AdaBoostRegressor(base_estimator = ridge,n_estimators = param)#   test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))#   test_scores.append(np.mean(test_score))# plt.plot(params,test_scores)# plt.title('n_estimators vs CV Error')# plt.show()# xgboostparams = [1,2,3,4,5,6]test_scores = []for param in params:clf = XGBRegressor(max_depth = param)test_score = np.sqrt(-cross_val_score(clf,X_train,y_train,cv = 10,scoring = 'neg_mean_squared_error'))test_scores.append(np.mean(test_score))plt.plot(params,test_scores)plt.title('max_depth vs CV Error')plt.show()xgb = XGBRegressor(max_depth = 5)xgb.fit(X_train, y_train)y_final = np.expm1(xgb.predict(X_test))# 提交结果submission_df = pd.DataFrame(data = {'Id':test_df.index,'SalePrice':y_final})print submission_df.head(10)submission_df.to_csv('./input/submission_xgboosting.csv',columns = ['Id','SalePrice'

总结：

并不是所有的数据源都是整齐划一的x=[var1,var2,var3...]方法：【非标准-现实生活数据】-->降维、取特征、数字化表达（特征工程）-->【高维数据】【文本数据】-->单词出现次数、单词出现频率、语义网络等（特征工程）-->【数据】【图片数据】-->RGB点阵-->【数组】【视频数据】-->分为【音轨】and【视频轨】-->【音轨：声波/语音识别】【视频轨：一维图片/图片识别】可参考：https://www.cnblogs.com/irenelin/p/7400388.html http://blog.csdn.net/qilixuening/article/details/75153131 http://blog.csdn.net/qilixuening/article/details/75151026http://blog.csdn.net/chris_lee_hehe/article/details/78700140