HBase初探
2015-08-21 18:17
232 查看
public class StockEntity { public string Name { get; set; } public double TodayOpeningPrice { get; set; } public double YesterdayClosingPrice { get; set; } public double CurrentPrice { get; set; } public double TodayMaxPrice { get; set; } public double TodayMinPrice { get; set; } public double BidPriceBuy { get; set; } public double BidPriceSell { get; set; } public int FixtureNumber { get; set; } public double FixtureAmount { get; set; } public int Buy1Number { get; set; } public double Buy1Price { get; set; } public int Buy2Number { get; set; } public double Buy2Price { get; set; } public int Buy3Number { get; set; } public double Buy3Price { get; set; } public int Buy4Number { get; set; } public double Buy4Price { get; set; } public int Buy5Number { get; set; } public double Buy5Price { get; set; } public int Sell1Number { get; set; } public double Sell1Price { get; set; } public int Sell2Number { get; set; } public double Sell2Price { get; set; } public int Sell3Number { get; set; } public double Sell3Price { get; set; } public int Sell4Number { get; set; } public double Sell4Price { get; set; } public int Sell5Number { get; set; } public double Sell5Price { get; set; } public DateTime TransactionTime { get; set; } }
View Code
数据拉下来之后,新开一个线程,让它去写到hbase中。
ThreadPool.QueueUserWorkItem(new WaitCallback(SaveStockDataToHbase), se);
具体干活代码如下:
private void SaveStockDataToHbase(object state) { StockEntity se = state as StockEntity; // Insert data into the HBase table. string rowKey = Guid.NewGuid().ToString(); CellSet cellSet = new CellSet(); CellSet.Row cellSetRow = new CellSet.Row { key = Encoding.UTF8.GetBytes(rowKey) }; cellSet.rows.Add(cellSetRow); Type t = typeof(StockEntity); foreach (string colname in stockEntityColumns) { var pi = t.GetProperty(colname); object val = pi.GetValue(se); Cell value = new Cell { column = Encoding.UTF8.GetBytes("charju:" + colname), data = Encoding.UTF8.GetBytes(Convert.ToString(val)) }; cellSetRow.values.Add(value); } try { hbaseClient.StoreCells(hbaseStockTableName, cellSet); } catch (Exception ex) { Console.WriteLine(ex.Message); } }
6~10行,是生成一个新Row。20行,是反射实体类的每一个Property 定义,来取对应的值(否则我要写一坨重复的代码)。21行,把对应的该列数据写到这个行上。
26行,就是真正的放到hbase中。
上面20行,你可能会注意到:charju,这是我的column family的名字。回过头来,看看hbase中的表是怎么建立的
string hbaseCluster = "https://charju.azurehdinsight.net"; string hadoopUsername = "<your name>"; string hadoopPassword = "<your password>"; string hbaseStockTableName = "StockInformation"; HBaseClient hbaseClient; public void CreateHbaseTable() { // Create a new HBase table. - StockInformation TableSchema stockTableSchema = new TableSchema(); stockTableSchema.name = hbaseStockTableName; stockTableSchema.columns.Add(new ColumnSchema() { name = "charju" }); hbaseClient.CreateTable(stockTableSchema); }
而hbaseClient的实例化,是在这里:
ClusterCredentials creds = new ClusterCredentials(new Uri(hbaseCluster), hadoopUsername, hadoopPassword); hbaseClient = new HBaseClient(creds);
数据写入后,我们可以有几个方式来。一是在hbase中配置一下,允许RDP,然后remote上去跑hbase shell命令,可惜我虚机里面RDP总失败,不知道为啥。第二种方式,就是用HIVE来查。
连接到hbase的网站后,在hive editor那个界面中,先创建对应的表
CREATE EXTERNAL TABLE StockInformation(rowkey STRING, TodayOpeningPrice STRING, YesterdayClosingPrice STRING, CurrentPrice STRING, TodayMaxPrice STRING, TodayMinPrice STRING, BidPriceBuy STRING, BidPriceSell STRING, FixtureNumber STRING, FixtureAmount STRING, Buy1Number STRING, Buy1Price STRING, Buy2Number STRING, Buy2Price STRING, Buy3Number STRING, Buy3Price STRING, Buy4Number STRING, Buy4Price STRING, Buy5Number STRING, Buy5Price STRING, Sell1Number STRING, Sell1Price STRING, Sell2Number STRING, Sell2Price STRING, Sell3Number STRING, Sell3Price STRING, Sell4Number STRING, Sell4Price STRING, Sell5Number STRING, Sell5Price STRING, TransactionTime STRING) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ('hbase.columns.mapping' = ':key,charju:TodayOpeningPrice ,charju:YesterdayClosingPrice ,charju:CurrentPrice ,charju:TodayMaxPrice ,charju:TodayMinPrice ,charju:BidPriceBuy ,charju:BidPriceSell ,charju:FixtureNumber ,charju:FixtureAmount ,charju:Buy1Number ,charju:Buy1Price ,charju:Buy2Number ,charju:Buy2Price ,charju:Buy3Number ,charju:Buy3Price ,charju:Buy4Number ,charju:Buy4Price ,charju:Buy5Number ,charju:Buy5Price ,charju:Sell1Number ,charju:Sell1Price ,charju:Sell2Number ,charju:Sell2Price ,charju:Sell3Number ,charju:Sell3Price ,charju:Sell4Number ,charju:Sell4Price ,charju:Sell5Number ,charju:Sell5Price ,charju:TransactionTime') TBLPROPERTIES ('hbase.table.name' = 'StockInformation');
创建成功后,然后就可以跑SQL了,比如说:
select * from StockInformation where buy1number=9800 order by transactiontime
今天小浪的最大一笔买入。当然,类似于select count(0) 之类的更OK了。
有用的连接:
https://azure.microsoft.com/en-us/documentation/articles/hdinsight-hbase-tutorial-get-started/
相关文章推荐
- cstring和相关类型的相互转换int,double,char*
- 第1章 unix基础知识
- 交易应用及网站驱动不兼容Windows 10的解决方案
- Delphi中ADO异步执行方式
- 读Java Concurrency in Practice. 第六章.
- 关于数据库索引
- 如何在“一周内”摸清一个行业
- Struts2学习笔记——02
- Spark修炼之道(基础篇)——Linux大数据开发基础:第一节、Linux介绍、安装及使用初步
- 创建com服务器
- 【感恩教师节】51CTO学院疯狂大让利活动
- input验证是否为有效手机号码,亲测可用
- 黑马程序员-----GUI
- C#实现由四周向中心缩小的窗体退出特效
- Spring MVC 中的基于注解的 Controller
- iOS 去掉导航栏的边界灰线;去掉搜索框的边界灰线和其中文本输入框的阴影
- iOS 获取设备类型
- assert()函数用法总结
- URAL 题目1517. Freedom of Choice(后缀数组,求公共最长串)
- 课程设计,文件加密