Hive实验-CompactIndex索引的建立与使用
2012-03-04 22:42
489 查看
一。准备数据
#! /bin/bash #generating 350M raw data. i=0 while [ $i -ne 1000000 ] do echo "$i\tA decade ago, many were predicting that Cooke, a New York City prodigy, would become a basketball shoe pitchman and would flaunt his wares and skills at All-Star weekends like the recent aerial show in Orlando, Fla. There was a time, however fleeting, when he was more heralded, or perhaps merely hyped, than any other high school player in America." i=$(($i+1)) done
$ sh gen-data.sh >dual.txt
二。创建表
drop table table01; create table table01( id int, name string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'; load data local inpath '/home/allen/Desktop/dual.txt' overwrite into table table01;
hive> dfs -ls /user/hive/warehouse/table01; Found 1 items -rw-r--r-- 1 allen supergroup 356888890 2012-03-04 22:22 /user/hive/warehouse/table01/dual.txt hive>create table02 as select id ,name as text from table01;
6个Mapper，0个Reducer
hive> dfs -ls /user/hive/warehouse/table02; Found 6 items -rw-r--r-- 1 allen supergroup 67109134 2012-03-04 22:28 /user/hive/warehouse/table02/000000_0 -rw-r--r-- 1 allen supergroup 67108860 2012-03-04 22:28 /user/hive/warehouse/table02/000001_0 -rw-r--r-- 1 allen supergroup 67108860 2012-03-04 22:30 /user/hive/warehouse/table02/000002_0 -rw-r--r-- 1 allen supergroup 67108860 2012-03-04 22:30 /user/hive/warehouse/table02/000003_0 -rw-r--r-- 1 allen supergroup 67108860 2012-03-04 22:32 /user/hive/warehouse/table02/000004_0 -rw-r--r-- 1 allen supergroup 21344316 2012-03-04 22:32 /user/hive/warehouse/table02/000005_0
hive> select * from table02 where id=500000; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201203042123_0003, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203042123_0003 Kill Command = /home/allen/Hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201203042123_0003 2012-03-04 22:42:09,609 Stage-1 map = 0%, reduce = 0% 2012-03-04 22:42:21,919 Stage-1 map = 19%, reduce = 0% 2012-03-04 22:42:25,020 Stage-1 map = 33%, reduce = 0% 2012-03-04 22:42:37,602 Stage-1 map = 61%, reduce = 0% 2012-03-04 22:42:39,715 Stage-1 map = 67%, reduce = 0% 2012-03-04 22:42:46,168 Stage-1 map = 77%, reduce = 0% 2012-03-04 22:42:49,552 Stage-1 map = 79%, reduce = 0% 2012-03-04 22:42:51,812 Stage-1 map = 80%, reduce = 0% 2012-03-04 22:42:55,015 Stage-1 map = 83%, reduce = 0% 2012-03-04 22:42:58,181 Stage-1 map = 100%, reduce = 0% 2012-03-04 22:43:13,500 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203042123_0003 OK 500000 A decade ago, many were predicting that Cooke, a New York City prodigy, would become a basketball shoe pitchman and would flaunt his wares and skills at All-Star weekends like the recent aerial show in Orlando, Fla. There was a time, however fleeting, when he was more heralded, or perhaps merely hyped, than any other high school player in America. Time taken: 70.275 seconds hive>
三。加Index
hive> dfs -ls /user/hive/warehouse/; Found 2 items drwxr-xr-x - allen supergroup 0 2012-03-04 22:22 /user/hive/warehouse/table01 drwxr-xr-x - allen supergroup 0 2012-03-04 22:33 /user/hive/warehouse/table02 hive> show tables; OK table01 table02 Time taken: 3.754 seconds hive> create index table02_index on table table02(id) > as 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' > with deferred rebuild; OK Time taken: 0.913 seconds hive> alter index table02_index on table02 rebuild; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks not specified. Estimated from input data size: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapred.reduce.tasks=<number> Starting Job = job_201203051029_0001, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203051029_0001 Kill Command = /home/allen/Hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201203051029_0001 2012-03-05 10:39:43,607 Stage-1 map = 0%, reduce = 0% 。。。 2012-03-05 10:43:50,295 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203051029_0001 Loading data to table default.default__table02_table02_index__ Deleted hdfs://localhost:9000/user/hive/warehouse/default__table02_table02_index__ Table default.default__table02_table02_index__ stats: [num_partitions: 0, num_files: 1, num_rows: 0, total_size: 74701985] OK Time taken: 269.383 seconds hive>
查询一下，看看效果：6个Mapper，不work
确认索引建立成功没，先看看meta中怎么样：
mysql> select * from IDXS; +----------+-------------+------------------+-------------------------------------------------------------+---------------+--------------+------------------+-------------+-------+ | INDEX_ID | CREATE_TIME | DEFERRED_REBUILD | INDEX_HANDLER_CLASS | INDEX_NAME | INDEX_TBL_ID | LAST_ACCESS_TIME | ORIG_TBL_ID | SD_ID | +----------+-------------+------------------+-------------------------------------------------------------+---------------+--------------+------------------+-------------+-------+ | 1 | 1330914783 | | org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler | table02_index | 6 | 1330914783 | 2 | 7 | +----------+-------------+------------------+-------------------------------------------------------------+---------------+--------------+------------------+-------------+-------+ 1 row in set (0.00 sec) mysql> select * from TBLS; +--------+-------------+-------+------------------+-------+-----------+-------+----------------------------------+---------------+--------------------+--------------------+ | TBL_ID | CREATE_TIME | DB_ID | LAST_ACCESS_TIME | OWNER | RETENTION | SD_ID | TBL_NAME | TBL_TYPE | VIEW_EXPANDED_TEXT | VIEW_ORIGINAL_TEXT | +--------+-------------+-------+------------------+-------+-----------+-------+----------------------------------+---------------+--------------------+--------------------+ | 1 | 1330870868 | 1 | 0 | allen | 0 | 2 | table01 | MANAGED_TABLE | NULL | NULL | | 2 | 1330871615 | 1 | 0 | allen | 0 | 3 | table02 | MANAGED_TABLE | NULL | NULL | | 6 | 1330914783 | 1 | 0 | NULL | 0 | 10 | default__table02_table02_index__ | INDEX_TABLE | NULL | NULL | +--------+-------------+-------+------------------+-------+-----------+-------+----------------------------------+---------------+--------------------+--------------------+ 3 rows in set (0.01 sec) mys b193 ql>
确实，IDX中已经有了这个索引，而TBLS中也有了索引表，看看HDFS中的情况：
hive> dfs -ls /user/hive/warehouse; Found 3 items drwxr-xr-x - allen supergroup 0 2012-03-05 10:43 /user/hive/warehouse/default__table02_table02_index__ drwxr-xr-x - allen supergroup 0 2012-03-04 22:22 /user/hive/warehouse/table01 drwxr-xr-x - allen supergroup 0 2012-03-04 22:33 /user/hive/warehouse/table02 hive> dfs -ls /user/hive/warehouse/default*; Found 1 items -rw-r--r-- 1 allen supergroup 74701985 2012-03-05 10:42 /user/hive/warehouse/default__table02_table02_index__/000000_0 hive> dfs -dus /user/hive/warehouse/default*; hdfs://localhost:9000/user/hive/warehouse/default__table02_table02_index__ 74701985 hive>可见，确实是干活了的，东西都在，我们来看看索引表里存了些什么：
hive> select * from default__table02_table02_index__ limit 3; OK 0 hdfs://localhost:9000/user/hive/warehouse/table02/000000_0 [0] 1 hdfs://localhost:9000/user/hive/warehouse/table02/000000_0 [352] 2 hdfs://localhost:9000/user/hive/warehouse/table02/000000_0 [704] Time taken: 0.44 seconds hive>不出意外，应该是{值，HDFS文件位置，偏移量的数组（可能有多个）}
下面手动设置索引：
hive> SET hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat; hive> Insert overwrite directory "/tmp/table02_index_data" select `_bucketname`, `_offsets` from default__table02_table02_index__ where id =500000; Total MapReduce jobs = 2 Launching Job 1 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201203051029_0009, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203051029_0009 Kill Command = /home/allen/Hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201203051029_0009 2012-03-05 11:32:28,733 Stage-1 map = 0%, reduce = 0% 2012-03-05 11:32:38,022 Stage-1 map = 100%, reduce = 0% 2012-03-05 11:32:41,163 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203051029_0009 Ended Job = 1423815964, job is filtered out (removed at runtime). Launching Job 2 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201203051029_0010, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203051029_0010 Kill Command = /home/allen/Hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201203051029_0010 2012-03-05 11:32:47,102 Stage-2 map = 0%, reduce = 0% 2012-03-05 11:32:50,152 Stage-2 map = 100%, reduce = 0% 2012-03-05 11:32:53,265 Stage-2 map = 100%, reduce = 100% Ended Job = job_201203051029_0010 Moving data to: /tmp/table02_index_data 1 Rows loaded to /tmp/table02_index_data OK Time taken: 33.348 seconds hive> select * from table02 where id =500000; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201203051029_0011, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203051029_0011 Kill Command = /home/allen/Hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201203051029_0011 2012-03-05 11:34:17,235 Stage-1 map = 0%, reduce = 0% 2012-03-05 11:34:29,421 Stage-1 map = 33%, reduce = 0% 2012-03-05 11:34:41,688 Stage-1 map = 67%, reduce = 0% 2012-03-05 11:34:50,824 Stage-1 map = 83%, reduce = 0% 2012-03-05 11:34:53,878 Stage-1 map = 100%, reduce = 0% 2012-03-05 11:34:56,903 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203051029_0011 OK 500000 A decade ago, many were predicting that Cooke, a New York City prodigy, would become a basketball shoe pitchman and would flaunt his wares and skills at All-Star weekends like the recent aerial show in Orlando, Fla. There was a time, however fleeting, when he was more heralded, or perhaps merely hyped, than any other high school player in America. Time taken: 45.991 seconds hive> Set hive.index.compact.file=/tmp/table02_index_data; hive> Set hive.optimize.index.filter=false; hive> Set hive.input.format=org.apache.hadoop.hive.ql.index.compact.HiveCompactIndexInputFormat; hive> select * from table02 where id =500000; Total MapReduce jobs = 1 Launching Job 1 out of 1 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201203051029_0012, Tracking URL = http://localhost:50030/jobdetails.jsp?jobid=job_201203051029_0012 Kill Command = /home/allen/Hadoop/hadoop-0.20.2/bin/../bin/hadoop job -Dmapred.job.tracker=localhost:9001 -kill job_201203051029_0012 2012-03-05 11:35:50,694 Stage-1 map = 0%, reduce = 0% 2012-03-05 11:35:56,797 Stage-1 map = 100%, reduce = 0% 2012-03-05 11:35:59,851 Stage-1 map = 100%, reduce = 100% Ended Job = job_201203051029_0012 OK 500000 A decade ago, many were predicting that Cooke, a New York City prodigy, would become a basketball shoe pitchman and would flaunt his wares and skills at All-Star weekends like the recent aerial show in Orlando, Fla. There was a time, however fleeting, when he was more heralded, or perhaps merely hyped, than any other high school player in America. Time taken: 14.367 seconds hive>
OK！生效了，变成了一个Mapper！不过这里只手动插入了id=500000的索引：
allen@allen-laptop:~$ hadoop dfs -cat /tmp/table02_index_data/000000_0; hdfs://localhost:9000/user/hive/warehouse/table02/000002_044170896 allen@allen-laptop:~$
至于Hive怎么做的，下篇文章探索吧
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- Hive-0.8.1 索引解析（CompactIndex）
- 外键要建立索引的原理和实验
- Nutz Dao实体中索引注解的使用(@TableIndexes@Index)
- Sqlserver 2005+：查看索引【index】的【使用情况】：无效的索引、高成本索引
- 建立HBase的集群和HDInsight在Hadoop中使用Hive来查询它们
- 游标的慎用和索引(Index)的使用
- 如何使用solr对文件建立索引
- Handlebars.js循环中索引(@index)使用技巧(访问父级索引)
- mySQL给空间字段建立索引出现错误All parts of a SPATIAL index must be NOT NULL的解决办法
- 使用NOSQL的MongoDB建立索引时需要注意的几点建议和Explain优化分析
- IndexSearch中增量索引使用reopen
- [MySQL 工具] pt-index-usage使用 --> 对查询中的索引进行分析
- 在MYSQL中使用全文索引(FULLTEXTindex)MYSQL的一个很有用的特性是使用全文索引
- mysql建立、删除索引及使用
- ★ 使用索引的误区之六：为所有列都建立索引
- 使用索引的误区之六：为所有列都建立索引
- mysql中index的如何使用，判断什么时候加索引（强烈推荐）
- mysql字段类型为char类型，建立索引但未使用原因
- 使用Lucene建立索引
- 使用Lucene对建立的索引进行搜索