org.apache.hadoop.hbase.coprocessor.AggregateImplementation 来统计表的行数
2015-04-24 20:58
826 查看
多谢楼主的分享!
转载自:http://blog.csdn.net/xiao_jun_0820/article/details/25413919
hbase自带了一个聚合coprocessor类:org.apache.hadoop.hbase.coprocessor.AggregateImplementation。使用该类可以count一张表的总记录数。
当然在hbase shell下面也可以count <table_name>来统计。我这里比较了一下两者的执行时间,我有一张表有700多万的数据,在hbase shell下count足足花费了我12分钟的时间,而用coprocessor来统计,只花费了78秒!!!由此可见coprocessor的强大。
hbase aip 添加协处理器:
[java] view
plaincopy
Configuration hbaseconfig = HBaseConfiguration.create();
HBaseAdmin hbaseAdmin = new HBaseAdmin(hbaseconfig);
hbaseAdmin.disableTable(TABLE_NAME);
HTableDescriptor htd = hbaseAdmin.getTableDescriptor(TABLE_NAME);
htd.addCoprocessor(AggregateImplementation.class.getName());
hbaseAdmin.modifyTable(TABLE_NAME, htd);
hbaseAdmin.enableTable(TABLE_NAME);
hbaseAdmin.close();
使用hbase提供的聚合coprocessor:
[java] view
plaincopy
AggregationClient aggregationClient = new AggregationClient(hbaseconfig);
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes("fr"));
Date start = new Date();
long rowcount = aggregationClient.rowCount(TABLE_NAME,
new LongColumnInterpreter(), scan);
Date end = new Date();
System.out.println("rowcount:" + rowcount);
System.out.println("timecost:" + (end.getTime() - start.getTime()));
hbase shell添加coprocessor:
disable 'member'
alter 'member',METHOD => 'table_att','coprocessor' => 'hdfs://master24:9000/user/hadoop/jars/test.jar|mycoprocessor.SampleCoprocessor|1001|'
enable 'member'
hbase shell 删除coprocessor:
disable 'member'
alter 'member',METHOD => 'table_att_unset',NAME =>'coprocessor$1'
enable 'member'
转载自:http://blog.csdn.net/xiao_jun_0820/article/details/25413919
hbase自带了一个聚合coprocessor类:org.apache.hadoop.hbase.coprocessor.AggregateImplementation。使用该类可以count一张表的总记录数。
当然在hbase shell下面也可以count <table_name>来统计。我这里比较了一下两者的执行时间,我有一张表有700多万的数据,在hbase shell下count足足花费了我12分钟的时间,而用coprocessor来统计,只花费了78秒!!!由此可见coprocessor的强大。
hbase aip 添加协处理器:
[java] view
plaincopy
Configuration hbaseconfig = HBaseConfiguration.create();
HBaseAdmin hbaseAdmin = new HBaseAdmin(hbaseconfig);
hbaseAdmin.disableTable(TABLE_NAME);
HTableDescriptor htd = hbaseAdmin.getTableDescriptor(TABLE_NAME);
htd.addCoprocessor(AggregateImplementation.class.getName());
hbaseAdmin.modifyTable(TABLE_NAME, htd);
hbaseAdmin.enableTable(TABLE_NAME);
hbaseAdmin.close();
使用hbase提供的聚合coprocessor:
[java] view
plaincopy
AggregationClient aggregationClient = new AggregationClient(hbaseconfig);
Scan scan = new Scan();
scan.addFamily(Bytes.toBytes("fr"));
Date start = new Date();
long rowcount = aggregationClient.rowCount(TABLE_NAME,
new LongColumnInterpreter(), scan);
Date end = new Date();
System.out.println("rowcount:" + rowcount);
System.out.println("timecost:" + (end.getTime() - start.getTime()));
hbase shell添加coprocessor:
disable 'member'
alter 'member',METHOD => 'table_att','coprocessor' => 'hdfs://master24:9000/user/hadoop/jars/test.jar|mycoprocessor.SampleCoprocessor|1001|'
enable 'member'
hbase shell 删除coprocessor:
disable 'member'
alter 'member',METHOD => 'table_att_unset',NAME =>'coprocessor$1'
enable 'member'
相关文章推荐
- org.apache.hadoop.hbase.coprocessor.AggregateImplementation 来统计表的行数
- HBase源码分析之org.apache.hadoop.hbase.client.coprocessor包
- trafodion Exception ERROR Class org.apache.hadoop.hbase.coprocessor.transactional.TrxRegionEndpoint
- org.apache.hadoop.hbase.DoNotRetryIOException: Class org.apache.phoenix.coprocessor.MetaDataEndpointImpl cannot be loaded Set hbase.table.sanity.checks to false at conf or table descriptor if you want
- HBase报错:No jar path specified for org.apache.phoenix.coprocessor.UngroupedAggregateRegionObserver
- 删除hbase表时 org.apache.hadoop.hbase.TableNotDisabledException 解决方法
- How-to: resolve hbase "org.apache.hadoop.hbase.TableExistsException: hbase:namespace"
- HBase运行中抛出异常org.apache.hadoop.hbase.client.ScannerTimeoutException
- HBase的NoSuchMethodError:org.apache.hadoop.net.NetUtils.getInputStream异常解决办法
- org.apache.hadoop.hbase.ClockOutOfSyncException
- Java 向Hbase表插入数据报错(org.apache.hadoop.hbase.client.HTablePool$PooledHTable cannot be cast to org.apa)
- hbase无法启动Regionserver:Failed construction of Regionserver: class org.apache.hadoop.hbase.regions问题解决
- 运行HBase应用开发程序产生异常,提示信息包含org.apache.hadoop.hbase.ipc.controller.ServerRpcControllerFactory的解决办法
- hadoop-0.20.2 & hbase-0.90.1 集群启动错误“org.apache.hadoop.ipc.RPC$VersionMismatch: Protocol org.apache.hadoop.hdfs.protocol.ClientP
- Hbase 出现 org.apache.hadoop.hbase.ipc.ServerNotRunningYetException: Server is not running yet 错误
- Spark 读取Hbase 映射到Hive中的外部表报java.lang.NoSuchMethodError: org.apache.hadoop.hive.serde2.lazy.LazySim
- ERROR: org.apache.hadoop.hbase.PleaseHoldException: Master is initializing
- org.apache.hadoop.hbase.NotServingRegionException: Region is not online 错误
- nutch2.3 hadoop2.6.0 hbase0.98.8 分布式爬虫NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfigurati
- WARN org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Region was hijacked