Stream

Gary Helmling
moderator

Discussion  - 
 
Announcing HBaseCon East 2016

HBaseCon East will be held in NYC on September 26th. For more information and to RSVP, go to http://www.meetup.com/HBase-NYC/events/233024937/

The Call for Papers is currently open until September 4th. If you have an interesting use of HBase to discuss, submit a proposal at https://easychair.org/conferences/?conf=hbaseconeast2016

Hope to see some of you there!
1
Add a comment...

shubham agarwal

Discussion  - 
 
Hi,
Whenever I tried to start Hbase Master it dies. The logs of Hbase Master says "HBase file layout needs to be upgraded. You have version null and I want version 8. Consult http://hbase.apache.org/book.html for further information about upgrading HBase. Is your hbase.rootdir valid? If so, you may need to run 'hbase hbck -fixVersionFile'." 

But I can't do a 'hbase hbck -fixVersionFile' since master is not running. So can anyone help me how to resolve this.
Herein you will find either the definitive documentation on an HBase topic as of its standing when the referenced HBase version shipped, or it will point to the location in Javadoc or JIRA where the pertinent information can be found. About This Guide. This reference guide is a work in progress.
1
Cosmin Lehene's profile photo
 
Perhaps you have an empty hdfs://hbase directory? If you don't need the data you can just delete it and let HBase create it on startup. 
Add a comment...

Justin Kurian

Discussion  - 
 
Review Copies available for “Learning Hbase”
We are currently looking for reviewers for our above book. I'll be happy to provide a free eBook to anyone interested in writing a review for this book specifically on their Blog/Website and on Amazon within 2 weeks after receiving the eBook.
For further information about the book, please visit: http://bit.ly/14cHpad

If interested, just drop a line in the comments below with following required details:
1. Your Email address
2. Choice of website where you wish to post the book-review

Note: Limited Copies Available. Please send in your entries within a week.
Apache HBase is a nonrelational NoSQL database management system that runs on top of HDFS. It is an open source, distributed, versioned, column-oriented store. It facilitates ...
3
Steve Johnson's profile photoÁlvaro García's profile photo
2 comments
 
Feel free to send it over JK.
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
HBaseCon 2015 registration is now open!  Sign up now to get the Early Bird rate.  Stay tuned for the conference Call-for-Papers.
1
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
The session list has been posted for #HBaseCon2014  and it's chock full of great talks!
HBaseCon 2014 User Conference: San Francisco CA, May 5.
2
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
Time is running out to get your submissions in for #HBaseCon2014!

Don't delay, get your submissions in now!

http://hbasecon.com/
1
Add a comment...

Andrew Purtell
moderator

Discussion  - 
 
Phoenix v 2.1 is released.

From James Taylor (@JamesPlusPlus):

The Phoenix team is pleased to announce the immediate availability of Phoenix 2.1 [1].

More than 20 individuals contributed to the release. Here are some of the new features now available:

* Secondary Indexing [2] to create and automatically maintain global
indexes over your primary table.
   - Queries automatically use an index when more efficient, turning your full table scans into point and range scans.
   - Multiple columns may be indexed in ascending or descending sort order.
   - Additional primary table columns may be included in the index to form a covered index.
   - Available in two flavors:
        o Server-side index maintenance for mutable data.
        o Client-side index maintenance optimized for write-once, append-only use cases.

* Row Value Constructors [3], a standard SQL construct to efficiently
locate the row at or after a composite key value.
   - Enables a query-more capability to efficiently step through your data.
   - Optimizes IN list of composite key values to be point gets.

* Map-reduce based CSV Bulk Loader [4] to build Phoenix-compliant HFiles and load them into HBase.

* MD5 hash and INVERT built-in functions

Phoenix 2.1 requires HBase 0.94.4 or above, with 0.94.10 or above required for mutable secondary indexing. For the best performance, we recommend HBase 0.94.12 or above. 

Regards,

James
@JamesPlusPlus
http://phoenix-hbase.blogspot.com/

[1] https://github.com/forcedotcom/phoenix/wiki/Download
[2] https://github.com/forcedotcom/phoenix/wiki/Secondary-Indexing
[3] https://github.com/forcedotcom/phoenix/wiki/Row-Value-Constructors
[4] https://github.com/forcedotcom/phoenix/wiki/Bulk-CSV-loading-through-map-reduce 
9
3
罗飞龙's profile photo
 
now, Phoenix v 2.2.3 is released
Add a comment...

Gary Helmling
moderator

Discussion  - 
Sub-task. [HBASE-9277] - REST should use listTableNames to list tables; [HBASE-9279] - Thrift should use listTableNames to list tables. Bug. [HBASE-7709] - Infinite loop possible in Master/Master replication; [HBASE-7954] - Fix the retrying logic of memstore flushes to avoid extra sleep ...
2
2
Add a comment...
 
hindex by Huawei:

https://github.com/Huawei-Hadoop/hindex

HBase Secondary Index is 100% server side implementation with co processors which persists index data in a separate table. Indexing is region wise and custom load balancer co-locates the index table regions with actual table regions.
6
1
Rex Wong's profile photo
 
this project change a lot of hbase native code.
Add a comment...

About this community

Apache HBase Related Discussions & News

Anoop Sam John
moderator

Discussion  - 
 
Offheaping the Read Path in Apache HBase:
https://blogs.apache.org/hbase/entry/offheaping_the_read_path_in
Apache HBase. « Offheaping the Read... | Main. Thursday Dec 17, 2015. Offheaping the Read Path in Apache HBase: Part 1 of 2. by HBase Committers Anoop Sam John, Ramkrishna S Vasudevan, and Michael Stack. Part one of a two part blog.
1
2
‫عابد کمی پور‬‎'s profile photo
 
OK Samsung s3 mino
Add a comment...

shubham agarwal

Discussion  - 
 
Hi, I am using Hbase for OpenTSDB and I accidentally deleted some of the hdfs datanodes from the cluster. Now when I am trying to start Hbase-Master and  Hbase-Regionserver, the region-server appears to be continuously splitting regions and after some times Hbase-Master dies. Can someone help to resolve the issue. 
1
shubham agarwal's profile photo
 
Here are the regionserver logs.

2015-11-24 11:26:00,548 INFO  [RS_LOG_REPLAY_OPS-] util.FSHDFSUtils: recoverLease=false, attempt=14 on file=hdfs://localhost/hbase/WALs/,1448343272089-splitting/%2C60020%2C1448343272089..meta.1448343280481.meta after 837759ms
2015-11-24 11:26:58,137 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.13 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=89, evicted=0, evictedPerRun=0.0
2015-11-24 11:27:04,654 INFO  [RS_LOG_REPLAY_OPS-:60020-0] util.FSHDFSUtils: recoverLease=false, attempt=15 on file=hdfs://localhost/hbase/WALs/,60020,1448343272089-splitting/%2C60020%2C1448343272089..meta.1448343280481.meta after 901865ms
2015-11-24 11:27:04,654 WARN  [RS_LOG_REPLAY_OPS-:60020-0] util.FSHDFSUtils: Cannot recoverLease after trying for 900000ms (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!; attempt=15 on file=hdfs://localhost/hbase/WALs/,60020,1448343272089-splitting/.visa.com%2C60020%2C1448343272089..meta.1448343280481.meta after 901865ms
2015-11-24 11:27:04,922 WARN  [RS_LOG_REPLAY_OPS-:60020-0] wal.WALFactory: Lease should have recovered. This is not expected. Will retry
java.io.IOException: Cannot obtain block length for LocatedBlock{BP-1837467880-10.211.26.203-1439511762417:blk_1073763333_22515; getBlockSize()=83; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[127.0.0.1:50010,DS-b74ae2a5-b2d6-42df-848d-c1aee5cfc112,DISK]]}
at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:386)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:329)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:257)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1492)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:290)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:266)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:839)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:763)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:304)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:242)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-11-24 11:31:58,137 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.13 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=119, evicted=0, evictedPerRun=0.0
2015-11-24 11:32:05,451 ERROR [RS_LOG_REPLAY_OPS-:60020-0] wal.WALFactory: Can't open after 300 attempts and 300797ms  for hdfs://localhost/hbase/WALs/,60020,1448343272089-splitting/%2C60020%2C1448343272089..meta.1448343280481.meta
2015-11-24 11:32:05,453 INFO  [RS_LOG_REPLAY_OPS--0] wal.WALSplitter: Processed 0 edits across 0 regions; edits skipped=0; log file=hdfs://localhost/hbase/WALs/,60020,1448343272089-splitting/%2C60020%2C1448343272089..meta.1448343280481.meta, length=83, corrupted=false, progress failed=false
2015-11-24 11:32:05,453 WARN  [RS_LOG_REPLAY_OPS:60020-0] regionserver.SplitLogWorker: log splitting of WALs/,60020,1448343272089-splitting/%2C60020%2C1448343272089..meta.1448343280481.meta failed, returning error
java.io.IOException: Cannot obtain block length for LocatedBlock{BP-1837467880-10.211.26.203-1439511762417:blk_1073763333_22515; getBlockSize()=83; corrupt=false; offset=0; locs=[DatanodeInfoWithStorage[127.0.0.1:50010,DS-b74ae2a5-b2d6-42df-848d-c1aee5cfc112,DISK]]}
at org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:386)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:329)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:257)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1492)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:766)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:290)
at org.apache.hadoop.hbase.wal.WALFactory.createReader(WALFactory.java:266)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:839)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:763)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:304)
at org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:242)
at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
at org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2015-11-24 11:32:05,458 INFO  [RS_LOG_REPLAY_OPS-:60020-0] coordination.ZkSplitLogWorkerCoordination: successfully transitioned task /hbase/splitWAL/WALs%2F%2C60020%2C1448343272089-splitting%2Fsl73operadbd001.visa.com%252C60020%252C1448343272089..meta.1448343280481.meta to final state ERR ,60020,1448363517961
2015-11-24 11:32:05,464 INFO  [RS_LOG_REPLAY_OPS-:60020-0] handler.WALSplitterHandler: worker ,60020,1448363517961 done with task org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$ZkSplitTaskDetails@a61bf66 in 1202744ms
2015-11-24 11:36:58,137 INFO  [LruBlockCacheStatsExecutor] hfile.LruBlockCache: totalSize=3.29 MB, freeSize=3.13 GB, max=3.13 GB, blockCount=0, accesses=0, hits=0, hitRatio=0, cachingAccesses=0, cachingHits=0, cachingHitsRatio=0,evictions=149, evicted=0, evictedPerRun=0.0
Add a comment...
 
Hbase Infrastructure sizing:
Please let me know is there any template or method available to estimate the number of Region servers required and data nodes based on the number of records to be ingested.

thanks,
Prabhanjana
2
Srigopal Mohanty's profile photoPrabhanjana Guttal's profile photo
2 comments
 
Thanks Srigopal, link is very very helpful.
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
HBaseCon 2015 Call for Papers is now open!  Have an interesting use case or lessons learned from HBase that you'd like to share?  Get your presentation proposal in!

Also, take advantage of the Early Bird registration discount before Feb 1st!
2
1
Add a comment...
 
I have created this HBase cluster based on Vagrant. It can be used for experimentation and has other apps installed on it, like Hive, Pig, Phoenix, Sqoop and Zookeeper.
hbase-cluster - Ready-made HBase cluster
2
Add a comment...

罗飞龙

Discussion  - 
 
6台dn,数据量是8亿,使用phoenix查询,所有region查找,时间消耗需要5s,是否正常呢
 ·  Translate
1
Rex Wong's profile photo罗飞龙's profile photo
2 comments
 
+Rex Wong 你好,是否能指点一下呢?我现在的方案是,为了满足业务需求,源数据表中使用GUID作为rowkey,然后再创建一个表(作为二级索引表),将查询条件作为这个表的rowkey,cf中放的是源数据表的rowkey值,查找的时候先查二级索引表,再根据rowkey值去源数据表查找。
目前数据量是8亿,源数据表和二级索引表的region数量都到400多,现在查询需要5s左右(查询时正在插入大量数据)
 ·  Translate
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
The HBaseCon 2014 website is now up and the Call for Papers is now open!  Get your submissions in before February 14th!
2
1
Kenneth Ho's profile photoGary Helmling's profile photo
2 comments
 
Hmm, ticket to HBaseCon?  It's the perfect gift!
Add a comment...

Sonal Gupta

Discussion  - 
 
Hi, we are Starting a new HBase Batch on 15 Oct'13 Weekend 7:00 AM-10:00AM (IST). if you are Interested please
give us a call on 9019368913.
sales[@]intellipaat[dot]com
1
Rahul Vishnoi's profile photo罗飞龙's profile photo
2 comments
 
we try to use phoenix to add record to hbase, do you have any better choice for me?
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
Apache HBase 0.94.11 has just been released with 39 issues resolved!

See the list of changes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12324741

And download the release:
http://www.apache.org/dyn/closer.cgi/hbase/
Sub-task. [HBASE-8779] - Add mutateRow method support to Thrift2; [HBASE-8946] - Add a new function to Thrift 2 to open scanner, get results and close scanner; [HBASE-8947] - Thrift 2 : Replace "bool writeToWAL" with "TDurability durability"; [HBASE-8948] - Bound table pool size in Thrift 2 ...
3
1
Add a comment...

Andrew Purtell
moderator

Discussion  - 
 
Taking the Bait. Lars Hofhansl, Andrew Purtell, and Michael Stack. HBase Committers. Information Week recently published an article titled “Will HBase Dominate NoSQL?”. Michael Hausenblas of MapR argues the 'For' HBase case and Jonathan Ellis of Apache Cassandra and vendor DataStax argues ...
5
1
Add a comment...

Gary Helmling
moderator

Discussion  - 
 
Presentations from HBaseCon 2013 are now available online.  Just click on the session name and see a link to the slides at the bottom of the page.  Videos should follow soon!

http://www.hbasecon.com/schedule/
HBaseCon 2013 is a wrap, thanks to our speakers, sponsors, and attendees! Session slides are now available via the session description pages; video will be available soon. HBaseCon 2013 sessions are organized into four tracks: Operations, Internals, Ecosystem, and Case Studies.
5
3
Add a comment...