Memstore is above high water mark and block 15001ms

[mattshma]

报错如下：

2016-03-14 12:36:50,882 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 20000ms
2016-03-14 12:36:50,883 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Memstore is above high water mark and block 20001ms
2016-03-14 12:37:35,666 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~8.5 M/8951872, currentsize=0/0 for region snsgz_log,2021310319_10015_169980174663818,1451216519140.a47bc9f9b49d8dbff9988149a4d903f6. in 141ms, sequenceid=53052750, compaction requested=true
2016-03-14 12:37:35,708 WARN org.apache.hadoop.ipc.RpcServer: RpcServer.listener,port=60020: count of bytes read: 0
java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:197)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
        at org.apache.hadoop.hbase.ipc.RpcServer.channelRead(RpcServer.java:2229)
        at org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1415)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:790)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:581)
        at org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:556)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
2016-03-14 12:37:59,013 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~11.0 M/11511720, currentsize=0/0 for region snsgz_log,2021310366_10560_16994290604709551604,1455903941943.cf8feecfa2b18eaa640c647e5a0b3586. in 172ms, sequenceid=56409502, compaction requested=true
2016-03-14 12:38:02,466 WARN org.apache.hadoop.hdfs.BlockReaderFactory: I/O error constructing remote block reader.
java.io.IOException: Got error for OP_READ_BLOCK, self=/10.2.96.34:16940, remote=/10.2.96.34:50010, for file /hbase/data/default/snsgz_log/46dbc88e3da64b8618aebff1f088b027/info/c01bb7dd72af409592bfa76cbb95dbf8, for pool BP-1471860497-10.2.72.29-1421306158975 block 1305075939_231354785
2016-03-14 12:38:02,467 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.2.96.34:50010 for block, add to deadNodes and continue. java.io.IOException: Got error for OP_READ_BLOCK, self=/10.2.96.34:16940, remote=/10.2.96.34:50010, for file /hbase/data/default/snsgz_log/46dbc88e3da64b8618aebff1f088b027/info/c01bb7dd72af409592bfa76cbb95dbf8, for pool BP-1471860497-10.2.72.29-1421306158975 block 1305075939_231354785
java.io.IOException: Got error for OP_READ_BLOCK, self=/10.2.96.34:16940, remote=/10.2.96.34:50010, for file /hbase/data/default/snsgz_log/46dbc88e3da64b8618aebff1f088b027/info/c01bb7dd72af409592bfa76cbb95dbf8, for pool BP-1471860497-10.2.72.29-1421306158975 block 1305075939_231354785
        at org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:432)
        at org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:397)
        at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReader(BlockReaderFactory.java:786)
        at org.apache.hadoop.hdfs.BlockReaderFactory.getRemoteBlockReaderFromTcp(BlockReaderFactory.java:665)
        at org.apache.hadoop.hdfs.BlockReaderFactory.build(BlockReaderFactory.java:325)
        at org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:566)
        at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:789)
2016-03-14 12:43:30,870 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: ABORTING region server 10-2-96-34.dn-hadoop-platform.dh.idc.youzu.com,60020,1457770771591: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 10-2-96-34.dn-hadoop-platform.dh.idc.youzu.com,60020,1457770771591 as dead server
        at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:341)
        at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:254)
        at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1357)
        at org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$2.callBlockingMethod(RegionServerStatusProtos.java:5087)
        at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2012)
        at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:98)
        at org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:73)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

org.apache.hadoop.hbase.YouAreDeadException: org.apache.hadoop.hbase.YouAreDeadException: Server REPORT rejected; currently processing 10-2-96-34.dn-hadoop-platform.dh.idc.youzu.com,60020,1457770771591 as dead server