org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for

[mattshma]

客户端机器在读写的过程连接一直断开，查看RegionServer log报错如下：

2016-06-20 10:22:45,238 ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":1,"families":{"info":[{"timestamp":1465982388077,"tag":[],"qualifier":"","vlen":115}]},"row":"2021310420_17179_16980762036709551540"}
2016-06-20 10:22:45,342 ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":1,"families":{"info":[{"timestamp":1466130504096,"tag":[],"qualifier":"","vlen":111}]},"row":"2033311012_16013_16980616350709551584"}

查看HBaseAsyncProcess代码：

    private void manageLocationError(Action<Row> action, Exception ex) {
      String msg = "Cannot get replica " + action.getReplicaId()
          + " location for " + action.getAction();
      LOG.error(msg);
      if (ex == null) {
        ex = new IOException(msg);
      }
      manageError(action.getOriginalIndex(), action.getAction(),
          Retry.NO_LOCATION_PROBLEM, ex, null);
    }

而在查找RS的过程中会调用该代码：

    private RegionLocations findAllLocationsOrFail(Action<Row> action, boolean useCache) {
      if (action.getAction() == null) throw new IllegalArgumentException("#" + id +
          ", row cannot be null");
      RegionLocations loc = null;
      try {
        loc = connection.locateRegion(
            tableName, action.getAction().getRow(), useCache, true, action.getReplicaId());
      } catch (IOException ex) {
        manageLocationError(action, ex);
      }
      return loc;
    }

connection定义为protected final ClusterConnection connection。ClusterConnection是Connection接口:

 /**
  *
  * @param tableName table to get regions of
  * @param row the row
  * @param useCache Should we use the cache to retrieve the region information.
  * @param retry do we retry
  * @param replicaId the replicaId for the region
  * @return region locations for this row.
  * @throws IOException if IO failure occurs
  */
  RegionLocations locateRegion(TableName tableName, byte[] row, boolean useCache, boolean retry,
     int replicaId) throws IOException;

当前实现该接口的类只有ConnectionImplementation，其实现如下：

  @Override
  public RegionLocations locateRegion(final TableName tableName,
    final byte [] row, boolean useCache, boolean retry, int replicaId)
  throws IOException {
    if (this.closed) {
      throw new IOException(toString() + " closed");
    }
    if (tableName== null || tableName.getName().length == 0) {
      throw new IllegalArgumentException(
          "table name cannot be null or zero length");
    }
    if (tableName.equals(TableName.META_TABLE_NAME)) {
      return locateMeta(tableName, useCache, replicaId);
    } else {
      // Region not in the cache - have to go to the meta RS
      return locateRegionInMeta(tableName, row, useCache, retry, replicaId);
    }
  }

大致过程就是熟知的过程：先从cache里面查找rs位置，若找不到，则从meta表中查找，针对该问题，聘问的情况即找不到rs位置，常见的原因一般是脚本机的/etc/hosts没配置regionserver的ip hostname对应关系。

解决方法：
在脚本机的/etc/hosts中添加所有RegionServer的ip hostname对应关系。

[hoangdh]

2016-06-20 10:22:45,238 ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":1,"families":{"info":[{"timestamp":1465982388077,"tag":[],"qualifier":"","vlen":115}]},"row":"2021310420_17179_16980762036709551540"}
2016-06-20 10:22:45,342 ERROR org.apache.hadoop.hbase.client.AsyncProcess: Cannot get replica 0 location for {"totalColumns":1,"families":{"info":[{"timestamp":1466130504096,"tag":[],"qualifier":"","vlen":111}]},"row":"2033311012_16013_16980616350709551584"}

"qualifier":"" Because, this field is empty. (Same my case)