Zookeeper

Zookeeper简介

##简介
Zookeeper是一个针对大型分布式系统的可靠协调系统，提供的功能包括：配置维护、名字服务、分布式同步、组服务等。

参考： http://www.ibm.com/developerworks/cn/opensource/os-cn-zookeeper/

##特点

###1. 支持集群部署、水平扩展
zookeeper部署

无论Client连接的是哪个Server，获取到的数据都是一致的

###2. 数据结构——znode
与文件系统目录结构类似，每个znode可存储少量数据，适合用来存储状态、配置、网络位置等信息
znode层次结构

客户端可以监控znode的状态

###3. 读取速度快
zookeeper的数据缓存在内存中，定期持久化到磁盘，所以读取速度很快
zookeeper性能测试数据

读占比越大，并发越高

###4. API简单
zookeeper只提供以下方法

1
2
3
4
5
6
7
create：创建节点
delete：删除节点
exists：判断节点是否存在
get data：从节点获取数据
set data：往节点写数据
get children：获取子节点
sync：同步数据

##启动zookeeper
进入$ZOOKEEPER_HOME/bin，运行zkServer.sh start

##命令行交互
运行zkCli.sh，输入help可以查看所有命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[zk: localhost:2181(CONNECTED) 0] help
ZooKeeper -server host:port cmd args
        connect host:port
        get path [watch]
        ls path [watch]
        set path data [version]
        rmr path
        delquota [-n|-b] path
        quit 
        printwatches on|off
        create [-s] [-e] path data acl
        stat path [watch]
        close 
        ls2 path [watch]
        history 
        listquota path
        setAcl path acl
        getAcl path
        sync path
        redo cmdno
        addauth scheme auth
        delete path [version]
        setquota -n|-b val path

HBase

HBase Shell常用命令

进入shellhbase shell

1
2
3
4
5
6
7
[hduser@hadoop2 applications]$ hbase shell
2014-05-20 14:48:54,159 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.96.2-hadoop2, r1581096, Mon Mar 24 16:03:18 PDT 2014
hbase(main):001:0>

退出shellexit

1
2
hbase(main):009:0> exit
[hduser@hadoop2 ~]$

列出所有表list

1
2
3
4
5
6
7
8
hbase(main):001:0> list
TABLE                                                                                                                                                         
table2              
tables              
test1                                                                                             
3 row(s) in 4.9810 seconds
=> ["table2", "tables", "test1"]

查看表定义describe

1
2
3
4
5
hbase(main):008:0> describe 'table2'
DESCRIPTION                                                                                                        ENABLED                                                        
 'table2', {NAME => 'col1', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'ROW', REPLICATION_SCOPE => '0', VERSION true                                                           
 S => '1', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSI                                                                
 ZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}

列出部分行scan

1
2
3
4
5
6
7
8
scan '<tablename>', {LIMIT=> 2}
例子
hbase(main):007:0> scan 'test',{LIMIT=>2}
ROW                                           COLUMN+CELL                                                                                                                         
 row1                                         column=col1:, timestamp=1400570581885, value=1                                                                                      
 row2                                         column=col1:, timestamp=1400571033424, value=2                                                                                      
2 row(s) in 0.0230 seconds
HBase

HBase API操作

##1. 新建Java Project
新建一个Java Project后，引入HBase的所有jar包，将服务器上的hbase-site.xml拷贝到src目录

##2. 总体架构
表管理类操作模式：

1
2
3
4
Configuration config = HBaseConfiguration.create();
HBaseAdmin admin = new HBaseAdmin(config);
......    
admin.doSomething();

表操作类模式：

1
2
3
4
5
Configuration conf = HBaseConfiguration.create();
String tableName = ...;
HTable table = new HTable(conf, tableName);
......
table.doSomething();

###2.1 创建表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
public class CreateTableAction {
    /**
     * @param args
     * @throws IOException 
     * @throws ZooKeeperConnectionException 
     * @throws MasterNotRunningException 
     */
    public static void main(String[] args) throws MasterNotRunningException, ZooKeeperConnectionException, IOException {
        Configuration config = HBaseConfiguration.create();
        HBaseAdmin admin = new HBaseAdmin(config);
        
        TableName tableName = TableName.valueOf("table3");
        HTableDescriptor descriptor = new HTableDescriptor(tableName);
        HColumnDescriptor colDescriptor = new HColumnDescriptor("col1");
        descriptor.addFamily(colDescriptor);
        
        admin.createTable(descriptor);
    }
}

###2.2 删除表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public class DeleteTableAction {
    /**
     * @param args
     * @throws IOException 
     * @throws ZooKeeperConnectionException 
     * @throws MasterNotRunningException 
     */
    public static void main(String[] args) throws MasterNotRunningException, ZooKeeperConnectionException, IOException {
        Configuration config = HBaseConfiguration.create();
        HBaseAdmin admin = new HBaseAdmin(config);
        
        String tableName = "table3";
        admin.disableTable(tableName);  // delete前必须先disable
        admin.deleteTable(tableName);  
    }
}

###2.3 列出所有表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
public class ListTablesAction {
    /**
     * @param args
     * @throws IOException 
     * @throws ZooKeeperConnectionException 
     * @throws MasterNotRunningException 
     */
    public static void main(String[] args) throws MasterNotRunningException, ZooKeeperConnectionException, IOException {
        Configuration config = HBaseConfiguration.create();
        HBaseAdmin admin = new HBaseAdmin(config);
        
        TableName[] tableNames = admin.listTableNames();
        for (TableName name : tableNames) {
            System.out.println("Table: " + name.getNameAsString());
        }
    }
}

###2.4 插入单条记录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class InsertOneRowAction {
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        Configuration conf = HBaseConfiguration.create();
        String tableName = "table2";
        HTable table = new HTable(conf, tableName);
        Put put = new Put("row2".getBytes());
        put.add("col1".getBytes(), null, "2".getBytes()); // 第一个参数是列簇名，第二个参数是限定符（我的理解就是子列名），第三个参数是值
        table.put(put);
    }
}

在hbase shell查询

1
2
3
4
hbase(main):004:0> scan 'table2'
ROW                                           COLUMN+CELL                                                                                                                         
 row2                                         column=col1:, timestamp=1400593778072, value=2                                                                                      
1 row(s) in 0.0700 seconds

PS: put方法不是线程安全的

###2.5 批量插入

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
public class InsertRowsAction {
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        Configuration conf = HBaseConfiguration.create();
        String tableName = "table2";
        HTable table = new HTable(conf, tableName);
        Put put3 = new Put("row3".getBytes());
        put3.add("col1".getBytes(), null, "3".getBytes());
        
        Put put4 = new Put("row4".getBytes());
        put4.add("col1".getBytes(), null, "4".getBytes());
        
        List<Put> puts = new ArrayList<Put>();
        puts.add(put3);
        puts.add(put4);
        
        table.put(puts);  //参数为List<Put>就是批量插入
        
        table.close();
    }
}

###2.5 单条查询

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
public class GetOneRowAction {
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        Configuration conf = HBaseConfiguration.create();
        String tableName = "table2";
        HTable table = new HTable(conf, tableName);
        Get get = new Get("row2".getBytes());
        Result result = table.get(get);
        System.out.println(new String(result.getRow()));                           // 获取row key
        System.out.println(new String(result.getValue("col1".getBytes(), null)));  // 获取指定字段的值
    }
}

###2.6 批量查询

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
public class GetRowsAction {
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        Configuration conf = HBaseConfiguration.create();
        String tableName = "table2";
        HTable table = new HTable(conf, tableName);
        
        System.out.println("======= 全表扫描 ========="); 
        Scan scan = new Scan();
        ResultScanner scaner = table.getScanner(scan);
        for (Result result : scaner) {
            System.out.println(new String(result.getRow()) + " : " + new String(result.getValue("col1".getBytes(), null)));                           // 获取row key
        } 
        
        System.out.println("======= 指定范围扫描 ========="); 
        Scan scan2 = new Scan("row2".getBytes());
        ResultScanner scaner2 = table.getScanner(scan2);
        for (Result result : scaner2) {
            System.out.println(new String(result.getRow()) + " : " + new String(result.getValue("col1".getBytes(), null)));                           // 获取row key
        } 
    }
}

输出结果：

1
2
3
4
5
6
7
======= 全表扫描 =========
ro43 : 4
row2 : 2
row3 : 3
======= 指定范围扫描 =========
row2 : 2
row3 : 3

###2.7 删除

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
public class DeleteOneRowAction {
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        Configuration conf = HBaseConfiguration.create();
        String tableName = "table2";
        HTable table = new HTable(conf, tableName);
        Delete delete = new Delete("ro43".getBytes());
        table.delete(delete);
    }
}

执行前

1
2
3
4
5
6
hbase(main):001:0> scan 'table2'
ROW                                           COLUMN+CELL                                      
 ro43                                         column=col1:, timestamp=1400594899049, value=4                                              
 row2                                         column=col1:, timestamp=1400593778072, value=2                                              
 row3                                         column=col1:, timestamp=1400594899049, value=3                                                                                      
3 row(s) in 0.2390 seconds

运行后

1
2
3
4
5
hbase(main):002:0> scan 'table2'
ROW                                           COLUMN+CELL                                         
 row2                                         column=col1:, timestamp=1400593778072, value=2                                              
 row3                                         column=col1:, timestamp=1400594899049, value=3                                                                                      
2 row(s) in 0.0680 seconds

###2.8 更新

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
public class UpdateOneRowAction {
    /**
     * @param args
     * @throws IOException 
     */
    public static void main(String[] args) throws IOException {
        Configuration conf = HBaseConfiguration.create();
        String tableName = "table2";
        HTable table = new HTable(conf, tableName);
        Put put = new Put("row2".getBytes());
        put.add("col1".getBytes(), null, "2_new".getBytes()); // HBase的更新其实就是put一条row key一样的记录，HBase根据记录的时间戳可以知道这是哪条是最新的记录
        table.put(put);
    }
}

更新前

1
2
3
4
5
hbase(main):002:0> scan 'table2'
ROW                                           COLUMN+CELL                                                                                                                         
 row2                                         column=col1:, timestamp=1400593778072, value=2                                                                                      
 row3                                         column=col1:, timestamp=1400594899049, value=3                                                                                      
2 row(s) in 0.0680 seconds

更新后

1
2
3
4
5
hbase(main):003:0> scan 'table2'
ROW                                           COLUMN+CELL                                                                                                                         
 row2                                         column=col1:, timestamp=1400598165614, value=2_new                                                                                  
 row3                                         column=col1:, timestamp=1400594899049, value=3                                                                                      
2 row(s) in 0.0860 seconds

HBase

HBase安装

HBase是一个分布式的、面向列的开源数据库

##下载后解压
我的机器名为hadoop2

##1. 单机版运行
使用本地文件系统，不需要启动HDFS，开发、调试方便

进入hbase的安装目录，修改`conf/hbase-site.xml`，加上一段：

1
2
3
4
<property>
   <name>hbase.rootdir</name>
   <value>file:///tmp/hbase</value>   #此处可替换成其他路径
 </property>

启动HBase

1
2
[root@hadoop2 hbase-0.96.2-hadoop2]# ./bin/start-hbase.sh 
starting master, logging to /usr/local/share/applications/hbase-0.96.2-hadoop2/bin/../logs/hbase-hduser-master-hadoop2.out

jps一下可以看到HMaster进程已经启动

1
2
3
4
5
6
7
[root@hadoop2 hbase-0.96.2-hadoop2]# jps
4961 DataNode
5130 SecondaryNameNode
6574 HMaster
5358 NodeManager
5260 ResourceManager
4872 NameNode

停止HBASE

1
2
[root@hadoop2 hbase-0.96.2-hadoop2]# stop-hbase.sh 
stopping hbase.........................

###1.1 HBase shell

进入shellhbase shell

1
2
3
4
5
6
7
[hduser@hadoop2 applications]$ hbase shell
2014-05-20 14:48:54,159 INFO  [main] Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 0.96.2-hadoop2, r1581096, Mon Mar 24 16:03:18 PDT 2014
hbase(main):001:0>

退出shellexit

1
exit

##2. 伪分布式运行
相关进程运行在同一台机器的不同JVM上，需要依赖HDFS，比较接近分布式运行，开发调试方便

###2.1 修改conf/hbase-site.xml
增加

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
<configuration>
    <property>
        <name>hbase.cluster.distributed</name>
        <value>true</value>
    </property>
    <property>
        <name>hbase.rootdir</name>
        <value>hdfs://hadoop2:9000/hbase</value>  <!-- hdfs://hadoop2:9000 是我本机hdfs的访问地址  -->
    </property>
    <!-- 进行仲裁的zookeeper服务器名称列表，以逗号分割，官方建议为基数个，另外，如果需要在客户端eclipse运行，该配置必须配置  -->
    <property>
        <name>hbase.zookeeper.quorum</name>
        <value>hadoop2</value>
    </property>
    <!-- zookeeper的datadir一定要配置，否则在使用hbase shell的时候会报错，官方文档没有很清楚地说明 -->
    <property>
        <name>hbase.zookeeper.property.datadir</name>
        <value>/home/hduser/zookeeper</value>
    </property>
</configuration>

启动start-hbase.sh，运行jps，可以看到HMaster、HRegionServer、HQuorumPeer三个进程在运行

1
2
3
4
5
6
7
8
9
10
11
12
[hduser@hadoop2 conf]$ jps
4961 DataNode
14155 Jps
5130 SecondaryNameNode
13918 HQuorumPeer
12369 Main
5358 NodeManager
13980 HMaster
5260 ResourceManager
9526 Main
14097 HRegionServer
4872 NameNode

启动hbase shell，手工创建一个表

1
2
3
4
hbase(main):002:0> create 'test1','c1'
0 row(s) in 1.1310 seconds
=> Hbase::Table - test1

查看hdfs

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[hduser@hadoop2 ~]$ hdfs dfs -ls -R /
drwxr-xr-x   - hduser supergroup          0 2014-05-19 16:39 /abc
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:11 /hbase
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/.tmp
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/.tmp/data
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/.tmp/data/default
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:11 /hbase/WALs
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/WALs/hadoop2,60020,1400577095283
-rw-r--r--   3 hduser supergroup          0 2014-05-20 17:11 /hbase/WALs/hadoop2,60020,1400577095283/hadoop2%2C60020%2C1400577095283.1400577104007
-rw-r--r--   3 hduser supergroup        393 2014-05-20 17:11 /hbase/WALs/hadoop2,60020,1400577095283/hadoop2%2C60020%2C1400577095283.1400577109953.meta
-rw-r--r--   3 hduser supergroup        180 2014-05-20 17:12 /hbase/WALs/hadoop2,60020,1400577095283/hadoop2%2C60020%2C1400577095283.1400577116884.meta
-rw-r--r--   3 hduser supergroup        363 2014-05-20 17:12 /hbase/WALs/hadoop2,60020,1400577095283/hadoop2%2C60020%2C1400577095283.1400577143759.meta
-rw-r--r--   3 hduser supergroup          0 2014-05-20 17:12 /hbase/WALs/hadoop2,60020,1400577095283/hadoop2%2C60020%2C1400577095283.1400577143916.meta
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:11 /hbase/corrupt
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:09 /hbase/data
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/data/default
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/data/default/test1
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/data/default/test1/.tabledesc
-rw-r--r--   1 hduser supergroup        283 2014-05-20 17:12 /hbase/data/default/test1/.tabledesc/.tableinfo.0000000001
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/data/default/test1/.tmp
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/data/default/test1/8bd0797eb991e9379975ad835c43ba20
-rwxr-xr-x   1 hduser supergroup         38 2014-05-20 17:12 /hbase/data/default/test1/8bd0797eb991e9379975ad835c43ba20/.regioninfo
drwxr-xr-x   - hduser supergroup          0 2014-05-20 17:12 /hbase/data/default/test1/8bd0797eb991e9379975ad835c43ba20/c1

###2.2 web管理界面
访问HMaster所在服务器的60010端口即可，我本机地址：http://hadoop2:60010/

##3. 升级

Hexo

Hexo配置

##1. 如何指定文章的categories和tags
修改markdown文件头部的内容，例如本文

1
2
3
4
title: Hexo配置
date: 2014-05-23 15:43:15
categories: Hexo
tags: [Hexo]       #多个tags的话，中间通过,分割

##2. 如何引用本地图片
将图片放到source目录下，按目录结构设置图片路径，如![写分发示意图](/img/UserGuide_image01.png)即可

Hadoop

Hadoop配置步骤

以下假设服务器上的hadoop用户为hduser

Hadoop版本为2.2.0

###1. JDK配置
下载并解压JDK的包，假设解压后目录为/usr/local/share/applications/jdk1.7.0_55，在hduser的用户配置文件增加JAVA_HOME环境变量

1
JAVA_HOME=/usr/local/share/applications/jdk1.7.0_55

###2. SSH配置
hduser运行以下命令

1
ssh-keygen -t rsa

会出现类似的界面，所有设置用默认值即可

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa): 
Created directory '/home/hduser/.ssh'.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is:
76:c6:77:0d:1a:0a:81:65:69:d9:b7:e7:7b:d8:a6:fc hduser@hadoop2
The key's randomart image is:
+--[ RSA 2048]----+
|       o++       |
|      ..+.. .    |
|       ..  ....  |
|         o ..o.o |
|        S = oo. .|
|       . o . ..  |
|               + |
|             .o +|
|              o=E|
+-----------------+

在/home/hduser/.ssh目录下会生成id_rsa和id_rsa.pub两个文件，如果需要真正多节点运行的话，需要将id_rsa.pub文件分发到每个节点中去，由于我这里采用伪分布式，所以把该文件的内容放到/home/hduser/.ssh目录下的一个名为authorized_keys的文件即可，可以采用以下命令

1
cp id_rsa.pub  authorized_keys  #注：使用cat id_rsa.pub > authorized_keys不行，具体原因没有深究

再利用ssh命令测试

1
ssh localhost

没有出现要求输入密码即为设置成功

###3. Hadoop配置文件配置
在hduser的用户配置文件增加HADOOP_HOME环境变量

1
HADOOP_HOME=/usr/local/share/applications/hadoop-2.2.0

修改$HADOOP_HOME/etc/hadoop/hadoop-env.sh，加上一段

1
JAVA_HOME=/usr/local/share/applications/jdk1.7.0_55

###4. 修改hosts文件
修改/etc/hosts文件，增加一行

1
127.0.0.1 hadoop2  #hadoop2是我本机的名字

###5. 格式化namenode
执行命令

1
$HADOOP_HOME/bin/hdfs namenode -format

###6. 运行Hadoop

1
2
cd $HADOOP_HOME/sbin
start-all.sh

正常执行完之后，执行jps命令，可以看到hadoop的相关进程

1
2
3
4
5
6
7
8
[hduser@hadoop2 bin]$ jps
1761 NodeManager
1667 ResourceManager
1293 NameNode
2041 Jps
1538 SecondaryNameNode
1378 DataNode
[hduser@hadoop2 bin]$

###7. 在windows客户端中使用eclipse访问hadoop
如果每次都在windows使用eclipse编辑java文件，再打成jar上传到hadoop的开发机上测试，实在太麻烦，而目前hadoop 2.2.0的eclipse插件官方也还没提供，虽然github里有这个插件的源码，可以下载下来编译，但实在是太懒不想去编译，于是尝试直接在eclipse里写一个类，然后以application的方式运行，代码很简单：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
package com.louz.hdfs;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class FileListAction {
    public static void main(String[] args) throws Exception {
        Configuration conf = new Configuration();
        FileSystem hdfs = FileSystem.get(conf);
        Path listf = new Path("/");
        FileStatus stats[] = hdfs.listStatus(listf);
        for (int i = 0; i < stats.length; ++i) {
            System.out.println(stats[i].getPath().toString());
        }
        hdfs.close();
    }
}

就是查看/目录下的文件列表，但中间也遇到了各种各样的问题。

由于是使用虚拟机部署的伪分布式hadoop，一开始使用virtualbox，无论怎么设置，都无法很好地访问到虚拟机里面的hadoop，后来转投vmware player，使用host-only模式，把虚拟机的防火墙关掉，轻易访问到22、50070、50075端口，但运行上面的程序时，报以下的错误：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
Exception in thread "main" java.net.ConnectException: Call From Louz-HP/192.168.154.1 to hadoop2:9000 failed on connection exception: java.net.ConnectException: Connection refused: no further information; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
    at java.lang.reflect.Constructor.newInstance(Unknown Source)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
    at org.apache.hadoop.ipc.Client.call(Client.java:1351)
    at org.apache.hadoop.ipc.Client.call(Client.java:1300)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    at $Proxy9.getListing(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
    at $Proxy9.getListing(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:482)
    at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1660)
    at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1643)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:640)
    at org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:92)
    at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:702)
    at org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:698)
    at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
    at org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:698)
    at com.louz.hdfs.FileListAction.main(FileListAction.java:15)
Caused by: java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:547)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:642)
    at org.apache.hadoop.ipc.Client$Connection.access$2600(Client.java:314)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1399)
    at org.apache.hadoop.ipc.Client.call(Client.java:1318)
    ... 20 more

在宿主机无论如何也telnet不通9000端口。折腾了几天，就快放弃的时候，看到网上有人说需要namenode机器上的hosts文件里要用具体的ip做映射，不能使用127.0.0.1，抱着死马当活马医的想法，修改/etc/hosts文件，使用本机ip进行映射：

1
192.168.154.129 hadoop2

重启hadoop，再次运行上面的程序，终于搞定！

1
2
hdfs://hadoop2:9000/abc
hdfs://hadoop2:9000/test

Linux

Linux常用命令

###1. 如何建用户

先增加用户

1
useradd <username> #默认方式，增加一个没有密码的用户

再初始化密码

1
passwd <username> #修改指定用户密码

###2. 挂载光盘

1
mount -t iso9660 源 目标

###3. 如何使用命令别名？

1
2
3
4
alias [别名]=[命令]
e.g.
alias l='ls -l'

###4. 关闭防火墙

1
/etc/init.d/iptables stop    #暂时关闭防火墙

###5. 更改目录所有者

1
chown <username> -R <dir>

###6. vi中全文替换字符串

1
:%s/fromStr/toString/g

###7. 启用某个网卡

1
ifconfig eth0 up   ## 假设要启用eth0

###8. 让某个网卡开机启用或network restart时能自动启用
修改 /etc/sysconfig/network-scripts/ifcfg-eth0 ## 假设需要修改eth0

1
ONBOOT=yes    # yes表示启动时启用， no表示启动时不启用

###9. 如何设定全局的变量
修改/etc/profile文件，增加

1
export VAR=val

然后source /etc/profile令其生效

###10. 单独赋权给用户/组/其他人

1
chmod <u|g|o><+|-><r|w|x> <filename>

Kindle DXG 美化教程

大纲

  1. kindle dxg存在问题
  2. 美化步骤
    2.1 升级
    2.2 越狱
    2.3 选择字体

Kindle

1
alert('Hello World!');

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque hendrerit lacus ut purus iaculis feugiat. Sed nec tempor elit, quis aliquam neque. Curabitur sed diam eget dolor fermentum semper at eu lorem. 

1
2
3
4
public class A {
    String a = "aaa";
    int i = 1;
}

asdf

aaaa
1
2
3
public class A {
	String a = "aaa";
}

11111
2222