colben/www.colben.cn
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
www.colben.cn/content/post/cloudera-manager.md
colben 1e344dc204 update
2021-11-14 15:52:46 +08:00

8.5 KiB
Raw Permalink Blame History

title, date, lastmod, keywords, tags, categories
title date lastmod keywords tags categories
CentOS7 安装 Cloudera Manager 2020-08-16T13:48:37+08:00 2020-08-16T18:27:00+08:00
cloudera cdh
hadoop

环境

角色 IP 主机名 服务
Utility 192.168.1.100 cm0.colben.cn ClouderaManager
ClouderaManagerManagementService
HiveMetastore
Gateway 192.168.1.101 gw0.colben.cn GatewayConfiguration
HiveServer2
Zookeeper
Master 192.168.1.102 m0.colben.cn NameNode
JournalNode
FailoverController
YarnResourceManager
Zookeeper
Master 192.168.1.103 m1.colben.cn NameNode
JournalNode
FailoverController
YarnResourceManager
Zookeeper
Worker 192.168.1.104 w0.colben.cn DataNode
NodeManager
Worker 192.168.1.105 w0.colben.cn DataNode
NodeManager
Worker 192.168.1.106 w0.colben.cn DataNode
NodeManager

配置 ssh 免密登陆

  • 在 cm0 上配置 ssh 可免密登陆全部服务器

    ssh-keygen
seq -f'192.168.1.%g' 100 106 | xargs -L1 ssh-copy-id

  • 这里的私钥会在后面通过界面增加主机时用到

关闭防火墙和 selinux

  • 在全部服务器上关闭防火墙

    systemctl stop firewalld
systemctl disable firewalld

  • 在全部服务器上关闭 selinux

    setenforce 0
sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config

配置网络名称

  • 在全部服务器上修改 /etc/hosts，增加如下解析记录

    192.168.1.100    cm0.colben.cn cm0
192.168.1.101    gw0.colben.cn gw0
192.168.1.102    m0.colben.cn m0
192.168.1.103    m1.colben.cn m1
192.168.1.104    w0.colben.cn w0
192.168.1.105    w1.colben.cn w1
192.168.1.106    w2.colben.cn w2

  • 在全部服务器上修改 /etc/sysconfig/network，增加各自主机名设置

    HOSTNAME=XXXX.colben.cn

  • 在全部服务器上设置各自主机名

    hostnamectl set-hostname XXXX.colben.cn

配置时间同步

  • 在 cm0 上配置修改 /etc/chrony.conf

    server ntp.aliyun.com iburst
allow 192.168.1.0/24

  • 在其他服务器上修改 /etc/chrony.conf

    server cm0.colben.cn iburst

  • 在全部服务器上重启 chronyd 服务

    systemctl restart chronyd

配置 cloudera manager 内网安装源

  • 在 cm0 上安装并启动 httpd 服务

    yum install httpd
systemctl enable httpd
systemctl start httpd

  • 在 cm0 上下载 cloudera manager yum 仓库

    mkdir -p /var/www/html/cloudera-repos/cm6/6.3.1
wget https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz
tar zxf cm6.3.1-redhat7.tar.gz -C /var/www/html/cloudera-repos/cm6/6.3.1 --strip-components=1
chmod -R ugo+rX /var/www/html/cloudera-repos/cm6

  • 在 cm0 上下载 cdh yum 仓库(体积较小，需手动升级)

    mkdir -p /var/www/html/cloudera-repos
wget --recursive --no-parent --no-host-directories \
    https://archive.cloudera.com/cdh6/6.3.2/redhat7/ -P /var/www/html/cloudera-repos
wget --recursive --no-parent --no-host-directories \
    https://archive.cloudera.com/gplextras6/6.3.2/redhat7/ -P /var/www/html/cloudera-repos
chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

  • 在 cm0 上下载 cdh parcel 仓库(体积较大，可通过界面操作自动升级)

    mkdir -p /var/www/html/cloudera-repos
wget --recursive --no-parent --no-host-directories \
    https://archive.cloudera.com/cdh6/6.3.2/parcels/ -P /var/www/html/cloudera-repos
wget --recursive --no-parent --no-host-directories \
    https://archive.cloudera.com/gplextras6/6.3.2/parcels/ -P /var/www/html/cloudera-repos
chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6
chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

  • 在 cm0 上创建 cloudera.repo，并重建 yum 缓存

    cat > /etc/yum.repos.d/cloudera.repo <<-EOF
[cloudera-manager]
name=cloudera-manager
baseurl=http://cm0.colben.cn/cloudera-repos/cm6/
enabled=1
gpgcheck=0
EOF
yum clean all
yum makecache fast

  • 通过界面增加其他主机时，cloudera manager 会自动部署该 repo 文件

安装 jdk

  • 在 cm0 上安装 jdk

    yum install oracle-j2sdk1.8

  • 通过界面增加其他主机时，可自动安装 oracle-j2sdk1.8

安装 cloudera manager

  • 在 cm0 上安装 cloudera-manager 
    yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

配置 cloudera manager 数据库

  • 在 cm0 上安装 mysql，参考这里

  • 官方推荐 my.cnf

    [mysqld]
transaction-isolation = READ-COMMITTED
symbolic-links = 0
sql_mode=STRICT_ALL_TABLES

key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 600

log_bin=/var/lib/mysql/mysql_binary_log
server_id=1
binlog_format = mixed

read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M

innodb_file_per_table = 1
innodb_flush_log_at_trx_commit  = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M

  • 在 cm0 上启动 mysql

    systemctl enable mysqld
systemctl start mysqld

  • 在 cm0 上的 mysql 中创建数据库和用户

    CREATE DATABASE scm       DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE amon      DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE rman      DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE hue       DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE metastore DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE sentry    DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE nav       DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE navms     DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
CREATE DATABASE oozie     DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci;
GRANT ALL ON scm.*       to scm@'%'    identified by 'Pass-1234';
GRANT ALL ON amon.*      to amon@'%'   identified by 'Pass-1234'; 
GRANT ALL ON rman.*      to rman@'%'   identified by 'Pass-1234';
GRANT ALL ON hue.*       to hue@'%'    identified by 'Pass-1234';
GRANT ALL ON metastore.* to hive@'%'   identified by 'Pass-1234';
GRANT ALL ON sentry.*    to sentry@'%' identified by 'Pass-1234';
GRANT ALL ON nav.*       to nav@'%'    identified by 'Pass-1234';
GRANT ALL ON navms.*     to navms@'%'  identified by 'Pass-1234';
GRANT ALL ON oozie.*     to oozie@'%'  identified by 'Pass-1234';

  • 在 cm0 上初始化数据库

    /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm       scm
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon      amon
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman      rman
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue       hue
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql metastore hive
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry    sentry
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav       nav
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms     navms
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie     oozie

  • 在全部服务器上安装 mysql connector

    wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
tar zxvf mysql-connector-java-5.1.46.tar.gz
mkdir -p /usr/share/java/
cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar \
    /usr/share/java/mysql-connector-java.jar

启动 cloudera manager

  • 在 cm0 上启动 cloudera manager

    systemctl start cloudera-scm-server

  • 日志: /var/log/cloudera-scm-server/cloudera-scm-server.log

  • 日志出现如下信息，表示 cloudera-scm-server 启动完成

    INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

  • 浏览器访问