环境安装

Doris 作为一款开源的 MPP 架构 OLAP 数据库,能够运行在绝大多数主流的商用服务器上。为了能够充分运用 MPP 架构的并发优势,以及 Doris 的高可用特性,我们建议 Doris 的部署遵循以下需求:

Linux 操作系统版本需求
Linux 系统版本
CentOS7.1 及以上
Ubuntu16.04 及以上

软件需求

软件版本
Java1.8 及以上
GCC4.8.2 及以上

操作系统安装要求

设置系统最大打开文件句柄数

vi /etc/security/limits.conf 
* soft nofile 65536
* hard nofile 65536

时钟同步

Doris 的元数据要求时间精度要小于5000ms,所以所有集群所有机器要进行时钟同步,避免因为时钟问题引发的元数据不一致导致服务出现异常。

关闭交换分区(swap)

Linux交换分区会给Doris带来很严重的性能问题,需要在安装之前禁用交换分区。
查看交换分区是否开启:

free -m

若开启则修改配置项,注释此行:

#/swapfile                                 none            swap    sw              0       0

然后reboot重启。

Liunx文件系统

这里我们推荐使用ext4文件系统,在安装操作系统的时候,请选择ext4文件系统。

开发测试环境

模块CPU内存磁盘网络实例数量
Frontend8核+8GB+SSD 或 SATA,10GB+ *千兆网卡1
Backend8核+16GB+SSD 或 SATA,50GB+ *千兆网卡1-3 *

生产环境

模块CPU内存磁盘网络实例数量(最低要求)
Frontend16核+64GB+SSD 或 RAID 卡,100GB+ *万兆网卡1-3 *
Backend16核+64GB+SSD 或 SATA,100G+ *万兆网卡3 *

java环境

命令行输入jps,若报错-bash: jps: command not found,则说明jdk需要重装。若已正常安装jdk,则跳过本节内容。
输入命令

whereis java


这只是java的执行路径,而非java的安装路径。
输入命令

which java


这个路径是java路径的软链接,获取该链接指向的地址:

ls -l /usr/bin/java


继续查询真实地址:

ls -l /etc/alternatives/java


发现路径是CentOS自带的openjdk,所以我们需要在oracle官网下载jdk1.8重新安装,首先卸载原有的openjdk:

rpm -qa | grep java #查看java文件
sudo rpm -qa | grep java | xargs rpm -e --nodeps #root权限下删除所有java文件

jdk下载官网:https://www.oracle.com/java/technologies/downloads/#java8-linux

下载最下面的版本,解压:

tar -zxvf jdk-8u351-linux-x64.tar.gz

linux目录下:
/usr/local 文件一般存放给另一个主机存放额外安装软件的目录,一般是通过编译源码方式安装的程序,如jdk、nginx、rabbitmq;
/opt 则是存放额外安装软件的目录,如mysql、redis、zookeeper;
/usr 用户的很多应用程序和文件都放在这个目录里面

配置java环境变量:

vim /etc/profile

在末尾追加:

export JAVA_HOME=/usr/local/jdk/jdk1.8.0_351  #jdk安装目录
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib:$CLASSPATH
export JAVA_PATH=${JAVA_HOME}/bin:${JRE_HOME}/bin
export PATH=$PATH:${JAVA_PATH}

刷新profile文件:

source /etc/profile

检查是否安装成功:

java -version
javac

输入命令行jps:

此时即为正常。

GCC环境

检查GCC是否安装,并保证版本在4.8.2 及以上:

a@a:~$ gcc

Command 'gcc' not found, but can be installed with:

sudo apt install gcc

a@a:~$ sudo apt install gcc   # 安装GCC

a@a:~$ gcc -v   # 查看GCC版本
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/9/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:hsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 9.4.0-1ubuntu1~20.04.1' --with-bugurl=file:///usr/share/doc/gcc-9/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,gm2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-9 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-9-Av3uEd/gcc-9-9.4.0/debian/tmp-nvptx/usr,hsa --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 9.4.0 (Ubuntu 9.4.0-1ubuntu1~20.04.1)

mysql安装

doris采用的mysql协议,需要通过mysql来连接使用,没有mysql的话需要先安装mysql:

mysql
sudo apt install mysql-client-core-8.0     # version 8.0.30-0ubuntu0.20.04.2, or
sudo apt install mariadb-client-core-10.3  # version 1:10.3.34-0ubuntu0.20.04.1

安装包下载


首先查询cpu型号:

cat /proc/cpuinfo


若cpu支持avx2,则直接下载avx2下的三个文件。
首先解压FE和BE, 然后第三个包java-udf-jar-with-dependencies.jar放到 apache-doris-be-1.2.1-bin-x86_64/lib 目录下。包位置放错,后续BE会启动不了。
设置环境变量

vim /etc/profile.d/doris.sh

FE

配置FE:

a@a:/opt/doris/apache-doris-fe-1.2.1-bin-x86_64/conf # pwd
a@a:/export/server/apache-doris-1.1.1-bin-x86/fe/conf # ll
# vim fe.conf
root@zhiyong-doris:/export/server/apache-doris-1.1.1-bin-x86/fe/conf# cat fe.conf
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

#####################################################################
## The uppercase properties are read and exported by bin/start_fe.sh.
## To see all Frontend configurations,
## see fe/src/org/apache/doris/common/Config.java
#####################################################################

# the output dir of stderr and stdout
LOG_DIR = ${DORIS_HOME}/log

DATE = `date +%Y%m%d-%H%M%S`
JAVA_OPTS="-Xmx8192m -XX:+UseMembar -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xloggc:$DORIS_HOME/log/fe.gc.log.$DATE"

# For jdk 9+, this JAVA_OPTS will be used as default JVM options
JAVA_OPTS_FOR_JDK_9="-Xmx8192m -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=7 -XX:+CMSClassUnloadingEnabled -XX:-CMSParallelRemarkEnabled -XX:CMSInitiatingOccupancyFraction=80 -XX:SoftRefLRUPolicyMSPerMB=0 -Xlog:gc*:$DORIS_HOME/log/fe.gc.log.$DATE:time"

##
## the lowercase properties are read by main program.
##

# INFO, WARN, ERROR, FATAL
sys_log_level = INFO

# store metadata, must be created before start FE.
# Default value is ${DORIS_HOME}/doris-meta
meta_dir = ${DORIS_HOME}/doris-meta

http_port = 8030
rpc_port = 9020
query_port = 9030
edit_log_port = 13536
mysql_service_nio_enabled = true

# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
# priority_networks = 10.10.10.0/24;192.168.0.0/16
priority_networks = 192.168.88.0/24

# Advanced configurations
# log_roll_size_mb = 1024
# sys_log_dir = ${DORIS_HOME}/log
# sys_log_roll_num = 10
# sys_log_verbose_modules = org.apache.doris
# audit_log_dir = ${DORIS_HOME}/log
# audit_log_modules = slow_query, query
# audit_log_roll_num = 10
# meta_delay_toleration_second = 10
# qe_max_connection = 1024
# max_conn_per_user = 100
# qe_query_timeout_second = 300
# qe_slow_log_ms = 5000

其中,edit_log_port的9010端口容易冲突,需注意。此外,${DORIS_HOME}/doris-meta目录需要提前创建好。

启动FE

FE的bin路径下执行:

./start_fe.sh --daemon  
jps # 查看FE是否启动,若有PaloFe,则说明FE启动成功

BE

修改BE配置文件夹,同理,修改BE的conf文件夹下的be.conf:

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#   http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied.  See the License for the
# specific language governing permissions and limitations
# under the License.

PPROF_TMPDIR="$DORIS_HOME/log/"

# INFO, WARNING, ERROR, FATAL
sys_log_level = INFO

# ports for admin, web, heartbeat service
be_port = 9060
webserver_port = 13535
heartbeat_service_port = 9050
brpc_port = 8060

# Choose one if there are more than one ip except loopback address.
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
# priority_networks = 10.10.10.0/24;192.168.0.0/16
priority_networks = 192.168.88.0/24

# data root path, separate by ';'
# you can specify the storage medium of each root path, HDD or SSD
# you can add capacity limit at the end of each root path, seperate by ','
# eg:
# storage_root_path = /home/disk1/doris.HDD,50;/home/disk2/doris.SSD,1;/home/disk2/doris
# /home/disk1/doris.HDD, capacity limit is 50GB, HDD;
# /home/disk2/doris.SSD, capacity limit is 1GB, SSD;
# /home/disk2/doris, capacity limit is disk capacity, HDD(default)
#
# you also can specify the properties by setting '<property>:<value>', seperate by ','
# property 'medium' has a higher priority than the extension of path
#
# Default value is ${DORIS_HOME}/storage, you should create it by hand.
# storage_root_path = ${DORIS_HOME}/storage
storage_root_path = /opt/doris/storage

# Advanced configurations
# sys_log_dir = ${DORIS_HOME}/log
# sys_log_roll_mode = SIZE-MB-1024
# sys_log_roll_num = 10
# sys_log_verbose_modules = *
# log_buffer_level = -1
# palo_cgroups

提前建好storage文件夹目录。
如果 be 部署在 hadoop 集群中,注意调整 be.conf 中的 webserver_port。

在 FE 中添加所有 BE 节点

连接doris:

mysql -h 192.168.88.21 -P 9030 -uroot
mysql> show databases;
mysql> use information_schema;
mysql> show tables;
mysql> ALTER SYSTEM ADD BACKEND "192.168.88.21:9050";

启动BE

BE的bin目录下输入:

./start_be.sh --daemon

查看BE状态:

mysql> SHOW PROC '/backends';

在这里插入图片描述

若Alive项为true则说明BE启动成功,若为false则说明启动失败。
启动失败可以检查doris-meta和storage文件夹是否提前建立,端口是否冲突,一般错误都是这两个原因引起的,错误往往是由于填写的配置信息有问题。

Logo

旨在为数千万中国开发者提供一个无缝且高效的云端环境,以支持学习、使用和贡献开源项目。

更多推荐