技术分享 | 使用 ANTMAN 工具替换 OceanBase 云平台节点

发布于

作者：张瑞远

上海某公司 DBA，曾经从事银行、证券数仓设计、开发、优化类工作，现主要从事电信级 IT 系统及数据库工作。有三年以上 OceanBase 工作经验，参与某通信行业核心数据库全面国产化工作。获得的专业技能与认证包括 OceanBase OBCP、Oracle OCP 11g、OracleOCM 11g 、MySQL OCP 5.7 、腾讯云TBase、腾讯云 TDSQL、阿里云 ACP 、KingBase KCP。

本文来源：原创投稿

* 爱可生开源社区出品，原创内容未经授权不得随意使用，转载请联系小编并注明来源。

1前言

OceanBase 云平台（OceanBase Cloud Platform，OCP），是以 OceanBase 为核心的企业级数据库管理平台。

我们生产环境基本都是需要先创建 OCP 平台，然后依赖 OCP 去创建及管理监控生产集群，所以安装 OCP 一般是系统上线的第一步。之后可能随着机房规划等问题，就会有需要搬迁或者替换 OCP 的机器的需求。

分别介绍两种 OCP 节点的替换方法。一种是使用 OAT 平台（OceanBase Admin Toolkit，管理者工具）来替换；另一种就是使用 ANTMAN 工具替换。上次我们介绍了第一种 OAT 工具的方案，本文介绍第二种。

PS：我的环境的 OCP 负载均衡使用的 F5，所以新的机器需要先配置 F5，其他负载均衡场景同理。

2环境背景

大家如果有接触 OB 生产环境的经验的话，可能会了解，前期版本在安装 OCP 的时候，需要安装 OCP 软件/metadb/obproxy 三个 Docker 包，后期 OCP 版本将 DB+Proxy 集成在了一个 Docker 包里，OAT 的话只能纳管 DB + Proxy。集成的 metadb，分开的情况还需要使用 ANTMAN 工具来替换。

软件信息

软件

版本

OCP

ocp-all-in-one:3.3.3-20220906114643

metadb+proxy

OB2277_OBP320_x86_20220429

Proxy

4.1.1_20230519_x86

antman

t-oceanbase-antman-1.4.3-20220807073355.alios7.x86_64

3操作过程

3.1 环境检查/准备

检查替换机器环境，包括分盘，创建 admin 用户，安装 Docker 软件等。安装好后检查下。

代码语言：javascript
复制
cd /root/t-oceanbase-antman/clonescripts/
sh precheck.sh -m ocp

登录 meta 库检查有没有 tenant 的主 ZONE 在要被替换的节点，提前切主。

代码语言：javascript
复制
MySQL [oceanbase]> select * from __all_Server;
+----------------------------+----------------------------+--------------+----------+----+----------------+------------+-----------------+--------+-----------------------+--------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| gmt_create                 | gmt_modified               | svr_ip       | svr_port | id | zone           | inner_port | with_rootserver | status | block_migrate_in_time | build_version                                                                        | stop_time | start_service_time | first_sessid | with_partition | last_offline_time |
+----------------------------+----------------------------+--------------+----------+----+----------------+------------+-----------------+--------+-----------------------+--------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
| 2022-03-17 22:59:19.979627 | 2023-03-20 10:27:01.147283 | 10.10.100.87 |     2882 |  6 | META_OB_ZONE_2 |       2881 |               0 | active |                     0 | 2.2.76_20210406232249-a1e144bdc179fbf473cea37f199e8a76c736b8d4(Apr  6 2021 23:55:12) |         0 |   1679279220991796 |            0 |              1 |  1679278517144838 |
| 2022-03-17 23:54:49.277939 | 2023-03-20 09:29:22.079578 | 10.10.100.9  |     2882 |  7 | META_OB_ZONE_1 |       2881 |               1 | active |                     0 | 2.2.76_20210406232249-a1e144bdc179fbf473cea37f199e8a76c736b8d4(Apr  6 2021 23:55:12) |         0 |   1679275725595691 |            0 |              1 |                 0 |
| 2021-12-21 22:44:16.476503 | 2023-03-20 09:29:22.080425 | 122.44.11.2  |     2882 |  5 | META_OB_ZONE_3 |       2881 |               0 | active |                     0 | 2.2.76_20210406232249-a1e144bdc179fbf473cea37f199e8a76c736b8d4(Apr  6 2021 23:55:12) |         0 |   1640097866698859 |            0 |              1 |                 0 |
+----------------------------+----------------------------+--------------+----------+----+----------------+------------+-----------------+--------+-----------------------+--------------------------------------------------------------------------------------+-----------+--------------------+--------------+----------------+-------------------+
MySQL [oceanbase]> select tenant_name,primary_zone  from __all_tenant;
+----------------+----------------------------------------------+
| tenant_name    | primary_zone                                 |
+----------------+----------------------------------------------+
| sys            | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 |
| ocp_meta       | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 |
| ocp_monitor    | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 |
| oms_tt_tenant  | META_OB_ZONE_1;META_OB_ZONE_3;META_OB_ZONE_2 |
| oms_cc7_tenant | META_OB_ZONE_3;META_OB_ZONE_2;META_OB_ZONE_1 |
| oms_ff9_tenant | META_OB_ZONE_1;META_OB_ZONE_2,META_OB_ZONE_3 |
| oms_cc9_tenant | META_OB_ZONE_3;META_OB_ZONE_1,META_OB_ZONE_2 |
| oms_dd_tenant  | META_OB_ZONE_3;META_OB_ZONE_1,META_OB_ZONE_2 |
| obdw_meta      | META_OB_ZONE_3;META_OB_ZONE_1,META_OB_ZONE_2 |
+----------------+----------------------------------------------+
MySQL [oceanbase]> alter tenant sys primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';
Query OK, 0 rows affected (0.04 sec)

MySQL [oceanbase]> alter tenant ocp_meta primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';

Query OK, 0 rows affected (1.27 sec)


MySQL [oceanbase]> alter tenant ocp_monitor primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';

Query OK, 0 rows affected (0.02 sec)


MySQL [oceanbase]> alter tenant oms_tt_tenant primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';

Query OK, 0 rows affected (0.03 sec)


MySQL [oceanbase]> alter tenant oms_ff9_tenant primary_zone='META_OB_ZONE_2;META_OB_ZONE_3,META_OB_ZONE_1';

Query OK, 0 rows affected (0.03 sec)

因为使用 ANTMAN 工具迁移，需要在执行机器上修改 obcluster.conf 文件，或者直接从原 OCP 上 copy 后检查下，镜像包也需要传到该机器 /root/t-oceanbase-antman 目录下。

代码语言：javascript
复制
55obffocp:~/t-oceanbase-antman # cat obcluster.conf

ZONE1_RS_IP=10.10.100.9

ZONE2_RS_IP=10.10.100.87

ZONE3_RS_IP=122.44.11.2


自动配置，无需修改 / AUTO-CONFIGURATION


OBSERVER01_HOSTNAME=OCP_META_SERVER_1

OBSERVER02_HOSTNAME=OCP_META_SERVER_2

OBSERVER03_HOSTNAME=OCP_META_SERVER_3

ZONE1_NAME=META_OB_ZONE_1                       --后续命令参数，主要和参数文件中对上

ZONE2_NAME=META_OB_ZONE_2

ZONE3_NAME=META_OB_ZONE_3

##there must be more than half zone within same region

ZONE1_REGION=OCP_META_REGION

ZONE2_REGION=OCP_META_REGION

ZONE3_REGION=OCP_META_REGION

MYSQL_PORT=2881

RPC_PORT=2882


OCP_VERSION=3.3.3

检查执行 ANTMAN 脚本机器上默认集群密码是否正确。cd /t-oceanbase-antman/tools，执行 getpass.sh 的脚本，如果不对需要使用 setpass.sh 修改，因为后续 Proxy 的 Docker 迁移后会有验证，OCP 的 Docker 迁移前也会验证。

代码语言：javascript
复制
55obffocp:/t-oceanbase-antman/tools # bash setpass.sh -s 0Aa255yK^F

password file sys in /root/.key already exist!




Password of root@sys is  CqVgg9}Aut

Password of root@ocp_meta is  r6kS^EINTU

Password of root@ocp_monitor is  pkJv1a{7J7

Password of root@odc is  j{fjdd3X9f

Password of root@oms is  {oOIsE9fdQ


55obffocp:~ # mv .key  .key_bak

55obffocp:~ # cd /root/t-oceanbase-antman/tools/

55obffocp:~/t-oceanbase-antman/tools # bash setpass.sh -s 0Aa255yK^F




Password of root@sys is  0Aa255yK^F

Password of root@ocp_meta is

Password of root@ocp_monitor is

Password of root@odc is

Password of root@oms is

55obffocp:~/t-oceanbase-antman/tools # bash setpass.sh -c rSf@jO%6EO




Password of root@sys is  0Aa255yK^F

Password of root@ocp_meta is  rSf@jO%6EO

Password of root@ocp_monitor is

Password of root@odc is

Password of root@oms is

3.2 添加新机器

执行 ANTMAN 的 manage 脚本进行新机器的添加。

PS：这个版本 manage 会有报错，文末会有分享。

代码语言：javascript
复制
55obffocp:~/t-oceanbase-antman # ./manage.sh -i ob,ocp,obproxy -l 133.55.22.19 -z 1 -R Jnydzycscc@123 -A OceanBase#123

[2023-06-16 16:31:45.375633] INFO [check conf file /root/t-oceanbase-antman/obcluster.conf format ...]

[2023-06-16 16:31:45.381844] INFO [conf file is upper case format.]

[2023-06-16 16:31:45.391446] INFO [SSH_AUTH=password SSH_USER=root SSH_PORT=22 SSH_PASSWORD= SSH_KEY_FILE=/root/.ssh/id_rsa]

LB_MODE=f5

INSTALL_COMPONENTS componets: ob obproxy ocp

CLEAR_COMPONENTS:

IP_LIST: 133.55.22.19

ZONE_LIST: 1

ROOT_PASSWORD_LIST: Jnydzycscc@123

ADMIN_PASSWORD_LIST: OceanBase#123

[2023-06-16 16:31:45.746503] INFO [INSTALL_COMPONENT: ob START  ######################################]

[2023-06-16 16:31:45.751057] INFO [deploy_ob: check whether OBSERVER port 2881,2882 are in use or not on 133.55.22.19]

[2023-06-16 16:31:45.806500] INFO [deploy_ob: OBSERVER port 2881,2882 are idle on 133.55.22.19]

[2023-06-16 16:31:45.810773] INFO [deploy_ob: installing ob cluster, logfile: /root/t-oceanbase-antman/logs/deploy_ob.log]

cp: '/root/t-oceanbase-antman/OB2276_x86_20210409.tar.gz' and '/root/t-oceanbase-antman/OB2276_x86_20210409.tar.gz' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/install_OB_docker.sh' and '/root/t-oceanbase-antman/install_OB_docker.sh' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same file

skip copy same file

cp: '/root/.key' and '/root/.key' are the same file

skip copy same file

nohup: ignoring input

[2023-06-16 16:31:45.841348] INFO [installing OB docker and starting OB server on 133.55.22.19, pid: 144513, log: /root/t-oceanbase-antman/logs/install_OB_docker.log and /home/admin/logs/ob-server/ inside docker]

[2023-06-16 16:31:45.925592] INFO [load docker image: docker load -i /root/t-oceanbase-antman/OB2276_x86_20210409.tar.gz]

[2023-06-16 16:31:45.930723] INFO [install_OB_docker.sh is still running on 133.55.22.19]

[2023-06-16 16:31:56.021465] INFO [install_OB_docker.sh is still running on 133.55.22.19]

Loaded image: reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409

[2023-06-16 16:32:06.111458] INFO [install_OB_docker.sh is still running on 133.55.22.19]

[2023-06-16 16:32:06.359285] INFO [start container: docker run -d -it --cap-add SYS_RESOURCE --name META_OB_ZONE_1 --net=host     -e OBCLUSTER_NAME=obcluster      -e DEV_NAME=bond0     -e ROOTSERVICE_LIST="10.10.100.9:2882:2881;10.10.100.87:2882:2881;122.44.11.2:2882:2881"     -e DATAFILE_DISK_PERCENTAGE=90     -e CLUSTER_ID=1632654636     -e ZONE_NAME=META_OB_ZONE_1     -e OBPROXY_PORT=2883     -e MYSQL_PORT=2881     -e RPC_PORT=2882     -e OCP_VIP=134.80.173.57     -e OCP_VPORT=80     -e app.password_root='Jnydzycscc@123'     -e app.password_admin='OceanBase#123'     -e OBPROXY_OPTSTR=""     -e OPTSTR="cpu_count=64,system_memory=50G,memory_limit=254G,__min_full_resource_pool_memory=1073741824,_ob_enable_prepared_statement=false,memory_limit_percentage=90"     --cpu-period 100000     --cpu-quota 6400000     --cpuset-cpus 0-63     --memory 256G     -v /home/admin/oceanbase:/home/admin/oceanbase     -v /data/log1:/data/log1     -v /data/1:/data/1     --restart on-failure:5     reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409]

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.

4f1c15e8194cc1ae2fffcc124ea9c982b3fda87ce1a6d0038db88435c737af89

[2023-06-16 16:32:16.209761] INFO [install_OB_docker.sh finished and reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409 started on 133.55.22.19]

[2023-06-16 16:32:16.214771] INFO [waiting on observer ready on 133.55.22.19]

[2023-06-16 16:35:16.244133] INFO [waiting on observer ready on 133.55.22.19 for 3 Minitues]

[2023-06-16 16:36:16.264776] INFO [waiting on observer ready on 133.55.22.19 for 4 Minitues]

[2023-06-16 16:37:16.285808] INFO [waiting on observer ready on 133.55.22.19 for 5 Minitues]

[2023-06-16 16:37:16.579057] INFO [observer on 133.55.22.19 is ready]

[2023-06-16 16:37:16.584583] INFO [deploy_ob: installation of ob cluster done]

[2023-06-16 16:37:16.588617] INFO [INSTALL_COMPONENT: ob DONE  ######################################]

[2023-06-16 16:37:16.593604] INFO [INSTALL_COMPONENT: obproxy START  ######################################]

这里日志太多，就不都粘贴出来了，可以从上面看到 metadb 的 Docker 服务添加完后开始了 OBProxy 的 Docker 服务添加。

过程说明

  1. 133.55.22.19 是要去替换 OCP 的服务器的实际物理 IP。
  2. -z 1 选项，指定的 133.55.22.19 会被添加到 OCP 环境中的第 1 个 ZONE，即和上文查到的 10.10.100.9 机器在同一个 ZONE 里。
    这里关于 ZONE 的定义，主要是针对 OCP 服务器上的 meta_ob docker 而言，obproxy docker 和 ocp docker 并没有 ZONE 的概念。
    关于每台 OCP 服务器上的 meta_ob docker 属于哪一个 ZONE，请参考 obcluster.conf 配置文件中的三个变量：
    • ZONE1_RS_IP
    • ZONE2_RS_IP
    • ZONE3_RS_IP
  3. -R 和 -A，后面需要分别填写成 133.55.22.19 服务器的 root 用户密码和 admin 用户密码。
  4. -i 是安装，如果替换成 -c 就是清除。

这时候正常的话可以通过新添加节点的 IP:8080 前台登录 OCP，也可以通过这个机器的 2883 端口去连 meta 库了。

3.3 新增租户

登录 OCP 的 metadb 的 sys 租户新增 meta_ob Docker 的上线。

代码语言：javascript
复制
MySQL [oceanbase]> alter system add server '133.55.22.19:2882' zone 'META_OB_ZONE_1';

Query OK, 0 rows affected (0.02 sec)


MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;

+--------------+----------------+-----------------+--------+--------------------+

| svr_ip       | zone           | with_rootserver | status | start_service_time |

+--------------+----------------+-----------------+--------+--------------------+

| 10.10.100.87 | META_OB_ZONE_2 |               1 | active |   1679279220991796 |

| 10.10.100.9  | META_OB_ZONE_1 |               0 | active |   1679275725595691 |

| 122.44.11.2  | META_OB_ZONE_3 |               0 | active |   1640097866698859 |

| 133.55.22.19 | META_OB_ZONE_1 |               0 | active |                  0 |

+--------------+----------------+-----------------+--------+--------------------+

4 rows in set (0.00 sec)


MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;

+--------------+----------------+-----------------+--------+--------------------+

| svr_ip       | zone           | with_rootserver | status | start_service_time |

+--------------+----------------+-----------------+--------+--------------------+

| 10.10.100.87 | META_OB_ZONE_2 |               1 | active |   1679279220991796 |

| 10.10.100.9  | META_OB_ZONE_1 |               0 | active |   1679275725595691 |

| 122.44.11.2  | META_OB_ZONE_3 |               0 | active |   1640097866698859 |

| 133.55.22.19 | META_OB_ZONE_1 |               0 | active |   1686908200404755 |

+--------------+----------------+-----------------+--------+--------------------+

4 rows in set (0.01 sec)

3.4 替换下线

登录 OCP 的 metadb 的 sys 租户将被替换 meta_ob Docker 的下线。

代码语言：javascript
复制
MySQL [oceanbase]>  alter system delete server '10.10.100.9:2882' zone 'META_OB_ZONE_1';

Query OK, 0 rows affected (0.19 sec)


MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;

+--------------+----------------+-----------------+----------+--------------------+

| svr_ip       | zone           | with_rootserver | status   | start_service_time |

+--------------+----------------+-----------------+----------+--------------------+

| 10.10.100.87 | META_OB_ZONE_2 |               1 | active   |   1679279220991796 |

| 10.10.100.9  | META_OB_ZONE_1 |               0 | deleting |   1679275725595691 |

| 122.44.11.2  | META_OB_ZONE_3 |               0 | active   |   1640097866698859 |

| 133.55.22.19 | META_OB_ZONE_1 |               0 | active   |   1686908200404755 |

+--------------+----------------+-----------------+----------+--------------------+

4 rows in set (0.01 sec)

MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;

+--------------+----------------+-----------------+--------+--------------------+

| svr_ip       | zone           | with_rootserver | status | start_service_time |

+--------------+----------------+-----------------+--------+--------------------+

| 10.10.100.87 | META_OB_ZONE_2 |               1 | active |   1679279220991796 |

| 122.44.11.2  | META_OB_ZONE_3 |               0 | active |   1640097866698859 |

| 133.55.22.19 | META_OB_ZONE_1 |               0 | active |   1686908200404755 |

+--------------+----------------+-----------------+--------+--------------------+

3.5 更新服务器信息

登录 ocp_meta 租户，手工更新 OCP 服务器信息。

前面步骤处理完，OCP 前台还可以看到残留的信息，需要替换下信息。

代码语言：javascript
复制
55obffocp:~/t-oceanbase-antman # mysql -h10.10.100.87 -P2883 -uroot@ocp_meta#obcluster -p'rSf@jO%6EO' -Docp -c


MySQL [ocp]>  select * from compute_host where inner_ip_address='10.10.100.9'\G

*************************** 1. row ***************************

id: 1

name: ocp1a

description: NULL

operating_system: 4.12.14-120-default

architecture: x86_64

inner_ip_address: 10.10.100.9

ssh_port: 2022

kind: DEDICATED_PHYSICAL_MACHINE

publish_ports: NULL

status: ONLINE

vpc_id: 1

idc_id: 1

host_type_id: 1

serial_number: NULL

alias: NULL

create_time: 2021-09-26 21:04:11

update_time: 2023-03-20 11:01:58

1 row in set (0.00 sec)


MySQL [ocp]> update compute_host set inner_ip_address='133.55.22.19', name='55obffocp' where inner_ip_address='10.10.100.9';

Query OK, 1 row affected (0.01 sec)

Rows matched: 1  Changed: 1  Warnings: 0


MySQL [ocp]> select * from compute_host where id =1;

+----+-----------+-------------+---------------------+--------------+------------------+----------+----------------------------+---------------+--------+--------+--------+--------------+---------------+-------+---------------------+---------------------+

| id | name      | description | operating_system    | architecture | inner_ip_address | ssh_port | kind                       | publish_ports | status | vpc_id | idc_id | host_type_id | serial_number | alias | create_time         | update_time         |

+----+-----------+-------------+---------------------+--------------+------------------+----------+----------------------------+---------------+--------+--------+--------+--------------+---------------+-------+---------------------+---------------------+

|  1 | 55obffocp | NULL        | 4.12.14-120-default | x86_64       | 133.55.22.19     |     2022 | DEDICATED_PHYSICAL_MACHINE | NULL          | ONLINE |      1 |      1 |            1 | NULL          | NULL  | 2021-09-26 21:04:11 | 2023-06-16 17:47:19 |

+----+-----------+-------------+---------------------+--------------+------------------+----------+----------------------------+---------------+--------+--------+--------+--------------+---------------+-------+---------------------+---------------------+

1 row in set (0.00 sec)

3.6 清理被替换机器上残留的服务

代码语言：javascript
复制
ocp1a:~/t-oceanbase-antman #  ./manage.sh -c ob,ocp,obproxy -l 10.10.100.9 -z 1 -R 'Dt!n(Rg4Av!t' -A OceanBase#123

grep: /etc/system-release: No such file or directory

[2023-06-16 22:45:44.101400] INFO [check conf file /root/t-oceanbase-antman/obcluster.conf format ...]

[2023-06-16 22:45:44.106779] INFO [conf file is upper case format.]

[2023-06-16 22:45:44.114437] INFO [SSH_AUTH=password SSH_USER=root SSH_PORT=22 SSH_PASSWORD= SSH_KEY_FILE=/root/.ssh/id_rsa]

LB_MODE=f5

INSTALL_COMPONENTS componets:

CLEAR_COMPONENTS: ob obproxy ocp

IP_LIST: 10.10.100.9

ZONE_LIST: 1

ROOT_PASSWORD_LIST: Dt!n(Rg4Av!t

ADMIN_PASSWORD_LIST: OceanBase#123

[2023-06-16 22:45:44.474268] INFO [CLEAR_COMPONENT: ob START  ######################################]

cp: '/root/t-oceanbase-antman/uninstall.sh' and '/root/t-oceanbase-antman/uninstall.sh' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same file

skip copy same file

grep: /etc/system-release: No such file or directory

[2023-06-16 22:45:44.504069] INFO [remove OB server and docker on host: 10.10.100.9]

[2023-06-16 22:45:44.548697] INFO [docker rm -f 62ab623cb4ed]

62ab623cb4ed

[2023-06-16 22:46:01.260706] INFO [remove OB server and docker on host: 10.10.100.9 done!]

[2023-06-16 22:46:01.370808] INFO [uninstall.sh ob  finished and reg.docker.alibaba-inc.com/antman/ob-docker:OB2276_x86_20210409 removed on 10.10.100.9]

[2023-06-16 22:46:01.375914] INFO [OB docker on 10.10.100.9 is removed]

[2023-06-16 22:46:01.380667] INFO [CLEAR_COMPONENT: ob DONE  ######################################]

[2023-06-16 22:46:01.385398] INFO [CLEAR_COMPONENT: obproxy START  ######################################]

cp: '/root/t-oceanbase-antman/uninstall.sh' and '/root/t-oceanbase-antman/uninstall.sh' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same file

skip copy same file

grep: /etc/system-release: No such file or directory

[2023-06-16 22:46:01.416495] INFO [remove obproxy docker on host:10.10.100.9]

[2023-06-16 22:46:01.514215] INFO [docker rm -f 01bdcadf2e11]

01bdcadf2e11

[2023-06-16 22:46:01.765459] INFO [remove obproxy docker on host:10.10.100.9 done!]

[2023-06-16 22:46:01.858848] INFO [uninstall.sh obproxy  finished and reg.docker.alibaba-inc.com/antman/obproxy:OBP186_20210315 removed on 10.10.100.9]

[2023-06-16 22:46:01.863806] INFO [obproxy docker on 10.10.100.9 is removed]

[2023-06-16 22:46:01.868778] INFO [CLEAR_COMPONENT: obproxy DONE  ######################################]

[2023-06-16 22:46:01.873368] INFO [CLEAR_COMPONENT: ocp START  ######################################]

cp: '/root/t-oceanbase-antman/uninstall.sh' and '/root/t-oceanbase-antman/uninstall.sh' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/obcluster.conf' and '/root/t-oceanbase-antman/obcluster.conf' are the same file

skip copy same file

cp: '/root/t-oceanbase-antman/common/utils.sh' and '/root/t-oceanbase-antman/common/utils.sh' are the same file

skip copy same file

grep: /etc/system-release: No such file or directory

[2023-06-16 22:46:01.906934] INFO [remove ocp docker on host:10.10.100.9]

[2023-06-16 22:46:01.944811] INFO [docker rm -f 8b044744a92e]

8b044744a92e

[2023-06-16 22:46:26.162467] INFO [remove ocp docker on host:10.10.100.9 done]

[2023-06-16 22:46:26.253927] INFO [uninstall.sh ocp  finished and reg.docker.alibaba-inc.com/oceanbase/ocp-all-in-one:3.3.3-20220906114643 removed on 10.10.100.9]

[2023-06-16 22:46:26.258281] INFO [ocp docker on 10.10.100.9 is removed]

[2023-06-16 22:46:26.263047] INFO [CLEAR_COMPONENT: ocp DONE  ######################################]

ocp1a:~/t-oceanbase-antman # docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

服务已经被清除。

报错记录处理

manage 脚本执行报错。

代码语言：javascript
复制
55obffocp:~/t-oceanbase-antman # ./manage.sh -i ob,ocp,obproxy -l 133.55.22.19 -z 1 -R Jnydzycscc@123 -A OceanBase#123

[2023-06-16 16:31:03.305062] INFO [check conf file /root/t-oceanbase-antman/obcluster.conf format ...]

[2023-06-16 16:31:03.309079] INFO [conf file is upper case format.]

[2023-06-16 16:31:03.315290] INFO [SSH_AUTH=password SSH_USER=root SSH_PORT=22 SSH_PASSWORD= SSH_KEY_FILE=/root/.ssh/id_rsa]

LB_MODE=f5

INSTALL_COMPONENTS componets: ob obproxy ocp

CLEAR_COMPONENTS:

IP_LIST: 133.55.22.19

ZONE_LIST: 1

ROOT_PASSWORD_LIST: Jnydzycscc@123

ADMIN_PASSWORD_LIST: OceanBase#123

/root/t-oceanbase-antman/common/utils.sh: line 484: -e: command not found

[2023-06-16 16:31:03.636862] ERROR [: ssh authorization to 133.55.22.19 failed, Please check SSH affinity environment varialbes.]

这个问题也需要修改脚本代码解决。

执行 alter system delete server 语句之后很久，被替换的 Server 没有 Delete 掉，一直是 deleting 状态，检查发现 OCP 的 meta 库内存参数调整过，新加的 Server 参数小，导致 UNIT 迁移卡住。

代码语言：javascript
复制
MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;

+--------------+----------------+-----------------+----------+--------------------+

| svr_ip       | zone           | with_rootserver | status   | start_service_time |

+--------------+----------------+-----------------+----------+--------------------+

| 10.10.100.87 | META_OB_ZONE_2 |               1 | active   |   1679279220991796 |

| 10.10.100.9  | META_OB_ZONE_1 |               0 | deleting |   1679275725595691 |

| 122.44.11.2  | META_OB_ZONE_3 |               0 | active   |   1640097866698859 |

| 133.55.22.19 | META_OB_ZONE_1 |               0 | active   |   1686908200404755 |

+--------------+----------------+-----------------+----------+--------------------+

4 rows in set (0.01 sec)


MySQL [oceanbase]> select count(),svr_ip from  gv$unit  group by svr_ip;

+----------+--------------+

| count() | svr_ip       |

+----------+--------------+

|       27 | 133.55.22.19 |

|       33 | 10.10.100.87 |

|       33 | 122.44.11.2  |

|        6 | 10.10.100.9  |

+----------+--------------+

4 rows in set (0.01 sec)

MySQL [oceanbase]> select *   from  gv$unit   where  svr_ip='10.10.100.9';

+---------+----------------+---------------------------------------------+------------------+----------------------------------------+----------------+-----------+-------------+-------------+----------+---------------------+-----------------------+---------+---------+-------------+-------------+----------+----------+---------------+-----------------+

| unit_id | unit_config_id | unit_config_name                            | resource_pool_id | resource_pool_name                     | zone           | tenant_id | tenant_name | svr_ip      | svr_port | migrate_from_svr_ip | migrate_from_svr_port | max_cpu | min_cpu | max_memory  | min_memory  | max_iops | min_iops | max_disk_size | max_session_num |

+---------+----------------+---------------------------------------------+------------------+----------------------------------------+----------------+-----------+-------------+-------------+----------+---------------------+-----------------------+---------+---------+-------------+-------------+----------+----------+---------------+-----------------+

|    1106 |           1090 | config_oms_tt_tenant_META_OB_ZONE_1_S2_gpa  |             1080 | pool_oms_tt_tenant_META_OB_ZONE_1_gpa  | META_OB_ZONE_1 |      NULL | NULL        | 10.10.100.9 |     2882 |                     |                     0 |       3 |       3 | 12884901888 | 12884901888 |     2500 |     2500 |  536870912000 |             750 |

|    1139 |           1094 | oms_unit                                    |             1129 | oms_ff9_tenant_resource_pool           | META_OB_ZONE_1 |      NULL | NULL        | 10.10.100.9 |     2882 |                     |                     0 |       2 |       2 |  5368709120 |  4294967296 |      128 |      128 |    5368709120 |           10000 |

|    1122 |           1097 | config_oms_c55_tenant_META_OB_ZONE_1_S1_ifu |             1088 | pool_oms_c55_tenant_META_OB_ZONE_1_ifu | META_OB_ZONE_1 |      NULL | NULL        | 10.10.100.9 |     2882 |                     |                     0 |     1.5 |     1.5 |  6442450944 |  6442450944 |     1250 |     1250 |  536870912000 |             375 |

|    1126 |           1100 | config_oms_ff6_tenant_META_OB_ZONE_1_S1_uzz |             1092 | pool_oms_ff6_tenant_META_OB_ZONE_1_uzz | META_OB_ZONE_1 |      NULL | NULL        | 10.10.100.9 |     2882 |                     |                     0 |     1.5 |     1.5 |  6442450944 |  6442450944 |     1250 |     1250 |  536870912000 |             375 |

|    1127 |           1101 | config_oms_ff7_tenant_META_OB_ZONE_1_S1_gkj |             1093 | pool_oms_ff7_tenant_META_OB_ZONE_1_gkj | META_OB_ZONE_1 |      NULL | NULL        | 10.10.100.9 |     2882 |                     |                     0 |     1.5 |     1.5 |  6442450944 |  6442450944 |     1250 |     1250 |  536870912000 |             375 |

|    1135 |           1108 | config_oms_cc8_tenant_META_OB_ZONE_1_S1_wwo |             1101 | pool_oms_cc8_tenant_META_OB_ZONE_1_wwo | META_OB_ZONE_1 |      NULL | NULL        | 10.10.100.9 |     2882 |                     |                     0 |     1.5 |     1.5 |  6442450944 |  6442450944 |     1250 |     1250 |  536870912000 |             375 |

+---------+----------------+---------------------------------------------+------------------+----------------------------------------+----------------+-----------+-------------+-------------+----------+---------------------+-----------------------+---------+---------+-------------+-------------+----------+----------+---------------+-----------------+

6 rows in set (0.02 sec)


MySQL [oceanbase]> alter  system migrate unit=1106 destination='133.55.22.19:2882';

ERROR 4624 (HY000): machine resource is not enough to hold a new unit              ----------------手动去迁移报资源不足


MySQL [oceanbase]> select zone,svr_ip, cpu_total, cpu_assigned,cpu_assigned_percent cpu_ass_pct, round(mem_total/1024/1024/1024) mem_total_gb,

->        round(mem_assigned/1024/1024/1024) mem_ass_gb, mem_assigned_percent mem_ass_pct, unit_num, migrating_unit_num, leader_count, round(load,2) load

-> from __all_virtual_server_stat

-> order by zone, svr_ip;                                ---------------------检查资源发现内存不足

+----------------+--------------+-----------+--------------+-------------+--------------+------------+-------------+----------+--------------------+--------------+------+

| zone           | svr_ip       | cpu_total | cpu_assigned | cpu_ass_pct | mem_total_gb | mem_ass_gb | mem_ass_pct | unit_num | migrating_unit_num | leader_count | load |

+----------------+--------------+-----------+--------------+-------------+--------------+------------+-------------+----------+--------------------+--------------+------+

| META_OB_ZONE_1 | 10.10.100.9  |        62 |           11 |          17 |          250 |         40 |          16 |        6 |                  0 |            0 | 0.17 |

| META_OB_ZONE_1 | 133.55.22.19 |        62 |           48 |          77 |          204 |        196 |          96 |       27 |                  0 |            0 | 0.87 |

| META_OB_ZONE_2 | 10.10.100.87 |        62 |           59 |          95 |          250 |        236 |          94 |       33 |                  0 |         2935 | 0.95 |

| META_OB_ZONE_3 | 122.44.11.2  |        62 |           59 |          95 |          250 |        236 |          94 |       33 |                  0 |         1051 | 0.95 |


MySQL [oceanbase]> show parameters  like  '%memory_limit%'

;

| META_OB_ZONE_3 | observer | 122.44.11.2  |     2882 | memory_limit                        | NULL      | 300G  | the size of the memory reserved for internal use(for testing purpose). Range: [0M,)                                                                                                                                                    | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |

| META_OB_ZONE_1 | observer | 133.55.22.19 |     2882 | memory_limit                        | NULL      | 254G  | the size of the memory reserved for internal use(for testing purpose). Range: [0M,)                                                                                                                                                    | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |

| META_OB_ZONE_2 | observer | 10.10.100.87 |     2882 | memory_limit                        | NULL      | 300G  | the size of the memory reserved for internal use(for testing purpose). Range: [0M,)                                                                                                                                                    | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |

| META_OB_ZONE_1 | observer | 10.10.100.9  |     2882 | memory_limit                        | NULL      | 300G  | the size of the memory reserved for internal use(for testing purpose). Range: [0M,)                                                                                                                                                    | OBSERVER | CLUSTER | DEFAULT | DYNAMIC_EFFECTIVE |


MySQL [oceanbase]> alter   system  set   memory_limit ='300G'  ;

Query OK, 0 rows affected (0.05 sec)


MySQL [oceanbase]> select count(),svr_ip from  gv$unit  group by svr_ip;

+----------+--------------+

| count() | svr_ip       |

+----------+--------------+

|       33 | 133.55.22.19 |

|       33 | 10.10.100.87 |

|       33 | 122.44.11.2  |

+----------+--------------+

3 rows in set (0.00 sec)


MySQL [oceanbase]> select svr_ip, zone, with_rootserver, status, start_service_time from __all_server;

+--------------+----------------+-----------------+--------+--------------------+

| svr_ip       | zone           | with_rootserver | status | start_service_time |

+--------------+----------------+-----------------+--------+--------------------+

| 10.10.100.87 | META_OB_ZONE_2 |               1 | active |   1679279220991796 |

| 122.44.11.2  | META_OB_ZONE_3 |               0 | active |   1640097866698859 |

| 133.55.22.19 | META_OB_ZONE_1 |               0 | active |   1686908200404755 |

+--------------+----------------+-----------------+--------+--------------------+

3 rows in set (0.00 sec)

4总结

到此，使用 ANTMAN 工具的方式去替换 OCP 机器的操作就结束了，包括前面一篇使用 OAT 替换 OCP 节点的文章可能看起来没什么难度，但是整个过程来回做了好几遍，充满我的坎坷和泪水。为了别人以后少踩坑，所以写了这两篇文章分享。

如果看了上篇文章的话，应该知道 OAT 替换 OCP 的时候，新加机器是在 metadb 中新创建了一个 ZONE，然后再把被替换机器下掉。其中还涉及新建资源池修改 Locality，增加副本数等操作。其实使用 ANTMAN 工具的话这个步骤就不太一样，是将新机器加入到需要替换机器的同一个 ZONE 内，然后做同 ZONE 内 UNIT 的迁移，然后把被替换的机器下线。

现阶段的话，相对来说使用 ANTMAN 工具替换之后对于 OCP 元数据的影响更小一些，但是 OAT 黑屏操作更少一些。对于 OBProxy 单独 Docker 的前期场景必须使用 ANTMAN，后期版本就看大家自己酌情选择了。

行之所向，莫问远方。

本文关键字：#OceanBase# #ANTMAN# #OCP#

#meta#oceanbase#root#工具#云平台