Tungsten Fabric实战：对接vMX虚拟路由平台填坑

我在CentOS虚拟机上做TF+k8s集成部署时，遇到的5个坑~

TungstenFabric

678人浏览 · 2020-05-20 15:08:04

TungstenFabric · 2020-05-20 15:08:04 发布

要了解Tungsten Fabric的硬件对接机制，不一定真的去购买交换机设备来部署，更现实的方法是使用虚拟机软件版本来做模拟实验，本文使用瞻博网络vMX虚拟路由平台，模拟其MX和QFX系列交换机的对接扩展操作。

下面做的实验就是基于EVE-NG模拟平台，在CentOS虚拟机上做Tungsten Fabric&k8s集成部署，然后启动vMX来做对接。

（编者按：本文着眼于能够对接并对vMX下发配置，如有疑问，欢迎加入社区与开源SDN爱好者们一起交流）

准备

后续对python文件的修改，都需要进入device-manager这个docker进行修改。

[root@master01 ~]# docker ps | grep devi
001caca78cde        hub.juniper.net/contrail-controller-config-devicemgr:1912-latest    "/entrypoint.sh /usr…"   8 hours ago         Up About an hour                        config_devicemgr_1
[root@master01 ~]# 
[root@master01 ~]# docker exec -it 001caca78cde bash
(config-device-manager)[root@master01 /]$ cd /usr/lib/python2.7/site-packages/device_manager
(config-device-manager)[root@master01 /usr/lib/python2.7/site-packages/device_manager]$ 
(config-device-manager)[root@master01 /usr/lib/python2.7/site-packages/device_manager]$ ll | egrep "*.py$"
-rw-r--r--. 1 root root      0 Nov 18 18:30 __init__.py
-rw-r--r--. 1 root root   6409 Nov 18 18:30 ansible_base.py
-rw-r--r--. 1 root root  27640 Apr 19 23:40 ansible_conf.py
-rw-r--r--. 1 root root  69004 Nov 18 18:30 ansible_role_common.py
-rw-r--r--. 1 root root   1256 Nov 18 18:30 assisted_replicator_feature.py
-rw-r--r--. 1 root root 120643 Nov 18 18:30 db.py
-rw-r--r--. 1 root root   6626 Apr 19 22:25 device_conf.py
-rw-r--r--. 1 root root  32866 Nov 18 18:30 device_job_manager.py
-rw-r--r--. 1 root root  20691 Nov 18 18:30 device_manager.py
-rw-r--r--. 1 root root  11726 Nov 18 18:30 device_ztp_manager.py
-rw-r--r--. 1 root root   2054 Nov 18 18:30 dm_amqp.py
-rw-r--r--. 1 root root   6351 Nov 18 18:30 dm_server.py
-rw-r--r--. 1 root root  11173 Nov 18 18:30 dm_server_args.py
-rw-r--r--. 1 root root  20935 Nov 18 18:30 dm_utils.py
-rw-r--r--. 1 root root  76077 Apr 19 22:03 e2_conf.py
-rw-r--r--. 1 root root   3397 Nov 18 18:30 e2_services_info.py
-rw-r--r--. 1 root root   9484 Nov 18 18:30 fabric_manager.py
-rw-r--r--. 1 root root  10850 Nov 18 18:30 feature_base.py
-rw-r--r--. 1 root root   1309 Nov 18 18:30 imports.py
-rw-r--r--. 1 root root   7797 Nov 18 18:30 job_handler.py
-rw-r--r--. 1 root root  26029 Apr 20 02:49 juniper_conf.py
-rw-r--r--. 1 root root   6247 Nov 18 18:30 l2_gateway_feature.py
-rw-r--r--. 1 root root   3367 Nov 18 18:30 l3_gateway_feature.py
-rw-r--r--. 1 root root  12661 Nov 18 18:30 logger.py
-rw-r--r--. 1 root root  39551 Nov 18 18:30 mx_conf.py
-rw-r--r--. 1 root root   4164 Nov 18 18:30 opserver_util.py
-rw-r--r--. 1 root root   6568 Nov 18 18:30 overlay_bgp_feature.py
-rw-r--r--. 1 root root   1082 Nov 18 18:30 overlay_conf.py
-rw-r--r--. 1 root root  16762 Nov 18 18:30 pnf_conf.py
-rw-r--r--. 1 root root   2029 Nov 18 18:30 qfx_10k.py
-rw-r--r--. 1 root root   1612 Nov 18 18:30 qfx_5k.py
-rw-r--r--. 1 root root  42072 Nov 18 18:30 qfx_conf.py
-rw-r--r--. 1 root root   4932 Nov 18 18:30 storm_control_feature.py
-rw-r--r--. 1 root root   7297 Nov 18 18:30 telemetry_feature.py
-rw-r--r--. 1 root root   5068 Nov 18 18:30 underlay_ip_clos_feature.py
-rw-r--r--. 1 root root   5485 Nov 18 18:30 vn_interconnect_feature.py
(config-device-manager)[root@master01 /usr/lib/python2.7/site-packages/device_manager]$

修改前记得先备份，例如：

 # cp juniper_conf.py juniper_conf.py_bak

修改后需要重启device-manager这个docker，修改才会生效。

[root@master01 ~]# docker restart 001caca78cde
001caca78cde

log打印最好一直开着，默认只有WARNNING和ERROR的信息会打印出来。

[root@master01 ~]# tailf /var/log/contrail/contrail-device-manager.log

根据打印信息找到代码位置，是最便捷的定位方法。
如果要自己向py文件中添加打印信息，下面是个例子。

self._logger.warning(">>password is %s" % self.user_creds[password])

如果发现修改后docker不停地自动重启，那就抓紧时间进入docker后，将之前修改的py文件恢复，例如：

#cd  /usr/lib/python2.7/site-packages/device_manager
#cp juniper_conf.py_bak juniper_conf.py

坑一：

描述：Tungsten Fabric5.1.0版本默认对接物理交换机的功能是关闭的。
解决方法：参考微信公众号【TF中文社区】–「 Tungsten Fabric入门宝典系列文章」，需要手动添加一项，建议接着删除所有contrail相关的容器后重建。

[root@master01 ~]# cat /etc/contrail/common_config.env 
KUBERNETES_API_SERVER=192.168.122.177
TTY=True
ANALYTICSDB_NODES=192.168.122.177
ANALYTICS_SNMP_ENABLE=False
STDIN_OPEN=True
WEBUI_NODES=192.168.122.177
CONTROLLER_NODES=192.168.122.177
KUBERNETES_API_NODES=192.168.122.177
ANALYTICS_NODES=192.168.122.177
ANALYTICSDB_ENABLE=True
CONFIGDB_NODES=192.168.122.177
ANALYTICS_ALARM_ENABLE=False
CLOUD_ORCHESTRATOR=kubernetes
CONTROL_NODES=192.168.122.177
KUBEMANAGER_NODES=192.168.122.177
CONFIG_NODES=192.168.122.177
CONTRAIL_VERSION=1912-latest
DEVICE_MANAGER__DEFAULTS__push_mode=0   #<<<<<<<新加此行
[root@master01 ~]# 
[root@master01 ~]# docker-compose -f /etc/contrail/config/docker-compose.yaml down
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.3) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
Stopping config_devicemgr_1  ... done
Stopping config_svcmonitor_1 ... done
Stopping config_api_1        ... done
Stopping config_schema_1     ... done
Stopping config_nodemgr_1    ... done
Stopping config_dnsmasq_1    ... done
Stopping config_stats_1      ... done
Removing config_devicemgr_1  ... done
Removing config_svcmonitor_1 ... done
Removing config_api_1        ... done
Removing config_schema_1     ... done
Removing config_nodemgr_1    ... done
Removing config_dnsmasq_1    ... done
Removing config_stats_1      ... done
Removing config_node-init_1  ... done
[root@master01 ~]# 

[root@master01 ~]# docker-compose -f /etc/contrail/config/docker-compose.yaml up -d
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.3) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
Creating config_node-init_1 ... done
Creating config_svcmonitor_1 ... done
Creating config_schema_1     ... done
Creating config_api_1        ... done
Creating config_nodemgr_1    ... done
Creating config_stats_1      ... done
Creating config_dnsmasq_1    ... done
Creating config_devicemgr_1  ... done
[root@master01 ~]#

坑二：

描述：Tungsten Fabric的netconf session会去对接交换机的接口值已经被写死(juniper_conf.py),所以在web录入的时候service port填的值不会生效。

def device_connect(self):
        if not self._nc_manager or not self.is_connected():
            try:
                self._nc_manager = manager.connect(host=self.management_ip, port=22,

解决方法：物理交换机的netconf port设置为22。

[edit]
netops@vMX-3# set system services netconf ssh port 22 ，才好对接

坑三：

描述：代码传入的netconf password是密文，但是ncclient只能接受密码的原文，导致：

04/19/2020 08:00:53 PM [contrail-device-manager] [ERROR]: could not establish netconf session with  router 192.168.122.103: AuthenticationException('Authentication failed.',)

例如我设置的密码是"Test123"，但是其实传的是：
在这里插入图片描述
解决方法(juniper_conf.py)：是的，我就直接赋值了。

def device_connect(self):
        if not self._nc_manager or not self.is_connected():
            self._logger.warning(">>keys are %s" % self.user_creds.keys())
            self._logger.warning(">>type of password is %s" % type(self.user_creds['password']))
            self._logger.warning(">>password is %s" % self.user_creds['password'])
            try:
                self._nc_manager = manager.connect(host=self.management_ip, port=22,
                             username=self.user_creds['username'],
                             #password=self.user_creds['password'],
                             password="Test123",

坑四：

描述：netconf对接成功后，Tungsten Fabric会取回device model信息，并判断是否是mx或vmx，但是我用的模拟器返回的是olive。

netops@vMX-3> show version
Hostname: vMX-3
Model: olive
Junos: 18.2R2.6

因此报错。

04/20/2020 09:02:06 PM [contrail-device-manager] [ERROR]: product model mismatch: device model = olive, plugin mode = ['mx', 'vmx']

解决方法：device_conf.py直接给model赋值。

 def validate_device(self):
        if not self.device_config:
            self.device_config = self.device_get()
        if not self.device_config:
            self.device_config = {}
            return False
        #model = self.device_config.get('product-model')
        model = 'mx'

坑五：

描述：添加BGP Router后，配置无法下发成功。明明输入的是64512，结果识别为"b’64512’"。
在这里插入图片描述

04/19/2020 09:25:38 PM [contrail-device-manager] [ERROR]: Router 192.168.122.103: error: as_number: 'b'64512'': Use format 'x' or 'x.y' to specify number 'x' (Range 1 .. 4294967295) or number with higher 2 bytes 'x' (Range 0 .. 65535) ay
error: syntax error, expecting <rpc> or </rpc>

将conf_str打印出来self._logger.warning(">>config_str is %s" % config_str)。

<configuration>
        <groups operation="replace">
            <junos:comment>/* Contrail Generated Group Config */</junos:comment>
            <name>__contrail__</name>
            <routing-options>
                <junos:comment>/* Global Routing Options */</junos:comment>
                <router-id>3.3.3.3</router-id>
                <route-distinguisher-id>3.3.3.3</route-distinguisher-id>
                <autonomous-system>b'64512'</autonomous-system>
                <resolution>
                    <rib>
                        <name>bgp.rtarget.0</name>
                        <resolution-ribs>inet.0</resolution-ribs>
                    </rib>
                </resolution>
            </routing-options>
            <protocols>
                <junos:comment>/* Protocols Configuration */</junos:comment>
                <bgp>
                    <group>
                        <junos:comment>/* overlay_bgp: BGP Router: qfx5100-48t-1, UUID: f0afa9d3-d388-45ea-b6e2-c5319e9664e0 */</junos:comment>
                        <name>b'_contrail_asn-64512'</name>
                        <type>internal</type>
                        <local-address>3.3.3.3</local-address>
                        <hold-time>90</hold-time>
                        <family>
                            <evpn>
                                <signaling></signaling>
                            </evpn>
                            <route-target></route-target>
                        </family>
                        <neighbor>
                            <junos:comment>/* overlay_bgp: BGP Router: master01.local, UUID: b04bc39c-57ea-4f0e-b04a-3d2737ce242f */</junos:comment>
                            <name>192.168.122.177</name>
                            <peer-as>64512</peer-as>
                        </neighbor>
                    </group>
                </bgp>
            </protocols>
            <policy-options>
                <junos:comment>/* Policy Options */</junos:comment>
                <community>
                    <name>_contrail_switch_policy_</name>
                    <members>b'target:64512:1'</members>
                </community>
            </policy-options>
            <switch-options>
                <vtep-source-interface>lo0.0</vtep-source-interface>
            </switch-options>
        </groups>
        <apply-groups>
            <name>__contrail__</name>
        </apply-groups>
    </configuration>
</config>

解决方法：经过大神指点，将contrail-controller-config-devicemgr镜像从1912-latest替换为1910-laset。
在我的deployer节点操作。

[root@deployer ~]# docker pull opencontrailnightly/contrail-controller-config-devicemgr:1910-latest  
[root@deployer ~]# docker tag opencontrailnightly/contrail-controller-config-devicemgr:1910-latest 192.168.122.211/contrail-controller-config-devicemgr:1910-latest
[root@deployer ~]# docker push 192.168.122.211/contrail-controller-config-devicemgr:1910-latest
[root@deployer ~]#
[root@deployer ~]# curl -X GET http://localhost/v2/contrail-controller-config-devicemgr/tags/list --cacert /etc/docker/certs.d/dockershare:2333/ca.crt | python -m json.tool
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    85  100    85    0     0   1072      0 --:--:-- --:--:-- --:--:--  1089
{
    "name": "contrail-controller-config-devicemgr",
    "tags": [
        "1912-latest",
        "1910-latest"
    ]
}
[root@deployer ~]#
12345678910111213141516

master01节点拉取(hosts文件里面已经将hub.juniper.net设置为deployer IP)。

[root@master01 ~]# ping hub.juniper.net
PING deployer (192.168.122.211) 56(84) bytes of data.
64 bytes from deployer (192.168.122.211): icmp_seq=1 ttl=64 time=1.47 ms
^C
--- deployer ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.472/1.472/1.472/0.000 ms
[root@master01 ~]#
[root@master01 ~]# docker pull hub.juniper.net/contrail-controller-config-devicemgr:1910-latest  
[root@master01 ~]#
[root@master01 ~]# docker image list | grep devicemgr
hub.juniper.net/contrail-controller-config-devicemgr    1912-latest         c08868a27a0a        5 months ago        772MB
hub.juniper.net/contrail-controller-config-devicemgr    1910-latest         a1b3ab402efc        5 months ago        862MB
[root@master01 ~]#
1234567891011121314

修改/etc/contrail/config/docker-compose.yaml文件，将devicemgr的镜像版本从1912-latest改为1910-latest。

  devicemgr:
    image: "hub.juniper.net/contrail-controller-config-devicemgr:1910-latest"

将contrail相关的容器删除然后重建。

 [root@master01 config]# docker-compose -f /etc/contrail/config/docker-compose.yaml down
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.3) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
Stopping config_devicemgr_1  ... done
Stopping config_dnsmasq_1    ... done
Stopping config_nodemgr_1    ... done
Stopping config_schema_1     ... done
Stopping config_svcmonitor_1 ... done
Stopping config_stats_1      ... done
Stopping config_api_1        ... done
Removing config_devicemgr_1  ... done
Removing config_dnsmasq_1    ... done
Removing config_nodemgr_1    ... done
Removing config_schema_1     ... done
Removing config_svcmonitor_1 ... done
Removing config_stats_1      ... done
Removing config_api_1        ... done
Removing config_node-init_1  ... done
[root@master01 config]#
[root@master01 config]# docker-compose -f /etc/contrail/config/docker-compose.yaml up -d
/usr/lib/python2.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.24.3) or chardet (2.2.1) doesn't match a supported version!
  RequestsDependencyWarning)
Creating config_node-init_1 ... done
Creating config_schema_1     ... done
Creating config_api_1        ... done
Creating config_svcmonitor_1 ... done
Creating config_nodemgr_1    ... done
Creating config_dnsmasq_1    ... done
Creating config_stats_1      ... done
Creating config_devicemgr_1  ... done
[root@master01 config]#
1234567891011121314151617181920212223242526272829303132

前面提到的坑1-4，需要在新的devicemgr中重新修改一遍。

验证

Physical Router已经添加(role为None)
在这里插入图片描述
添加BGP Router

查看vMX上新增配置

netops@vMX-3> show configuration | compare rollback 2   
[edit]
+ groups {
+     /* Contrail Generated Group Config */
+     __contrail__ {
+         routing-options {
+             /* Global Routing Options */
+             router-id 3.3.3.3;
+             route-distinguisher-id 3.3.3.3;
+             autonomous-system 64512;
+             dynamic-tunnels {
+                 _contrail_asn-64512 {
+                     source-address 3.3.3.3;
+                     gre;
+                     destination-networks {
+                         /* IP Fabric Subnet */
+                         192.168.122.0/24;
+                         /* BGP Router : MX3 */
+                         3.3.3.3/32;
+                         /* BGP Router : master01.local */
+                         192.168.122.177/32;
+                     }
+                 }
+             }
+         }
+         protocols {
+             /* Protocols Configuration */
+             bgp {
+                 /* overlay_bgp: BGP Router: MX3, UUID: 2cf712e4-a3f2-4978-82a3-88c993754202 */
+                 group _contrail_asn-64512 {
+                     type internal;
+                     local-address 3.3.3.3;
+                     hold-time 90;
+                     family inet-vpn {
+                         unicast;
+                     }
+                     family evpn {
+                         signaling;
+                     }
+                     family route-target;
+                     export _contrail_ibgp_export_policy;
+                     /* overlay_bgp: BGP Router: master01.local, UUID: b04bc39c-57ea-4f0e-b04a-3d2737ce242f */
+                     neighbor 192.168.122.177 {
+                         peer-as 64512;
+                     }
+                 }                    
+             }                        
+         }
+         policy-options {
+             /* iBGP Export Policy */
+             policy-statement _contrail_ibgp_export_policy {
+                 term inet-vpn {
+                     from family inet-vpn;
+                     then {
+                         next-hop self;
+                     }
+                 }
+             }
+         }
+     }
+ }
+ apply-groups __contrail__;
netops@vMX-3>

本文为盛科网络刘敬一的原创文章。
原文链接：https://blog.csdn.net/ljyfree/article/details/105652253

Tungsten Fabric入门宝典系列文章——
1.首次启动和运行指南
2. TF组件的七种“武器”
3. 编排器集成
4.关于安装的那些事（上）
5.关于安装的那些事（下）
6.主流监控系统工具的集成
7.开始第二天的工作
8.8个典型故障及排查Tips

在这里插入图片描述

K8S/Kubernetes

K8S/Kubernetes社区为您提供最前沿的新闻资讯和知识内容

更多推荐

【深度】阿里巴巴万级规模 K8s 集群全局高可用体系之美

作者 | 韩堂、柘远、沉醉来源 | 阿里巴巴云原生公众号前言台湾作家林清玄在接受记者采访的时候，如此评价自己 30 多年写作生涯：“第一个十年我才华横溢，‘贼光闪现’，令周边黯然失色；第二个十年，我终于‘宝光现形’，不再去抢风头，反而与身边的美丽相得益彰；进入第三个十年，繁华落尽见真醇，我进入了‘醇光初现’的阶段，真正体味到了境界之美”。长夜有穷，真水无香。领略过了 K8s“身在江

K8S/Kubernetes

如何基于 K8s 构建下一代 DevOps 平台？

作者 | 孙健波（天元）导读：当前云原生 DevOps 体系现状如何？面临哪些挑战？如何通过 OAM 解决云原生 DevOps 场景下的诸多问题？云原生开发应用模型 OAM(Open Application Model) 社区核心成员孙健波将为大家一一解答，并分享如何基于 OAM 和 Kubernetes 打造无限能力的下一代 DevOps 平台。什么是 DevOps？为什么基于 Kub