In the previous article Using MetalLB as LoadBalancer in Kubernetes cluster (Part 1) In, we use MetalLB's Layer2 pattern as the implementation of LoadBalancer to expose the services in Kubernetes cluster outside the cluster.

Remember the IP address pool we assigned to MetalLB in Configmap?

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.1.30-192.168.1.49

The 192.168.1.30-192.168.1.49 IP segment allocated here happens to be in the author's home network. When we use 192.168.1.30, we can successfully access the service.

When I mentioned the disadvantages of Layer2 before, I also missed one point. In addition to the impact on availability and single point network bottleneck during failover, the client needs to be located in the same subnet as the address pool (if the address pool is changed to 192.168.1.30-192.168.1.49, the service will not be accessible). However, in the experimental environment or homelab environment like the author, the first two are not problems, and the latter is a little troublesome in network configuration.

Although the disadvantages are obvious, Layer2 mode has stronger universality. Unlike BGP mode, it needs to support BGP routing. However, these can not stop the author's exploration (strong) cable (compulsive) desire (disease), because there is also an OpenWrt soft route running in My Proxmox virtual machine Yes. This OpenWrt provides external routing services in the form of soft routing through 192.168.1.2, and supports BGP by installing routing software suite.

Before we start, let's take a look at what BPG is and related terms. Students who already know or feel too abstract can skip directly and look back after reading the demo.

What is BGP

BGP is the abbreviation of Border Gateway Protocol.

Border gateway protocol is a core decentralized autonomous routing protocol on the Internet. It realizes the reachability between autonomous systems (AS) by maintaining IP routing table or "prefix" table. It belongs to vector routing protocol. BGP does not use the indicators of traditional internal gateway protocol (IGP), but uses routing based on path, network policy or rule set. Therefore, it is more suitable to be called vector protocol than routing protocol.

The neighbor relationship of BGP (or communication peer / peer entity, peer) is realized through manual configuration. The peer entities establish a session to exchange data through TCP port 179. The BGP router will periodically send 19 bytes of keep alive messages to maintain the connection (the default cycle is 60 seconds). Among various routing protocols, only BGP uses TCP as the transport layer protocol.

BGP running between two or more peer entities in the same AS autonomous system is called iBGP (internal / internal BGP). BGP running between peer entities belonging to different AS is called eBGP (external / external BGP). The router that exchanges information with other AS on the AS boundary is called border/edge router, and the border routers are eBGP opposite ends to each other. In Cisco IOS, the route distance advertised by iBGP is 200, and the priority is lower than that advertised by eBGP and any internal gateway protocol (IGP). In other router implementations, the priority order is that eBGP is higher than IGP, and IGP is higher than iBGP. The main difference between Mar 6, 2022Mar 6, 2022 iBGP and eBGP is the behavior of forwarding routing information. For example, the routing information obtained from eBGP peers will be distributed to all iBGP peers and eBGP peers, but the routing information obtained from iBGP peers will only be distributed to all eBGP peers. All iBGP peers need to be fully interconnected.

Three terms are mentioned here: autonomous system (AS), internal gateway protocol (IGP) and external gateway protocol (EGP).

Autonomous system AS

Let's take a look at the introduction from Wikipedia:

The combination of self-made router and Internet system refers to the combination of self-made router and Internet system. Autonomous system numbers are 16 bit integers, which can be assigned to up to 65536 autonomous systems. The autonomous system number is divided into two ranges. The first range is the public AS N, from 1 to 64511, which can be used on the Internet; The second range is those from 64512 to 65535 called private numbers, which can only be used within an organization's own network.

In short, China Telecom, China Mobile and China Unicom all have their own AS numbers, and there is more than one. If you are interested, you can check the AS numbers in Wikipedia China Internet backbone network Entry.

In addition to the ASN publicly available on the Internet, private numbers can be used internally. For example, I can create several AS using private numbers in my home network.

Internal routing protocol IGP

Quoting from encyclopedia is not the focus of this article, so I won't introduce it too much.

Internal gateway protocol (IGP) refers to a routing protocol used within an autonomous system (AS).

External gateway protocol EGP

External gateway protocol (EGP) is an outdated Internet routing protocol. Replaced by BPG.

Origin of BPG

BPG is created to replace EGP, and can be applied not only outside the AS, but also inside the AS. Therefore, it is divided into EBGP and IBGP.

Having said so much, it may be a little abstract. Go directly to the demo.

Demo

The environment is the same AS before. According to the pre assumption, we want to create two AS: 65000 and 65001< u> The former is the AS where the router and client are located, while the latter is the AS where our cluster service LoadBalancer IP is located < / u >.

We need to make OpenWrt support BGP first.

OpenWrt supports BGP

In order to make OpenWrt support BGP, the routing software suite is used here Quagga . Quagga provides the implementation of OSPFv2, OSPFv3, RIP v1 v2, RIPng and BGP-4.

Quagga architecture consists of core daemon and zebra, which serves as the abstraction layer of the underlying Unix kernel and provides Zserv API to quagga clients through Unix or TCP. It is these Zserv clients that implement the routing protocol and send the routing updates to the zebra daemon. The current implementation of Zserv is:

Quagga's daemon can be configured through a network accessible cli (vty). The CLI follows a similar style to other routing software. An additional tool vtysh is provided, which acts as the aggregation front end of all daemons, allowing all functions of all quagga daemons to be managed in one place.

Execute the following command to complete the installation:

$ opkg update && opkg install quagga quagga-zebra quagga-bgpd quagga-vtysh

After successful installation, it will automatically start and listen to the port:

$ netstat -lantp | grep -e 'zebra\|bgpd'
tcp        0      0 0.0.0.0:2601            0.0.0.0:*               LISTEN      2984/zebra
tcp        0      0 0.0.0.0:2605            0.0.0.0:*               LISTEN      3000/bgpd
tcp        0      0 :::2601                 :::*                    LISTEN      2984/zebra
tcp        0      0 :::2605                 :::*                    LISTEN      3000/bgpd

The 179 port that bpgd listens to for receiving routing information is not seen here, because the route has not been assigned AS yet. Don't worry, let's use the command vtysh to enter vty for configuration:

$ vtysh
OpenWrt# conf t
OpenWrt(config)# router bgp 65000
OpenWrt(config-router)# neighbor 192.168.1.5 remote-as 65001
OpenWrt(config-router)# neighbor 192.168.1.5 description ubuntu-dev1
OpenWrt(config-router)# neighbor 192.168.1.6 remote-as 65001
OpenWrt(config-router)# neighbor 192.168.1.6 description ubuntu-dev2
OpenWrt(config-router)# exit
OpenWrt(config)# exit

Use the show ip bgp summary command in vty to view:

OpenWrt# show ip bgp summary
BGP router identifier 192.168.1.2, local AS number 65000
RIB entries 0, using 0 bytes of memory
Peers 2, using 18 KiB of memory

Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
192.168.1.5     4 65001       0       0        0    0    0 never    Active
192.168.1.6     4 65001       0       0        0    0    0 never    Active

Total number of neighbors 2

Total num. Established sessions 0
Total num. of routes received     0

At this point, we can see that bgpd is already monitoring port 179 by checking port monitoring:

$ netstat -lantp | grep -e 'zebra\|bgpd'
tcp        0      0 0.0.0.0:179             0.0.0.0:*               LISTEN      3000/bgpd
tcp        0      0 0.0.0.0:2601            0.0.0.0:*               LISTEN      2984/zebra
tcp        0      0 0.0.0.0:2605            0.0.0.0:*               LISTEN      3000/bgpd
tcp        0      0 :::179                  :::*                    LISTEN      3000/bgpd
tcp        0      0 :::2601                 :::*                    LISTEN      2984/zebra
tcp        0      0 :::2605                 :::*                    LISTEN      3000/bgpd

After the BGP route is set, it is part of MetalLB.

MetalLB BGP mode

Let's update the configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  namespace: metallb-system
  name: config
data:
  config: |
    peers:
    - peer-address: 192.168.1.2
      peer-asn: 65000
      my-asn: 65001
    address-pools:
    - name: default
      protocol: bgp
      addresses:
      - 192.168.0.30-192.168.0.49

After the update, you will find that the EXTERNAL-IP of the Service has not been reassigned, and the controller of MetalLB has not been automatically configured. We delete the controller pod to restart:

$ kubectl delete po -n metallb-system -l app=metallb,component=controller
pod "controller-66445f859d-vss2t" deleted

You can see that the Service is assigned a new IP address:

$ kubectl get svc -n default
NAME         TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)          AGE
kubernetes   ClusterIP      10.43.0.1       <none>         443/TCP          25m
nginx-lb     LoadBalancer   10.43.188.185   192.168.0.30   8080:30381/TCP   21m
nginx2-lb    LoadBalancer   10.43.208.169   192.168.0.31   8080:32319/TCP   21m

Check the log of speaker POD. You can see that the communication with peer 192.168.1.2 has started and the announcement of IP address has been released:

{"caller":"level.go:63","configmap":"metallb-system/config","event":"peerAdded","level":"info","msg":"peer configured, starting BGP session","peer":"192.168.1.2","ts":"2022-03-06T22:56:17.336335657Z"}
{"caller":"level.go:63","configmap":"metallb-system/config","event":"configLoaded","level":"info","msg":"config (re)loaded","ts":"2022-03-06T22:56:17.336366122Z"}
struct { Version uint8; ASN16 uint16; HoldTime uint16; RouterID uint32; OptsLen uint8 }{Version:0x4, ASN16:0xfde8, HoldTime:0xb4, RouterID:0xc0a80102, OptsLen:0x1e}
{"caller":"level.go:63","event":"sessionUp","level":"info","localASN":65001,"msg":"BGP session established","peer":"192.168.1.2:179","peerASN":65000,"ts":"2022-03-06T22:56:17.337341549Z"}
{"caller":"level.go:63","event":"updatedAdvertisements","ips":["192.168.0.30"],"level":"info","msg":"making advertisements using BGP","numAds":1,"pool":"default","protocol":"bgp","service":"default/nginx-lb","ts":"2022-03-06T22:56:17.341939983Z"}
{"caller":"level.go:63","event":"serviceAnnounced","ips":["192.168.0.30"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"default/nginx-lb","ts":"2022-03-06T22:56:17.341987657Z"}
{"caller":"level.go:63","event":"updatedAdvertisements","ips":["192.168.0.31"],"level":"info","msg":"making advertisements using BGP","numAds":1,"pool":"default","protocol":"bgp","service":"default/nginx2-lb","ts":"2022-03-06T22:56:17.342041554Z"}
{"caller":"level.go:63","event":"serviceAnnounced","ips":["192.168.0.31"],"level":"info","msg":"service has IP, announcing","pool":"default","protocol":"bgp","service":"default/nginx2-lb","ts":"2022-03-06T22:56:17.342056076Z"}

Then you can view the routing table in vty:

OpenWrt# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel, N - NHRP,
       > - selected route, * - FIB route

K>* 0.0.0.0/0 via 192.168.1.1, br-lan
C>* 127.0.0.0/8 is directly connected, lo
B>* 192.168.0.30/32 [20/0] via 192.168.1.5, br-lan, 00:00:06
B>* 192.168.0.31/32 [20/0] via 192.168.1.5, br-lan, 00:00:06
C>* 192.168.1.0/24 is directly connected, br-lan

From the table, we can find the routes of 192.168.0.30/32 and 192.168.0.31/32 bgps.

test

Our new IP access:

$ curl -I 192.168.0.30:8080
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Sun, 06 Mar 2022 23:10:33 GMT
Content-Type: text/html
Content-Length: 615
Last-Modified: Tue, 25 Jan 2022 15:03:52 GMT
Connection: keep-alive
ETag: "61f01158-267"
Accept-Ranges: bytes

summary

So far, we have tried two modes of MetalLB: Layer2 has strong universality and does not need any other dependencies, but its disadvantages are also obvious; In addition to relying on the route supporting BGP, BGP mode has no restrictions in other aspects, and there is no availability problem.

BGP should be the ultimate mode of LoadBalancer, but Layer2 is not useless. We still need to make rational choices based on the scenarios used. For example, I will choose Layer2 mode for homelab.

The article is unified in the official account of the cloud.