F5 CIS Installation

Installation

Install on kube-system

Install
kubectl create secret generic bigip-login --from-literal=username=admin --from-literal=password=admin -n kube-system
kubectl create serviceaccount bigip-ctlr -n kube-system
kubectl create -f rbac.yaml
kubectl create -f cis.yaml
Uninstall
cd install-on-kube-system/
kubectl delete -f cis.yaml
kubectl delete -f rbac.yaml
kubectl delete serviceaccount bigip-ctlr -n kube-system
kubectl delete secret bigip-login -n kube-system

Install on bigip-ctlr

Install
kubectl create ns bigip-ctlr
kubectl create secret generic bigip-login --from-literal=username=admin --from-literal=password=admin -n bigip-ctlr
kubectl create serviceaccount bigip-ctlr -n bigip-ctlr
kubectl create -f rbac.yaml
kubectl create -f cis.yaml
Uninstall
cd install-on-bigip-ctlr/
kubectl delete -f cis.yaml
kubectl delete -f rbac.yaml
kubectl delete serviceaccount bigip-ctlr -n bigip-ctlr
kubectl delete secret bigip-login -n bigip-ctlr
kubectl delete ns bigip-ctlr

livenessProbe & readinessProbe

This page provide an example of CIS Pod with livenessProbe & readinessProbe.

Key Config
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 15
            periodSeconds: 15
            successThreshold: 1
            timeoutSeconds: 15
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 15

VE 健壮性检测(curl)

          livenessProbe:
            failureThreshold: 3
            exec:
              command:
              - curl
              - -k
              - -s
              - -o
              - /dev/null
              - https://192.168.200.208
            initialDelaySeconds: 15
            periodSeconds: 15
            successThreshold: 1
Note
curl 命令只存在于 Red Hat Linux Base 镜像中。

VE 健壮性检测(sh + python)

          livenessProbe:
            exec:
              command:
              - sh
              - -c
              - echo 'import requests' > /tmp/live.py ; echo 'response = requests.get("https://192.168.200.208", verify=False, timeout=3)'  >> /tmp/live.py ; python -W ignore /tmp/live.py
            initialDelaySeconds: 15
            periodSeconds: 15
            successThreshold: 1
            failureThreshold: 3
            timeoutSeconds: 5
timeoutSeconds timeout VE 重启过程 CIS Restart 次数

1

1

7

3

2

6

5

3

6

15

15

5

Integrate CIS with flannel (etcd mode, docker0 network)

Refer to flannel-etcd-mode for detailed steps.

Install on bigip-ctlr flannel vxlan Control Hub Mode

What’s Control Hub Mode

cis vxlan.png

Control Hub Mode is mainly for Ops Convenient, which use a Control Hub(one or multiple namespaces) to hold CIS configmap only.

HubMode expands on current ConfigMap implementation in CIS using the AS3 API. One of key strength of CIS is it can help multiple team collaborate better together. With microservices architecture we are seeing organizations create dedicate team that combine network and system personas. This allegement request that network engineers (NetOps) be added to these teams. F5 CIS ConfigMap is a perfect fit as it can accelerates the developer experience and workflow tooling.

Setup on F5 VE

1. 创建 partition
create auth partition kubernetes
  • kubernetes - Partition 的名称需要与 CIS Deployment 中--bigip-partition参数指定的partition一致。

2. 创建 VxLan 隧道
create net tunnels vxlan fl-vxlan port 8472 flooding-type none
create net tunnels tunnel flannel_vxlan key 1 profile fl-vxlan local-address 192.168.200.208
create net self 10.244.20.100 address 10.244.20.100/16 allow-service none vlan flannel_vxlan
save sys config
show net tunnels tunnel flannel_vxlan all-properties
  • 192.168.200.208 – F5 VE 上VS业务VLAN的 Self IP或管理IP,K8S上创建 BIG-IP Node 时会使用到改地址,对应 flannel flannel.alpha.coreos.com/backend-data标注。

  • 10.244.20.100/16 – K8S Flannel vxlan self IP, 必需是 16 位,这样才可以确保所有目的网络都走vtep

  • show net tunnels tunnel flannel_vxlan all-properties 会输出 tunnel vxlan interface MAC 地址,该地址会在K8S上创建 BIG-IP Node 时使用,对应 flannel flannel.alpha.coreos.com/public-ip 标注。

Setup on K8s

1. Install
kubectl create ns bigip-ctlr
kubectl create secret generic bigip-login --from-literal=username=admin --from-literal=password=admin -n bigip-ctlr
kubectl create serviceaccount bigip-ctlr -n bigip-ctlr
kubectl create -f rbac.yaml
kubectl create -f cis.yaml
2. 创建 Big-IP Node
kubectl apply -f bigip-node.yaml
3. 查看日志
kubectl logs -f $(kubectl get pods -n bigip-ctlr --no-headers | awk '{print $1}') -n bigip-ctlr

部署测试应用

1. 部署测试应用
kubectl apply -f deploy.yaml
2. 创建 CIS Configmap
kubectl apply -f cm.yaml
3. 查看日志输出
$ kubectl logs -f $(kubectl get pods -n bigip-ctlr --no-headers | awk '{print $1}') -n bigip-ctlr
...
2021/05/12 11:27:27 [INFO] [CCCL] Successfully Sent the FDB Records
2021/05/12 11:27:29 [INFO] [2021-05-12 11:27:29,320 f5_cccl.resource.resource INFO] Updating ApiFDBTunnel: /Common/flannel_vxlan
2021/05/12 11:27:45 [INFO] [2021-05-12 11:27:45,726 f5_cccl.resource.resource INFO] Creating ApiArp: /Common/k8s-10.244.1.47
2021/05/12 11:31:06 [INFO] [CCCL] Successfully Sent the FDB Records
2021/05/12 11:31:24 [INFO] [2021-05-12 11:31:24,519 f5_cccl.resource.resource INFO] Creating ApiArp: /Common/k8s-10.244.1.49

TroubleShoting

1. FDB
# show net fdb

---------------------------------------------------------------------
Net::FDB
Tunnel         Mac Address        Member                      Dynamic
---------------------------------------------------------------------
flannel_vxlan  16:73:97:7f:83:28  endpoint:192.168.200.210%0  no
flannel_vxlan  e2:85:68:42:be:b7  endpoint:192.168.200.211%0  no
2. ARP
# show net arp

-------------------------------------------------------------------------------------------------------
Net::Arp
Name                     Address          HWaddress          Vlan               Expire-in-sec  Status
-------------------------------------------------------------------------------------------------------
/Common/k8s-10.244.1.47  10.244.1.47      e2:85:68:42:be:b7  -                  -              static
/Common/k8s-10.244.1.49  10.244.1.49      e2:85:68:42:be:b7  -                  -              static
3. 抓包
tcpdump -nni flannel_vxlan icmp
tcpdump -nni  0.0:nnnp -s0 host ip and port  -w  /var/tmp/zhuabao.pcap  –vvv

访问服务

$ curl 192.168.200.34/coffee
Server address: 10.244.1.47:8080
Server name: app-1-7f4585dc79-l6sx4
Date: 12/May/2021:11:35:07 +0000
URI: /coffee
Request ID: 03f319a0fde3233c478e16e34115386e

results matching ""

    No results matching ""