Using Octavia (load balancing) with Kubernetes (k8s)
In Kubernetes objects of type LoadBalancer are an essential part for making applications accessible from the outside world. Of course, there are alternative methods like NodePort but only with a service of type LoadBalancer one can easily configure an ingress-controller to handle incoming traffic for different DNS-Names for the same external IP address and have incoming traffic automatically routed to the correct pods.
Following lines describe how to integrate the Openstack Cloud Controller Manager (OCCM) to your cluster. After the successful integration, whenever you create a service of type LoadBalancer, Oopenstack will spawn an amphora VM connect it to the appropriate network and make your service accessible via an external IP address. All resources will be deployed in the kube-system namespace.
Create an application secret
Create an application secret for the OCCM to be able to connect to Openstack.
openstack application credential create --description "Kubernetes" kubernetes --secret **********
Create a configuration secret
For the Kubernetes side of the communication you need to create a configuration secret which will contain the application id and secret (and a bit more) and will be mounted into the OCCM-Containers.
- Create a file called
cloud.confwhich contains at least the following:
[Global] auth-url=https://cc.lrz.de:5000 application-credential-id=XXXXXXXXXXXXXXXX application-credential-secret=XXXXXXXXXXXXXXXX [LoadBalancer] use-octavia=true floating-network-id=XXXXXXXXXXXXXXXXX subnet-id=XXXXXXXXXXXXXXXXXXXXXXXXX
auth-url: self explanatory
application-credential-id: The id of the application credential you just created
application-credential-secret: The secret of the created application credential
floating-network-id: The ID of the external Openstack network. Eg.:
3f1c6c34-2be9-44b3-9f21-c3e031ab8e5cfor the MWN.subnet-id: The ID of the external pool to use. Eg.
a3e4d020-c8b4-48b5-beb1-5f0d47d06ed7for MWN addresses.Create the configuration secret based on the file (make sure to have the name right!)
kubectl create secret -n kube-system generic cloud-config --from-file=cloud.conf
Rollout the OCCM
Execute following commands in the context of your cluster to deploy the Openstack Controller Manager
kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-roles.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/cloud-controller-manager-role-bindings.yaml kubectl apply -f https://raw.githubusercontent.com/kubernetes/cloud-provider-openstack/master/manifests/controller-manager/openstack-cloud-controller-manager-ds.yaml
The OCCM will now be deployed as a daemonset. Note, that by default it will prefer the control-plane for scheduling. So if you have non schedulable control-planes, adjust the nodeSelector in the daemonset manifest accordingly or just remove the entire two lines:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
in the daemonset manifest.
Test functionality
If everything is configured correctly OCCM will now create an amphora-VM in Openstack whenever you create an object of type LoadBalancer. In the following we create a simple test deployment with three replicas. Each replica will simply return the hostname of the underling container.
- Create a file containing the declaration of the deployment and the service:
cat << 'EOF' >> test-octavia.yaml
> ---
apiVersion: apps/v1
kind: Deployment
metadata:
name: octavia-test-deployment
spec:
replicas: 3
selector:
matchLabels:
app: occm-test
template:
metadata:
labels:
app: occm-test
spec:
containers:
- name: occm-test
image: lingxiankong/alpine-test
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: occm-test-svc
spec:
selector:
app: occm-test
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 8080
> EOF
Create the resources:
kubectl apply -f test-octavia.yamlWait for the loadbalancer object to be created and get an ip assigned via octavia
kubectl get svc/occm-test-svc -w NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE occm-test-svc LoadBalancer 10.240.19.110 <pending> 80:31833/TCP 35s
After the process is completed it should look similar to:
kubectl get svc/occm-test-svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE occm-test-svc LoadBalancer 10.240.19.110 10.195.11.99 80:31833/TCP 2m9s
Note: The service now got an external IP assigned.
Verify actual loadbalancing
The functionality can easily be verified via curl. Just curl the external IP in a loop an see how the different containers respond to the requests.
for i in {1..27}; do curl http://10.195.11.99; echo -n; done
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-7xl2k
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-7xl2k
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-7xl2k
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-lvslp
octavia-test-deployment-86d647c98d-7xl2k
octavia-test-deployment-86d647c98d-z5nzj
octavia-test-deployment-86d647c98d-7xl2k
octavia-test-deployment-86d647c98d-lvslp
We can also see that the traffic gets balanced to the different pods nearly equally:
for i in {1..27}; do curl http://10.195.11.99; echo -n; done &>/dev/null | sort | uniq -c | sort -nr
11 octavia-test-deployment-86d647c98d-7xl2k
8 octavia-test-deployment-86d647c98d-z5nzj
8 octavia-test-deployment-86d647c98d-lvslp