Introduction

Charmed Kubernetes in Production on Ubuntu , offered by Ubuntu, is an enterprise-grade Kubernetes distribution designed to streamline the deployment, scaling, and management of Kubernetes clusters. This solution leverages the power of Juju, Ubuntu’s model-driven operations tool, to deliver a robust and flexible infrastructure for running containerized applications. With Charmed Kubernetes, organizations can achieve a higher level of automation and operational efficiency, making it an ideal choice for businesses looking to optimize their cloud-native environments.

In addition to its operational capabilities, Charmed Kubernetes benefits from Ubuntu’s extensive support network. Users can access a wealth of documentation, community-driven resources, and enterprise-grade support services, ensuring that any challenges can be addressed promptly and effectively. This combination of powerful tooling and reliable support makes Charmed Kubernetes a compelling option for organizations aiming to enhance their Kubernetes operations.

Benefits

• Ease of Deployment: Charmed Kubernetes automates the complex setup of Kubernetes clusters, making it accessible even for teams with limited Kubernetes expertise.
• Scalability and Flexibility: With Juju, you can easily scale your cluster and customize deployments to meet specific needs, ensuring your infrastructure grows with your application.
• Seamless Integration: It integrates smoothly with a variety of cloud environments, offering a consistent experience across on-premises, public, and hybrid clouds.
• Comprehensive Support: Backed by Ubuntu, users benefit from extensive documentation, community support, and enterprise-grade service options.

Downsides

• Learning Curve: While Juju simplifies many aspects of Kubernetes management, it introduces its own set of concepts and tools that users need to learn.
• Complexity of Customization: Despite its flexibility, tailoring Charmed Kubernetes to very specific use cases can require significant expertise and effort.
• Potential for Overhead: The additional layers of abstraction introduced by Juju can sometimes lead to performance overhead, which might affect the efficiency of the cluster

This expanded introduction provides a more comprehensive overview of the advantages and potential drawbacks of using Charmed Kubernetes.

Lab Setup 

The diagram illustrates a typical deployment architecture for Charmed Kubernetes using Juju, highlighting the key components and their interactions.

Juju CLI on Laptop: The process begins with a user operating a laptop equipped with the Juju Command Line Interface (CLI). . The Juju CLI allows users to define, deploy, and manage their Kubernetes cluster efficiently. https://juju.is/docs/juju/install-juju

SSH Connection: The Juju CLI communicates with the Kubernetes environment via a secure SSH connection. This connection ensures that commands and configurations are transmitted securely from the user’s laptop to the Kubernetes controller.

Controller: The controller is a critical component managed by Juju. It orchestrates the deployment and scaling of Kubernetes clusters, ensuring that all nodes are properly configured and functioning. Additionally, the controller handles the lifecycle of the Kubernetes cluster, automating tasks such as updates, scaling, and monitoring.

Ubuntu Servers : We will use 6 Ubuntu servers for our installation , 3 master & 3 workers. 2 cores / 4GB RAM / 150GB Disk. Specifically the server role will be determined later in the process.

Reference architecture – Ubuntu Charmed Kubernetes

Install Charmed Kubernetes on Ubuntu

At first, before installation please validate ssh keys are copied for all managed servers.

$ for i in 34.228.186.208 54.146.161.19 54.226.81.66 18.215.169.93 34.202.237.50 34.224.81.74 ; do cat ~/.ssh/id_rsa.pub | ssh -i kirson-dev.pem ubuntu@$i  'cat >> .ssh/authorized_keys' ; done

First part is to create a controller server & model  for installation – needed by charmed

$ juju bootstrap manual/ubuntu@54.237.197.239 production-baremetal
$ juju add-model production-baremetalJuju models # ( view we are in the production-baremetal env )

Using the juju add-machine command – we will add all servers to juju machine configuration.

$ for i in 34.228.186.208 54.146.161.19 54.226.81.66 18.215.169.93 34.202.237.50 34.224.81.74 ; do  juju add-machine  ssh:ubuntu@$i ; done

Production configuration

Using the Juju machines command we can list servers available for our deployment
In this case, we will use servers 0,1 and 2 as masters and 3,4 and 5 as workers, 

$ juju machines
Machine  State    Address         Inst id                Base          AZ  Message
0        started  34.228.186.208  manual:34.228.186.208  ubuntu@22.04      Manually provisioned machine
1        started  54.146.161.19   manual:54.146.161.19   ubuntu@22.04      Manually provisioned machine
2        started  54.226.81.66    manual:54.226.81.66    ubuntu@22.04      Manually provisioned machine
3        started  18.215.169.93   manual:18.215.169.93   ubuntu@22.04      Manually provisioned machine
4        started  34.202.237.50   manual:34.202.237.50   ubuntu@22.04      Manually provisioned machine
5        started  34.224.81.74    manual:34.224.81.74    ubuntu@22.04      Manually provisioned machine

deploy.yaml

The file included the exact configuration needed for charmed –
Pay attention to “machines” & count section. This is where you can specify GPU servers. In this case, GPU server will be added later

description: A minimal two-machine Kubernetes cluster, appropriate for development.
issues: https://bugs.launchpad.net/charmed-kubernetes-bundles
description: A minimal two-machine Kubernetes cluster, appropriate for development.
issues: https://bugs.launchpad.net/charmed-kubernetes-bundles
series: jammy
source: https://github.com/charmed-kubernetes/bundle
website: https://ubuntu.com/kubernetes/charmed-k8s
name: kubernertes-core
machines:
  '0':
  '1':
  '2':
  '3':
  '4':
  '5':
applications:
  calico:
    channel: stable
    charm: calico
    options:
      vxlan: Always
  containerd:
    channel: stable
    charm: containerd
    options:
      gpu_driver: none
  easyrsa:
    channel: stable
    charm: easyrsa
    num_units: 1
    to:
      - '1'
  etcd:
    channel: stable
    charm: etcd
    num_units: 3
    options:
      channel: 3.4/stable
    to:
      - '0'
      - '1'
      - '2'
  kubernetes-control-plane:
    channel: stable
    charm: kubernetes-control-plane
    expose: true
    num_units: 3
    options:
      channel: 1.29/stable
    to:
      - '0'
      - '1'
      - '2'
  kubernetes-worker:
    channel: stable
    charm: kubernetes-worker
    expose: true
    num_units: 3
    options:
      channel: 1.29/stable
    to:
      - '3'
      - '4'
      - '5'
relations:
  - - kubernetes-control-plane:kube-control
    - kubernetes-worker:kube-control
  - - kubernetes-control-plane:certificates
    - easyrsa:client
  - - kubernetes-control-plane:etcd
    - etcd:db
  - - kubernetes-worker:certificates
    - easyrsa:client
  - - etcd:certificates
    - easyrsa:client
  - - calico:etcd
    - etcd:db
  - - calico:cni
    - kubernetes-control-plane:cni
  - - calico:cni
    - kubernetes-worker:cni
  - - containerd:containerd
    - kubernetes-worker:container-runtime
  - - containerd:containerd
    - kubernetes-control-plane:container-runtim

Start Charmed Kubernetes installation

Let’s start the installation by using the command below will install ALL servers automatically based on the YAML file.

$ juju deploy ./deploy.yaml --map-machines=existing

Monitor Installation process

$ watch -c juju status --color
$ juju debug-log

Connecting to our Kubernetes Cluster

$ juju ssh kubernetes-control-plane/leader -- cat config > ./kubeconfig-charmed
$ export KUBECONFIG=./kubeconfig-charmed
$ kubectl get nodes
NAME             STATUS   ROLES           AGE     VERSION
ip-10-0-17-202    Ready    control-plane   7m57s   v1.29.6
ip-10-0-21-212    Ready    <none>          9m20s   v1.29.6
ip-10-0-23-206   Ready    <none>          9m20s   v1.29.6
ip-10-0-24-213    Ready    control-plane   9m56s   v1.29.6
ip-10-0-27-126    Ready    <none>          9m20s   v1.29.6
ip-10-0-31-186    Ready    control-plane   9m27s   v1.29.6

For additional information feel free to use Ubuntu official documentation

Adding additional GPU server 

Using juju CLI – we can easily add additional servers to our cluster ( validate ssh keys before running the command )

$ juju add-machine ssh:ubuntu@54.90.206.107 
$ juju machines # ( use the last index number )
$ juju add-unit kubernetes-worker --to 6

Advanced day 2 config

I have included the following day-2 operations for your to be able to work with this cluster once deployed. One could do other day-2 operations if needed.

Configure kube-api for Charmed Kubernetes in Production

We can use the command below to configure the kube-api server – example for external auth

$ juju config kubernetes-control-plane api-extra-args="oidc-client-id=ai oidc-issuer-url=https://ai.com/auth/realms/site  oidc-username-prefix=-

Install Prometheus

$ helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
$ helm repo update
$ helm install prometheus prometheus-community kube-prometheus-stack -n monitoring --create-namespace --set grafana.enabled=false

Install NGINX

In this case – we are using the external & internal IP address for NGINX – easily expose web application. NOTE: LoadBalancer / MetaLB is recommended.

$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginxhelm repo updatehelm upgrade -i nginx-ingress ingress-nginx/ingress-nginx  --namespace nginx-ingress \
--create-namespace \
--set controller.kind=DaemonSet \-
 -set controller.service.externalIPs="{10.0.21.212,18.215.169.93}"

Install GPU Operator

This will install the latest GPU Operator 

$ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia  
$ helm repo update 
$ helm install --wait --generate-name  -n gpu-operator --create-namespace    nvidia/gpu-operator

Create NFS CSI Driver

$ curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-nfs/v4.4.0/deploy/install-driver.sh | bash -s v4.4.0 --

External NFS Server

NOTE: In case you don’t have an NFS server – example of NFS server configuration or you could Use OpenEBS with hostPath on Kubernetes

$ sudo apt-get install nfs-kernel-server
$ sudo mkdir /nfs
$ sudo chown 1001:1001 /nfs
$ sudo chmod g+s /nfs
Add this line to /etc/exports
/nfs      *(rw,sync,no_subtree_check,all_squash,anonuid=1001,anongid=1001)
exportfs -r ( reload the file )

Create NFS StorageClass

$ cat <<EOF > sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-csi  
  annotations:   
       storageclass.kubernetes.io/is-default-class: "true"
provisioner: nfs.csi.k8s.io
parameters:
  server: 10.0.17.114
  share: /nfs
reclaimPolicy: Delete
volumeBindingMode: Immediate
mountOptions:
  - nfsvers=4.1
EOF
$ kubectl create -f sc.yaml

Now you can start changing the world with your new Charmed Kubernetes in Production on Ubuntu

Summary

In conclusion, Charmed Kubernetes in Production on Ubuntu offers a powerful and scalable solution for managing containerized applications. With Juju’s simplified deployment and management, it provides efficiency and reliability, despite some initial learning curves and resource requirements. The benefits of automated operations and multi-cloud flexibility make it an excellent choice for DevOps teams aiming to enhance their Kubernetes capabilities. Charmed Kubernetes empowers teams to focus on innovation, ensuring faster and more reliable application delivery.