Installing Kubernetes (k3s) on Proxmox LXC container

Create a privileged container in Proxmox

Make sure container doesn’t start right after creation

Append extra config for to lxc container conf file

On the proxmox host edit the file /etc/pve/lxc/<container id>.conf and add the following content

lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"

Create missing /dev/kmsg file

Start the container and inside the container create the file /etc/rc.local if it doesnt exist. Add the following content to the file and update the file permissions like so:

cat <<EOF > /etc/rc.local
#!/bin/sh -e
if [ ! -e /dev/kmsg ]; then
    ln -s /dev/console /dev/kmsg
fi
mount --make-rshared /
EOF

chmod +x /etc/rc.local
/etc/rc.local

Install K3s

curl -sfL https://get.k3s.io | sh -s - --disable=traefik --disable=servicelb --node-name control.k8s

# Setup kubectl for non-root user access
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
source ~/.bashrc
mkdir ~/.kube 2> /dev/null
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
# Test to make sure non-root kubectl is working
kubectl get nodes

Install Helm

# Install Helm
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
helm version

Install MetalLB

# Install MetalLB
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb --namespace metallb-system --create-namespace

cat <<EOF > metallb_config.yaml
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: nginx-ingress-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.10.10.100/32
  autoAssign: false
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: regular-lb-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.10.10.110-10.10.10.120
  autoAssign: false
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-l2advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - nginx-ingress-pool
  - regular-lb-pool
EOF

kubectl apply -f metallb_config.yaml
# Ensure the address pools are well created
kubectl describe ipaddresspools.metallb.io -n metallb-system

Install Nginx Ingress Controller

# Install NGINX ingress controller
cat <<EOF > nginx-ingress_values.yaml
controller:
    service:
        annotations:
            metallb.universe.tf/address-pool: nginx-ingress-pool
EOF

helm install nginx-ingress-release --values app1/nginx-ingress_values.yaml --create-namespace --namespace nginx-ingress oci://ghcr.io/nginxinc/charts/nginx-ingress --version 1.4.0 --values app1/nginx-ingress_values.yaml # --set-json 'controller.service.annotations={"test1": "de"}'
helm upgrade nginx-ingress-release --values app1/nginx-ingress_values.yaml -n nginx-ingress oci://ghcr.io/nginxinc/charts/nginx-ingress

# Ensure that the supplied values are taken into account
helm get values -n nginx-ingress nginx-ingress-release
# Uninstall to reinstall again if needed
helm -n nginx-ingress uninstall nginx-ingress-release
# Check the nginx-ingress controller service is provided an IP address by MetalLB
kubectl describe svc -n nginx-ingress nginx-ingress-release-controller

Install cert-manager

helm repo add jetstack https://charts.jetstack.io --force-update

helm install cert-manager jetstack/cert-manager --create-namespace --namespace cert-manager --version v1.16.0 --set crds.enabled=true --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'

Sources:

• https://kevingoos.medium.com/kubernetes-inside-proxmox-lxc-cce5c9927942

• https://bobcares.com/blog/rancher-lxc-proxmox/

• https://docs.k3s.io/quick-start#install-script

• https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#bash

• https://devops.stackexchange.com/questions/16043/error-error-loading-config-file-etc-rancher-k3s-k3s-yaml-open-etc-rancher

• https://helm.sh/docs/intro/install/

• https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/

• https://blog.mossroy.fr/2024/04/16/passage-sur-metallb-au-lieu-de-servicelb-sur-un-cluster-k3s/

• https://metallb.universe.tf/installation/

• https://github.com/metallb/metallb/issues/1964

• https://blog.mossroy.fr/2024/04/16/passage-sur-metallb-au-lieu-de-servicelb-sur-un-cluster-k3s/

• https://cert-manager.io/docs/installation/helm/