# Installing Kubernetes (k3s) on Proxmox LXC container

## Create a privileged container in Proxmox

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728214716729/18e877df-308f-4701-af57-daae112c3b38.png align="center")

## Make sure container doesn’t start right after creation

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1728214729962/3895236e-b577-4267-b80e-59faa1e34b57.png align="center")

Append extra config for to lxc container conf file

On the proxmox host edit the file `/etc/pve/lxc/<container id>.conf` and add the following content

```plaintext
lxc.apparmor.profile: unconfined
lxc.cgroup.devices.allow: a
lxc.cap.drop:
lxc.mount.auto: "proc:rw sys:rw"
```

## Create missing /dev/kmsg file

Start the container and inside the container create the file `/etc/rc.local` if it doesnt exist. Add the following content to the file and update the file permissions like so:

```bash
cat <<EOF > /etc/rc.local
#!/bin/sh -e
if [ ! -e /dev/kmsg ]; then
    ln -s /dev/console /dev/kmsg
fi
mount --make-rshared /
EOF

chmod +x /etc/rc.local
/etc/rc.local
```

## Install K3s

```bash
curl -sfL https://get.k3s.io | sh -s - --disable=traefik --disable=servicelb --node-name control.k8s

# Setup kubectl for non-root user access
echo 'export KUBECONFIG=~/.kube/config' >> ~/.bashrc
echo 'source <(kubectl completion bash)' >>~/.bashrc
echo 'alias k=kubectl' >>~/.bashrc
echo 'complete -o default -F __start_kubectl k' >>~/.bashrc
source ~/.bashrc
mkdir ~/.kube 2> /dev/null
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"
# Test to make sure non-root kubectl is working
kubectl get nodes
```

## Install Helm

```bash
# Install Helm
curl https://baltocdn.com/helm/signing.asc | gpg --dearmor | sudo tee /usr/share/keyrings/helm.gpg > /dev/null
sudo apt-get install apt-transport-https --yes
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/helm.gpg] https://baltocdn.com/helm/stable/debian/ all main" | sudo tee /etc/apt/sources.list.d/helm-stable-debian.list
sudo apt-get update
sudo apt-get install helm
helm version
```

## Install MetalLB

```bash
# Install MetalLB
helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb --namespace metallb-system --create-namespace

cat <<EOF > metallb_config.yaml
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: nginx-ingress-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.10.10.100/32
  autoAssign: false
---
apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: regular-lb-pool
  namespace: metallb-system
spec:
  addresses:
  - 10.10.10.110-10.10.10.120
  autoAssign: false
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: default-l2advertisement
  namespace: metallb-system
spec:
  ipAddressPools:
  - nginx-ingress-pool
  - regular-lb-pool
EOF

kubectl apply -f metallb_config.yaml
# Ensure the address pools are well created
kubectl describe ipaddresspools.metallb.io -n metallb-system
```

## Install Nginx Ingress Controller

```bash
# Install NGINX ingress controller
cat <<EOF > nginx-ingress_values.yaml
controller:
    service:
        annotations:
            metallb.universe.tf/address-pool: nginx-ingress-pool
EOF

helm install nginx-ingress-release --values app1/nginx-ingress_values.yaml --create-namespace --namespace nginx-ingress oci://ghcr.io/nginxinc/charts/nginx-ingress --version 1.4.0 --values app1/nginx-ingress_values.yaml # --set-json 'controller.service.annotations={"test1": "de"}'
helm upgrade nginx-ingress-release --values app1/nginx-ingress_values.yaml -n nginx-ingress oci://ghcr.io/nginxinc/charts/nginx-ingress

# Ensure that the supplied values are taken into account
helm get values -n nginx-ingress nginx-ingress-release
# Uninstall to reinstall again if needed
helm -n nginx-ingress uninstall nginx-ingress-release
# Check the nginx-ingress controller service is provided an IP address by MetalLB
kubectl describe svc -n nginx-ingress nginx-ingress-release-controller
```

## Install cert-manager

```bash
helm repo add jetstack https://charts.jetstack.io --force-update

helm install cert-manager jetstack/cert-manager --create-namespace --namespace cert-manager --version v1.16.0 --set crds.enabled=true --set 'extraArgs={--dns01-recursive-nameservers-only,--dns01-recursive-nameservers=8.8.8.8:53\,1.1.1.1:53}'
```

**Sources:**

* [https://kevingoos.medium.com/kubernetes-inside-proxmox-lxc-cce5c9927942](https://kevingoos.medium.com/kubernetes-inside-proxmox-lxc-cce5c9927942)
    
* [https://bobcares.com/blog/rancher-lxc-proxmox/](https://bobcares.com/blog/rancher-lxc-proxmox/)
    
* [https://docs.k3s.io/quick-start#install-script](https://docs.k3s.io/quick-start#install-script)
    
* [https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#bash](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/#bash)
    
* [https://devops.stackexchange.com/questions/16043/error-error-loading-config-file-etc-rancher-k3s-k3s-yaml-open-etc-rancher](https://devops.stackexchange.com/questions/16043/error-error-loading-config-file-etc-rancher-k3s-k3s-yaml-open-etc-rancher)
    
* [https://helm.sh/docs/intro/install/](https://helm.sh/docs/intro/install/)
    
* [https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/](https://docs.nginx.com/nginx-ingress-controller/installation/installing-nic/installation-with-helm/)
    
* [https://blog.mossroy.fr/2024/04/16/passage-sur-metallb-au-lieu-de-servicelb-sur-un-cluster-k3s/](https://blog.mossroy.fr/2024/04/16/passage-sur-metallb-au-lieu-de-servicelb-sur-un-cluster-k3s/)
    
* [https://metallb.universe.tf/installation/](https://metallb.universe.tf/installation/)
    
* [https://github.com/metallb/metallb/issues/1964](https://github.com/metallb/metallb/issues/1964)
    
* [https://blog.mossroy.fr/2024/04/16/passage-sur-metallb-au-lieu-de-servicelb-sur-un-cluster-k3s/](https://blog.mossroy.fr/2024/04/16/passage-sur-metallb-au-lieu-de-servicelb-sur-un-cluster-k3s/)
    
* [https://cert-manager.io/docs/installation/helm/](https://cert-manager.io/docs/installation/helm/)