= Using Kubernetes on Fedora
Bradley G Smith
:revnumber: F37,F38,F39,rawhide
:revdate: 2023-09-19
:category: Installation
:tags: How-to, kubernetes, dnf, rpm, containers

// Optional  free form useful additional information as comment

//include::{partialsdir}/3rdparty-message.adoc[]
include::partial$3rdparty-message.adoc[]

[[sect-overview]]
== Overview

This how-to provides an overview of the link:https://kubernetes.io[Kubernetes] (K8s) rpms in the Fedora repositories, how to use them in a few scenarios and a short cluster creation guide using `kubeadm` on a single Fedora machine.
The guide also touches on an alternative source for Kubernetes rpms available in link:https://copr.fedorainfracloud.org[COPR] and potential benefits.

[[sect-what-is-kubernetes]]
=== What is Kubernetes?

link:https:/kubernetes.io[Kubernetes] is an "open-source system for automating deployment, scaling, and management of containerized applications" on one or more machines.
Kubernetes had its genesis in the concepts and principles used at Google to run container-base workloads at scale and with resilience.
Kubernetes is now at the center of a vast ecosystem of products and services (link:https://cncf.io/[Cloud Native Computing Foundation]) that help organizations create, install, run, manage and secure container-based applications and services at any possible scale.

There are numerous ways to install and configure Kubernetes depending on purpose and target environment.
Is this for a home lab on a single machine, a small cluster for home or business automation, edge-based services and applications in remote offices or enterprise scale production workloads in the cloud?
This guide is narrowly focused on the Kubernetes rpms available from Fedora and using `dnf` and the command line to install these rpms on Fedora and create a basic cluster using `kubeadm`.

[[sect-terminology]]
=== Terminology

Kubernetes is complex and like many complex systems has its own terminology.
The terminology used in this guide are defined here.
The Kubernetes teams maintains a comprehensive link:https://kubernetes.io/docs/reference/glossary/[glossary] which is used in the subset below.

[horizontal]
cluster:: a set of one or more nodes managed as an entity.
A cluster has at least one node and one control plane (these can be on the same or separate machines).
control plane:: the container orchestration layer in a cluster which manages the pods in the cluster.
At least one node in a cluster has a control plane.
node:: a worker machine (either a virtual machine or physical machine) in a Kubernetes cluster that has the services required to run pods.
These services include the `kubelet` container runtime and `kube-proxy`.
pods:: containerized applications are deployed and managed in Kubernetes as pods.
A pod is the base object managed by Kubernetes in a cluster.
A pod typically has a single primary container but may include more capabilities.

[[sect-kubernetes-rpms]]
== Kubernetes rpms in Fedora

The number, name, and organization of content in Fedora Kubernetes rpms depends on the Fedora release.
Fedora 40 and newer releases will have one set of rpms (link:https://fedoraproject.org/wiki/Changes/RestructureKubernetesPackages[pending change proposal for Fedora 40]).
Fedora 39 and older releases have the legacy set of rpms.

[[sect-fedora-39-and-older]]
=== Fedora 39 and older releases

The table below lists the available Kubernetes rpms in Fedora 39 and older releases, what the rpm contains, and notes on purpose and any restrictions or cautions.

.Kubernetes rpms in Fedora 39 (and older)
[cols="1,1,1", options="header"]
|===
|RPM Name |Contents |Notes
|kubernetes
|Empty
|Also installs kubernetes-node and kubernetes-master.

|kubernetes-client
|kubectl
|Kubernetes command line client.
Recommended on any node configured as a control plane as it allows the cluster administrator control over the cluster from an ssh session on the control plane.
Install on a machine that can connect to the cluster over the network.

|kubernetes-kubeadm
|kubeadm
|Bootstraps Kubernetes on a node.
Required on each node.

|kubernetes-master
|kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler
|Systemd services for a kubernetes node.
Not needed for most installations as kubeadm will install these components as static pods.
If used, then install on each node.
Also installs kubernetes-client.

|kubernetes-node
|kubelet
|Kubernetes runtime on a node.
Required on each node.
|===

[[sect-fedora-39-recommendations]]
==== Fedora 39 (and older) Installation recommendations

For most modern kubernetes clusters install kubernetes-node, kubernetes-kubeadm, and kubernetes-client on each machine in the cluster.
If disk space is a constraint only install kubernetes-client on control-plane machines.

[source,bash]
----
sudo dnf install kubernetes-kubeadm kubernetes-node kubernetes-client
----

If conducting a manual installation of Kubernetes (see link:https://github.com/kelseyhightower/kubernetes-the-hard-way[Kubernetes The Hard Way]) then install kubernetes-master and kubernetes-kubeadm.

[source,bash]
----
sudo dnf install kubernetes-master kubernetes-kubeadm
----
////
[[sect-fedora40-and-newer]]
=== Fedora 40 and newer releases

Kubernetes rpms have been reorganized starting with Fedora 40 (rawhide in Sept 2023).
The table below lists the available Kubernetes rpms, what the rpm contains, and notes on purpose and any cautions or restrictions.

.Kubernetes rpms in Fedora 40 (and newer)
[cols="1,1,1", options="header"]
|===
|RPM Name|Contents|Notes
|kubernetes
|kubeadm, kubelet
|Kubeadm bootstraps the cluster on a node.
Kubelet is the Kubernetes runtime on a node.
This rpm is required on every Fedora machine in a cluster.

|kubernetes-client
|kubectl
|Kubernetes command line client.
Recommended on any node configured as a control plane as it allows the cluster administrator control over the cluster from an ssh session on the control plane.
Install on a machine that can connect to the cluster over the network.

|kubernetes-legacy-systemd
|kube-apiserver, kube-controller-manager, kube-proxy, kube-scheduler
|Systemd services for a kubernetes control-plane and/or node.
Not needed for most installations as kubeadm will install these components as static pods.
If used, then install on all nodes.
Use systemctl to enable kube-proxy on all nodes. Enable kube-apiserver, kube-controller-manager, and kube-scheduler on control plane nodes.

|===

[[sect-fedora-40-recommendations]]
==== Fedora 40 (and newer) installation recommendations

For most modern kubernetes clusters install kubernetes, and kubernetes-client on each machine in the cluster.
If disk space is a constraint only install kubernetes-client on control-plane machines.

[source,bash]
----
sudo dnf install kubernetes kubernetes-client
----

If conducting a manual installation of Kubernetes (see link:https://github.com/kelseyhightower/kubernetes-the-hard-way[Kubernetes The Hard Way]) then install all kubernetes rpms.

[source,bash]
----
sudo dnf install kubernetes kubernetes-kubeadm kubernetes-control-plane-services kubernetes-node-services kubernetes-client
----
////
[[sect-kubernetes-fedora-crosswalk]]
== Kubernetes and Fedora version crosswalk

Each Fedora release has a corresponding version of Kubernetes available as listed below.
The goal is to provide the most current Kubernetes release available when a Fedora release reaches General Availability (GA).
This is not always possible resulting in skipped Kubernetes releases. Skipping a release causes problems for Kubernetes cluster administrators given the Kubernetes cluster upgrade process.
Alternative ways to package Kubernetes for Fedora are being explored.

The version of the Go programming language supported for a given Fedora release can also limit the version of Kubernetes available if Kubernetes requires a newer version of Go.

.Kubernetes versions and the corresponding Fedora release
[cols="1,1,1,1", options="header"]
|===
|Kubernetes Version |Target Fedora Release | Kubernetes End-of-Life | Kubernetes Golang 'Built-With' Version
|1.28
|F40
|2024.10.28
|1.20

|1.27
|F39
|2024.06.28
|1.20

|1.26
|F38
|2024.02.24
|1.20 (was 1.19)

|1.25
|COPR/F37^1^
|2023.10.27
|1.20 (was 1.19)
|===

^1^ F37 provides Go version 1.19. When Kubernetes switched to Go 1.20 (1.25.12) a link:https://copr.fedorainfracloud.org/coprs/buckaroogeek/copr-k8s-1.25/[COPR project] was created for more recent versions of Kubernetes 1.25.

[cluster-creation]
== Creating a Kubernetes cluster with kubeadm using Fedora rpms

Below is a guide to creating a Kubernetes cluster on a single Fedora machine.
Each Fedora release has a corresponding Kubernetes release as documented at the link:https://src.fedoraproject.org/rpms/kubernetes[Fedora Package Sources repository for Kubernetes].
Fedora 39, for example, has rpms for Kubernetes 1.27.
The cluster initialization is the same for all current Fedora releases.
These instructions have been tested on Fedora 38 and Fedora 39 virtual machines and on Raspberry Pi 4 hardware running Fedora 38 and Fedora 39 minimal.
The guide below generally follows the link:https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/[Creating a cluster with kubeadm] guide.

. Update system with DNF.
Reboot if necessary, although a reboot can be deferred until after the next step.
+
[source,bash]
----
sudo dnf update
----

. Disable swap.
Kubernetes is configured to generate an installation error if swap is detected (see link:https://github.com/kubernetes/kubernetes/issues/53533[this ticket for details]).
Modern Fedora systems use zram by default.
Reboot after disabling swap.
+
[source,bash]
----
sudo systemctl stop swap-create@zram0
sudo dnf remove zram-generator-defaults
sudo reboot now
----

. Disable the firewall.
Kubernetes is configured to generate an installation error if the firewall is running.
Modern Fedora systems use firewalld.
See link:https://devopstales.github.io/kubernetes/k8s-security[https://devopstales.github.io/kubernetes/k8s-security] for an alternative solution that retains the firewall and opens necessary ports.
+
[source,bash]
----
sudo systemctl stop firewalld.system
sudo systemctl disable firewalld.system
----

. Install `iptables` and `iproute-tc.`
+
[source,bash]
----
sudo dnf install iptables iproute-tc
----

. Configure IPv4 forwarding and bridge filters.
Below copied from link:https://kubernetes.io/docs/setup/production-environment/container-runtimes/[https://kubernetes.io/docs/setup/production-environment/container-runtimes/]
+
[source,bash]
----
sudo cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
----

. Load the overlay and bridge filter modules.
+
[source,bash]
----
sudo modprobe overlay
sudo modprobe br_netfilter
----

. Add required `sysctl` parameters and persist.
+
[source,bash]
----
# sysctl params required by setup, params persist across reboots
sudo cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
----

. Apply `sysctl` parameters without a reboot.
+
[source,bash]
----
sudo sysctl --system
----

. Verify `br_filter` and overlay modules are loaded.
+
[source,bash]
----
lsmod | grep br_netfilter
lsmod | grep overlay
----

. Verify that the `net.bridge.bridge-nf-call-iptables`, `net.bridge.bridge-nf-call-ip6tables`, and `net.ipv4.ip_forward` system variables are set to `1` in your sysctl configuration by running the following command:
+
[source,bash]
----
sysctl net.bridge.bridge-nf-call-iptables net.bridge.bridge-nf-call-ip6tables net.ipv4.ip_forward
----

. Install a link:https://kubernetes.io/docs/setup/production-environment/container-runtimes/[container runtime].
CRI-O is installed in this example.
Containerd is also an option.
Note: If using cri-o, verify that the major:minor version of cri-o is the same as the version of Kubernetes (installed below).
+
[source,bash]
----
sudo dnf install cri-o containernetworking-plugins
----

. Install Kubernetes. This installs the three necessary Kubernetes applications (`kubectl`, `kubelet`, and `kubeadm`) that need to be on each machine in the cluster.
+
[source,bash]
----
# fedora 39 and earlier use:
sudo dnf install kubernetes-client kubernetes-node kubernetes-kubeadm
----
+
////
[source,bash]
----
# fedora 39 and earlier use:
sudo dnf install kubernetes-client kubernetes-node kubernetes-kubeadm

#fedora 40 and later use:
sudo dnf install kubernetes kubernetes-client
----
////

. Start and enable cri-o.
+
[source,bash]
----
sudo systemctl enable --now crio
----

. Pull needed system container images for Kubernetes.
+
[source,bash]
----
sudo kubeadm config images pull
----

. Start and enable `kubelet`.
Kubelet will be in a crash loop until the cluster is initialized in the next step.
+
[source,bash]
----
sudo systemctl enable --now kubelet
----

. Initialize the cluster.
+
[source,bash]
----
sudo kubeadm init --pod-network-cidr=10.244.0.0/16
----

. kubeadm will generate output to the terminal tracking initialization steps.
If successful, the output below is displayed.
At this point there is a cluster running on this single machine.
After kubeadm finishes you should see:
+
----
Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf
----

. The steps listed above allow a non-root user to use `kubectl`, the Kubernetes command line tool.
Run these commands now.
+
[source,bash]
----
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
----

. Allow the control plane machine to also run pods for applications.
Otherwise more than one machine is needed in the cluster.
+
[source,bash]
----
kubectl taint nodes --all node-role.kubernetes.io/control-plane-
----

. Install flannel into the cluster to provide cluster networking.
There are many other networking solutions besides flannel.
Flannel is straightforward and suitable for this guide.
+
[source,bash]
----
kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
----

. Display list of running pods in the cluster.
All pods should display a status of Running.
A status of CrashLoopBackOff may show up for the coredns pod.
This happens commonly when installing Kubernetes on a virtual machine and the DNS service in the cluster may not select the proper network.
Use your favorite internet search engine to find possible solutions.
+
[source,bash]
----
kubectl get pods --all-namespaces
----

At this point there is a single machine in the cluster running the control plane and available for work as a node.

Upgrades to Kubernetes clusters requires care and planning.
See link:https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/[Upgrading kubeadm clusters] for more information.

The xref:dnf.adoc#sect-using-dnf-plugin[DNF Versionlock plugin] is useful in blocking unplanned updates to Kubernetes rpms.
Occasionally, the Kubernetes version in a Fedora release reaches end-of-life and a new version of Kubernetes is added to the repositories.
Or, an upgrade to Fedora on a cluster machine will also result in a different version of Kubernetes.
Once DNF Versionlock is installed, the following command will hold kubernetes rpms and the cri-o rpm at the 1.28 major:minor version but still allow patch updates to occur:
[source,bash]
----
sudo dnf versionlock add kubernetes*-1.28.* cri-o-1.28.*
----

[[sect-kubernetes-projects-in-copr]]
== Kubernetes projects in COPR

There are Kubernetes projects in link:https://copr.fedorainfracloud.org/[COPR] that might be useful.

[[sect-versioned-kubernetes-rpms]]
=== Versioned Kubernetes RPMS

The link:https://copr.fedorainfracloud.org/coprs/buckaroogeek/copr-k8s-versioned/[Versioned Kubernetes Packages] project is an experimental project exploring the use of versioned Kubernetes packages.
A versioned package has a version number as part of the name such as `kubernetes1.28` or `kubernetes1.28-client`.
The goal is to have multiple versions of Kubernetes available for a given Fedora release.
This uncouples Fedora versions from Kubernetes versions allowing version upgrades to either Fedora or Kubernetes.
A cluster manager can update the Fedora machines while maintaining the cluster version constant.
Or the cluster manager can update Kubernetes while retaining the same Fedora release.

[[sect-kubernetes-1.27]]
=== Kubernetes 1.27 RPMS
The link:https://copr.fedorainfracloud.org/coprs/buckaroogeek/copr-k8s-1.27/[Kubernetes 1.27] project provides Kubernetes 1.27 rpms for all current Fedora releases that provide Go language 1.20 or newer.
This includes Fedora 38, Fedora 39, and Fedora 40 (rawhide).

[[sect-kubernetes-1.25]]
=== Kubernetes 1.25 RPMS

The link:https://copr.fedorainfracloud.org/coprs/buckaroogeek/copr-k8s-1.25/[Kubernetes 1.25] project provides rpms for all Fedora releases.
Kubernetes 1.25 part of the Fedora 37 release along with Go language version 1.19.
The upstream Kubernetes team  recently updated Kubernetes 1.25 to use Go language version 1.20.
Kubernetes 1.25 can no longer be built on standard Fedora 37.
This project uses a custom version of Go available in COPR to continue to provide this version of Kubernetes for Fedora 37 as well as for newer Fedora releases. Both Fedora 37 and Kubernetes 1.25 will reach end of life in the October-November 2023 time frame.

[references]
== References

. https://kubernetes.io/
. https://kubernetes.io/docs/home/
. https://kubernetes.io/docs/concepts/overview/