Differences

This shows you the differences between two versions of the page.

--- raspberry [2022/10/02 19:32] – skipidar
+++ raspberry [2023/03/29 06:56] (current) – [Remote install of image per ssh] skipidar
@@ Line 693: / Line 693: @@
-=== Install OS ===
+=== Install OS Raspberry ===
+https://360techexplorer.com/install-k3s-on-raspberry-pi/
+=== Install OS Ubuntu ===
 Use Ubuntu Server 22.04.1. The Current LTS version.
@@ Line 701: / Line 709: @@
 {{https://s3.eu-central-1.amazonaws.com/alf-digital-wiki-pics/sharex/g21bLrn4DT.png}}
-**Attention:** pick the 64bit version of Ubuntu Server. Its based on **arm64** architecture. Only for **arm64** there is a version of **MicroK8s** available in the snap-repository.
+**Attention:** pick the 64bit version of Ubuntu Server. Its based on **arm64** architecture. Only for **arm64** there is a leight weighted version of **Kubernetes** available.
 https://askubuntu.com/questions/1301295/ubuntu-20-04-on-raspberry-pi-4-cannot-install-microk8s
-The 32bit version is based on "armhf" architecture, for which no MicroK8s is available. you will get.
+The 32bit version is based on "armhf" architecture, for which no Kubernetes is available. you will get.
 <code>
 error: snap "microk8s" is not available on stable for this architecture (armhf) but exists on other architectures (amd64, arm64, ppc64el)
@@ Line 711: / Line 719: @@
-=== Install "microk8s" Kubernetes ===
-  * Kubernetes: **MicroK8s** is a low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT. It was only supporting a single node but recently they added clustering.
-<color #ed1c24>
+=== Install "k3s" Kubernetes ===
-Attention: didnt work For some reason microk8s was not running after the installation.
+http://www.armand.nz/2022/06/01/install-k3s-kubernetes-on-raspberry-pi.html
-All the start/stop commands failed. As it was about a fast and easy way to start - moved on to get.k3s.io</color>
+https://www.puzzle.ch/de/blog/articles/2020/10/13/k3s-on-raspberry-pi
-As described e.g. in https://discourse.ubuntu.com/t/how-to-build-a-raspberry-pi-kubernetes-cluster-using-microk8s/14792
-Install microk8s.
+Install k3s.
+https://docs.k3s.io/quick-start
-<code>
+==Prerequisites for the installation==
-sudo snap install microk8s --classic
-</code>
-<sxh shell>
+Mount the sd-card, open the file `cmdline.txt` and append.
-pi@ubuntu:~$ sudo snap install microk8s --classic
+It enables some backward compatibility raspberry kernel modules.
-microk8s (1.25/stable) v1.25.0 from Canonical✓ installed
-pi@ubuntu:~$
-</sxh>
-You can also learn about available versions and pick a concrete version
 <code>
-snap info microk8s
+cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
 </code>
-Returns
+The file `cmdline.txt`
 <sxh shell>
-pi@ubuntu:~$ snap info microk8s
+console=serial0,115200 dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash
-name:      microk8s
+cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
-summary:   Kubernetes for workstations and appliances
-publisher: Canonical✓
-store-url: https://snapcraft.io/microk8s
-contact:   https://github.com/ubuntu/microk8s
-license:   unset
-description: |
-  MicroK8s is a small, fast, secure, single node Kubernetes that installs on
-  just about any Linux box. Use it for offline development, prototyping,
-  testing, or use it on a VM as a small, cheap, reliable k8s for CI/CD. It's
-  also a great k8s for appliances - develop your IoT apps for k8s and deploy
-  them to MicroK8s on your boxes.
-commands:
-  - microk8s.add-node
-  - microk8s.addons
-  - microk8s.cilium
-  - microk8s.config
-  - microk8s.ctr
-  - microk8s.dashboard-proxy
-  - microk8s.dbctl
-  - microk8s.disable
-  - microk8s.enable
-  - microk8s.helm
-  - microk8s.helm3
-  - microk8s.images
-  - microk8s.inspect
-  - microk8s.istioctl
-  - microk8s.join
-  - microk8s.kubectl
-  - microk8s.leave
-  - microk8s.linkerd
-  - microk8s
-  - microk8s.refresh-certs
-  - microk8s.remove-node
-  - microk8s.reset
-  - microk8s.start
-  - microk8s.status
-  - microk8s.stop
-  - microk8s.version
-services:
-  microk8s.daemon-apiserver-kicker: simple, enabled, active
-  microk8s.daemon-apiserver-proxy:  simple, enabled, inactive
-  microk8s.daemon-cluster-agent:    simple, enabled, active
-  microk8s.daemon-containerd:       notify, enabled, active
-  microk8s.daemon-etcd:             simple, enabled, inactive
-  microk8s.daemon-flanneld:         simple, enabled, inactive
-  microk8s.daemon-k8s-dqlite:       simple, enabled, active
-  microk8s.daemon-kubelite:         simple, enabled, active
-snap-id:      EaXqgt1lyCaxKaQCU349mlodBkDCXRcg
-tracking:     1.25/stable
-refresh-date: today at 06:48 UTC
-channels:
-.25/stable:           v1.25.0         2022-09-12 (3886) 146MB classic
-.25/candidate:        v1.25.0         2022-09-09 (3886) 146MB classic
-.25/beta:             v1.25.0         2022-09-09 (3886) 146MB classic
-.25/edge:             v1.25.2         2022-09-28 (4042) 146MB classic
-  latest/stable:         v1.25.0         2022-09-07 (3843) 146MB classic
-  latest/candidate:      v1.25.1         2022-09-15 (3938) 146MB classic
-  latest/beta:           v1.25.1         2022-09-15 (3938) 146MB classic
-  latest/edge:           v1.25.2         2022-09-28 (4040) 147MB classic
-.26/stable:           –
-.26/candidate:        –
-.26/beta:             –
-.26/edge:             v1.26.0-alpha.1 2022-09-23 (3983) 146MB classic
-.25-strict/stable:    v1.25.0         2022-09-12 (3885) 146MB -
 </sxh>
-And a concrete version can be installed via
-<code>
-sudo snap install microk8s --classic --channel=1.25/stable
-</code>
-== Pick the master node ==
-On the master node command. It will take some minutes.
-<code>
-sudo microk8s.add-node
-</code>
-Output:
-<sxh shell>
-pi@ubuntu:~$ sudo microk8s.add-node
-From the node you wish to join to this cluster, run the following:
-microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de
-Use the '--worker' flag to join a node as a worker not running the control plane, eg:
-microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de --worker
-If the node you are adding is not reachable through the default interface you can use one of the following:
-microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de
-pi@ubuntu:~$
-</sxh>
-== Join the cluster with second pi ==
-On the second pi, in same network
-<code>
-sudo microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de
-</code>
 == Debug check success ==
-To check the status of cluster do
-<code>
-sudo microk8s.status
-</code>
@@ Line 873: / Line 764: @@
 Sep 30 07:03:10 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
 Sep 30 07:04:30 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
 </sxh>
-<code>
+An easy enough debugging step is to run a stop and start.
-journalctl -xeu snap.microk8s.daemon-containerd.service
-systemctl status snap.microk8s.daemon-containerd.service
-</code>
-An easy enough debugging step is to run a stop and start.
+==Install linux-modules-extra-raspi extra package==
+Through lots of frustration, I discovered ubuntu installation were missing a kernel module that always resulted in `STATUS: NotReady` when my Kubernetes K3S was instsalled.
-<code>
+Only by installing this kernel module I was able to finally get my ubuntu raspberry pi modules to a `STATUS: Ready state`. Evidently, this only affects the Rasberry Pi install of Ubuntu.
-sudo microk8s.stop
-sudo microk8s.start
-</code>
+On each Raspberry PI Install the `linux-modules-extra-raspi` extra package, a specific requirement for Ubuntu 21.10+ and k3s on Raspberry Pis
-Look at the cluster info and make sure it’s up:
+Run the following to install the module.
+A reboot is required to take effect
 <code>
-$ sudo microk8s.kubectl cluster-info
+sudo apt install linux-modules-extra-raspi
+sudo reboot
 </code>
-<sxh shell>
-$ sudo microk8s.kubectl cluster-info
-Kubernetes master is running at https://127.0.0.1:16443
-To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
+==Setup the Master k3s Node==
-</sxh>
+The Master node’s (k8s0) initial k3s installation will serve as the control plane for our Kubernetes Cluster.
-== Reset the k8s ==
+Install K3s with the flags to make sure /etc/rancher/k3s/k3s.yaml is world-readable
-You can clean-up MicroK8s installation by running the following command:
+as in https://k3s.io/
-This command is used to return the MicroK8s node to the default initial state. This process may take some time and will remove any resources, authentication, running services, pods and optionally, storage. All addons will be disabled and the configuration will be reinitialised.
-This commands makes it easy to revert your MicroK8s to an ‘install fresh’ state wihout having to reinstall anything.
+Start installation
 <code>
-sudo microk8s.reset
+# seem to have been required to set ownership on configs in the past https://0to1.nl/post/k3s-kubectl-permission/
+export K3S_KUBECONFIG_MODE="644"
+# the command might use prev. environment variable
+curl -sfL https://get.k3s.io | sh -
 </code>
 <sxh shell>
-Cleaning resources in namespace default
+pi@pi2:~$ curl -sfL https://get.k3s.io | sh -
-No resources found
+[INFO]  Finding release for channel stable
-endpoints "kubernetes" deleted
+[INFO]  Using v1.24.6+k3s1 as release
-secret "default-token-5lqdh" deleted
+[INFO]  Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.24.6+k3s1/sha256sum-arm64.txt
-serviceaccount "default" deleted
+[INFO]  Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.24.6+k3s1/k3s-arm64
-service "kubernetes" deleted
+[INFO]  Verifying binary download
+[INFO]  Installing k3s to /usr/local/bin/k3s
+[INFO]  Skipping installation of SELinux RPM
+[INFO]  Creating /usr/local/bin/kubectl symlink to k3s
+[INFO]  Creating /usr/local/bin/crictl symlink to k3s
+[INFO]  Creating /usr/local/bin/ctr symlink to k3s
+[INFO]  Creating killall script /usr/local/bin/k3s-killall.sh
+[INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh
+[INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env
+[INFO]  systemd: Creating service file /etc/systemd/system/k3s.service
+[INFO]  systemd: Enabling k3s unit
+Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
+[INFO]  systemd: Starting k3s
 </sxh>
+Check for Ready node, takes ~30 seconds
+<code>
+sudo systemctl status k3s
+</code>
-=== Install "k3s" Kubernetes ===
+<sxh shell>
-http://www.armand.nz/2022/06/01/install-k3s-kubernetes-on-raspberry-pi.html
+pi@pi2:~$ sudo systemctl status k3s
-https://www.puzzle.ch/de/blog/articles/2020/10/13/k3s-on-raspberry-pi
+● k3s.service - Lightweight Kubernetes
+     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: e>
+     Active: active (running) since Wed 2022-10-05 06:46:54 UTC; 2min 7s ago
+       Docs: https://k3s.io
+    Process: 6146 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --qu>
+    Process: 6148 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status>
+    Process: 6149 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SU>
+   Main PID: 6150 (k3s-server)
+      Tasks: 119
+     Memory: 558.5M
+        CPU: 4min 13.990s
+     CGroup: /system.slice/k3s.service
+             ├─6150 "/usr/local/bin/k3s server"
+             ├─6185 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/con>
+             ├─6879 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─6880 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─6881 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─6882 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─7027 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             └─7266 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+Oct 05 06:49:01 pi2 k3s[6150]: I1005 06:49:01.422427    6150 trace.go:205] Trac>
+Oct 05 06:49:01 pi2 k3s[6150]: Trace[1718652963]: ---"Object stored in database>
+</sxh>
+You can check if the master node is working. At this point, there is only one Master node
+<code>
+k3s kubectl get node
+</code>
-  * Kubernetes: **MicroK8s** is a low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT. It was only supporting a single node but recently they added clustering.
+<sxh shell>
+pi@pi2:~$ k3s kubectl get node
+NAME   STATUS   ROLES                  AGE   VERSION
+pi2    Ready    control-plane,master   12m   v1.24.6+k3s1
+</sxh>
-Install k3s.
+By default, the Kubernetes API server listens on port **6443**.
-https://docs.k3s.io/quick-start
+You can check if the k8s server is listening on the port 6443
-==Prerequisites for the installation==
-Mount the sd-card, open the file `cmdline.txt` and append.
-It enables some backward compatibility raspberry kernel modules.
 <code>
-cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
+sudo ss -tulpn | grep :6443
 </code>
-The file `cmdline.txt`
 <sxh shell>
-console=serial0,115200 dwc_otg.lpm_enable=0 console=tty1 root=LABEL=writable rootfstype=ext4 rootwait fixrtc quiet splash
+pi@pi2:~$ sudo ss -tulpn | grep :6443
-cgroup_enable=cpuset cgroup_memory=1 cgroup_enable=memory
+tcp   LISTEN 0      4096                     *:6443             *:*    users:(("k3s-server",pid=6150,fd=16))
-</sxh>
+pi@pi2:~$ sudo ps aux | grep 6150
+root        6150 53.9 52.2 1250788 484324 ?      Ssl  06:46   8:15 /usr/local/bin/k3s server
+pi          9717  2.0  0.1   6420  1824 pts/0    S+   07:01   0:00 grep --color=auto 6150
+pi@pi2:~$ ls /usr/local/bin/
+crictl  ctr  k3s  k3s-killall.sh  k3s-uninstall.sh  kubectl
+</sxh>
-== Debug check success ==
+==Attach worker nodes to master==
+On k8s server / mater - get teh token
-Sometimes, if the above command doesn’t respond. There might be something or other wrong. Checking on the kubelet with journalctl is a good place to start looking for info.
-Or maybe do a common error check
-all errors
 <code>
-journalctl -p err -b
+# NODE_TOKEN comes from /var/lib/rancher/k3s/server/node-token on your server
+sudo cat /var/lib/rancher/k3s/server/node-token
 </code>
 <sxh shell>
-Sep 30 07:01:59 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
+pi@pi2:~$ sudo cat /var/lib/rancher/k3s/server/node-token
-Sep 30 07:02:27 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
+T10eb2811f7856b5895acac5179c6ae29787957f5289bb00737cf144ac58da851c4::server:9acae609ca530528b4bcc3219720705b
-Sep 30 07:02:37 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:02:45 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:03:10 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:04:30 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
 </sxh>
+Figure out the ip of your server
 <code>
-journalctl -xeu snap.microk8s.daemon-containerd.service
+ip a
-systemctl status snap.microk8s.daemon-containerd.service
 </code>
+The **eth0** adapter is relevant for me.
+<sxh shell>
+pi@pi2:~$ ip a
+: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host
+       valid_lft forever preferred_lft forever
+: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
+    link/ether b8:27:eb:c8:d0:18 brd ff:ff:ff:ff:ff:ff
+    inet 192.168.191.32/24 metric 100 brd 192.168.191.255 scope global dynamic eth0
+       valid_lft 82237sec preferred_lft 82237sec
+    inet6 fe80::ba27:ebff:fec8:d018/64 scope link
+       valid_lft forever preferred_lft forever
+: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
+    link/ether b8:27:eb:9d:85:4d brd ff:ff:ff:ff:ff:ff
+: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
+    link/ether 3a:27:9b:9b:d2:5f brd ff:ff:ff:ff:ff:ff
+    inet 10.42.0.0/32 scope global flannel.1
+       valid_lft forever preferred_lft forever
+    inet6 fe80::3827:9bff:fe9b:d25f/64 scope link
+       valid_lft forever preferred_lft forever
+: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
+    link/ether 2e:80:11:68:72:4e brd ff:ff:ff:ff:ff:ff
+    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
+       valid_lft forever preferred_lft forever
+    inet6 fe80::2c80:11ff:fe68:724e/64 scope link
+       valid_lft forever preferred_lft forever
+: vethf8a47774@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 22:9e:ba:f3:6d:7b brd ff:ff:ff:ff:ff:ff link-netns cni-afc93733-4c10-7c13-e702-e2159f2115e0
+    inet6 fe80::209e:baff:fef3:6d7b/64 scope link
+       valid_lft forever preferred_lft forever
+: veth4dd4db4c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 92:68:4b:e9:89:af brd ff:ff:ff:ff:ff:ff link-netns cni-520c6f0a-d043-1c4e-10cd-1877f56186fa
+    inet6 fe80::9068:4bff:fee9:89af/64 scope link
+       valid_lft forever preferred_lft forever
+: veth82490627@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 92:49:38:ab:7a:a6 brd ff:ff:ff:ff:ff:ff link-netns cni-69a4e084-4fd7-25df-0237-d034c7549c13
+    inet6 fe80::9049:38ff:feab:7aa6/64 scope link
+       valid_lft forever preferred_lft forever
+: veth99adcc15@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether f2:a9:ea:60:33:79 brd ff:ff:ff:ff:ff:ff link-netns cni-d01d4a95-b65c-54c4-76c2-0752a4639660
+    inet6 fe80::50b3:2bff:fe69:7914/64 scope link
+       valid_lft forever preferred_lft forever
+: veth748d4e42@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 32:16:9b:b6:f4:9f brd ff:ff:ff:ff:ff:ff link-netns cni-099b5741-6df4-3801-6694-a6357459a63d
+    inet6 fe80::3016:9bff:feb6:f49f/64 scope link
+       valid_lft forever preferred_lft forever
+</sxh>
-An easy enough debugging step is to run a stop and start.
+On a different node run the below command.
+<code>
+export K3S_KUBECONFIG_MODE="644"
+export K3S_URL="https://192.168.191.32:6443"
+export K3S_TOKEN="T10eb2811f7856b5895acac5179c6ae29787957f5289bb00737cf144ac58da851c4::server:9acae609ca530528b4bcc3219720705b"
-==Install linux-modules-extra-raspi extra package==
+curl -sfL https://get.k3s.io | sh -
+</code>
-Through lots of frustration, I discovered ubuntu installation were missing a kernel module that always resulted in `STATUS: NotReady` when my Kubernetes K3S was instsalled.
-Only by installing this kernel module I was able to finally get my ubuntu raspberry pi modules to a `STATUS: Ready state`. Evidently, this only affects the Rasberry Pi install of Ubuntu.
-On each Raspberry PI Install the `linux-modules-extra-raspi` extra package, a specific requirement for Ubuntu 21.10+ and k3s on Raspberry Pis
+==== Remote install of image per ssh ====
+see
+https://weberblog.net/reinstall-your-raspberry-over-the-network/
+  * Login via SSH to the system on which the image is located and read it via dd. The stdout is tunneled through SSH to the local system.
+  * Using pv you can see how much data is transferred. (You may need to install it with apt-get, but you can also omit this.)
+  * funzip extracts the first file and sends it to stdout.
+  * dd writes it to the SD-card.
 <code>
-sudo apt install linux-modules-extra-raspi
- sudo reboot
-</code>
+#This will enable the reset of the Raspberry without regular commands.
+echo 1 > /proc/sys/kernel/sysrq
-<code>
+#download image
+curl -L http://downloads.raspberrypi.org/raspbian_lite_latest | funzip | dd bs=4M of=/dev/mmcblk0
+# restart the pi (Sofortiger Neustart des Systems (ohne Synchronisation der Speichermedien!))
+echo b > /proc/sysrq-trigger
 </code>