Differences

This shows you the differences between two versions of the page.

--- raspberry [2022/10/05 06:21] – skipidar
+++ raspberry [2023/03/29 06:56] (current) – [Remote install of image per ssh] skipidar
@@ Line 693: / Line 693: @@
-=== Install OS ===
+=== Install OS Raspberry ===
+https://360techexplorer.com/install-k3s-on-raspberry-pi/
+=== Install OS Ubuntu ===
 Use Ubuntu Server 22.04.1. The Current LTS version.
@@ Line 710: / Line 718: @@
-=== Install "microk8s" Kubernetes ===
-  * Kubernetes: **MicroK8s** is a low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT. It was only supporting a single node but recently they added clustering.
-<color #ed1c24>
-Attention: didnt work For some reason microk8s was not running after the installation.
-All the start/stop commands failed. As it was about a fast and easy way to start - moved on to get.k3s.io</color>
-As described e.g. in https://discourse.ubuntu.com/t/how-to-build-a-raspberry-pi-kubernetes-cluster-using-microk8s/14792
-Install microk8s.
-<code>
-sudo snap install microk8s --classic
-</code>
-<sxh shell>
-pi@ubuntu:~$ sudo snap install microk8s --classic
-microk8s (1.25/stable) v1.25.0 from Canonical✓ installed
-pi@ubuntu:~$
-</sxh>
-You can also learn about available versions and pick a concrete version
-<code>
-snap info microk8s
-</code>
-Returns
-<sxh shell>
-pi@ubuntu:~$ snap info microk8s
-name:      microk8s
-summary:   Kubernetes for workstations and appliances
-publisher: Canonical✓
-store-url: https://snapcraft.io/microk8s
-contact:   https://github.com/ubuntu/microk8s
-license:   unset
-description: |
-  MicroK8s is a small, fast, secure, single node Kubernetes that installs on
-  just about any Linux box. Use it for offline development, prototyping,
-  testing, or use it on a VM as a small, cheap, reliable k8s for CI/CD. It's
-  also a great k8s for appliances - develop your IoT apps for k8s and deploy
-  them to MicroK8s on your boxes.
-commands:
-  - microk8s.add-node
-  - microk8s.addons
-  - microk8s.cilium
-  - microk8s.config
-  - microk8s.ctr
-  - microk8s.dashboard-proxy
-  - microk8s.dbctl
-  - microk8s.disable
-  - microk8s.enable
-  - microk8s.helm
-  - microk8s.helm3
-  - microk8s.images
-  - microk8s.inspect
-  - microk8s.istioctl
-  - microk8s.join
-  - microk8s.kubectl
-  - microk8s.leave
-  - microk8s.linkerd
-  - microk8s
-  - microk8s.refresh-certs
-  - microk8s.remove-node
-  - microk8s.reset
-  - microk8s.start
-  - microk8s.status
-  - microk8s.stop
-  - microk8s.version
-services:
-  microk8s.daemon-apiserver-kicker: simple, enabled, active
-  microk8s.daemon-apiserver-proxy:  simple, enabled, inactive
-  microk8s.daemon-cluster-agent:    simple, enabled, active
-  microk8s.daemon-containerd:       notify, enabled, active
-  microk8s.daemon-etcd:             simple, enabled, inactive
-  microk8s.daemon-flanneld:         simple, enabled, inactive
-  microk8s.daemon-k8s-dqlite:       simple, enabled, active
-  microk8s.daemon-kubelite:         simple, enabled, active
-snap-id:      EaXqgt1lyCaxKaQCU349mlodBkDCXRcg
-tracking:     1.25/stable
-refresh-date: today at 06:48 UTC
-channels:
-.25/stable:           v1.25.0         2022-09-12 (3886) 146MB classic
-.25/candidate:        v1.25.0         2022-09-09 (3886) 146MB classic
-.25/beta:             v1.25.0         2022-09-09 (3886) 146MB classic
-.25/edge:             v1.25.2         2022-09-28 (4042) 146MB classic
-  latest/stable:         v1.25.0         2022-09-07 (3843) 146MB classic
-  latest/candidate:      v1.25.1         2022-09-15 (3938) 146MB classic
-  latest/beta:           v1.25.1         2022-09-15 (3938) 146MB classic
-  latest/edge:           v1.25.2         2022-09-28 (4040) 147MB classic
-.26/stable:           –
-.26/candidate:        –
-.26/beta:             –
-.26/edge:             v1.26.0-alpha.1 2022-09-23 (3983) 146MB classic
-.25-strict/stable:    v1.25.0         2022-09-12 (3885) 146MB -
-</sxh>
-And a concrete version can be installed via
-<code>
-sudo snap install microk8s --classic --channel=1.25/stable
-</code>
-== Pick the master node ==
-On the master node command. It will take some minutes.
-<code>
-sudo microk8s.add-node
-</code>
-Output:
-<sxh shell>
-pi@ubuntu:~$ sudo microk8s.add-node
-From the node you wish to join to this cluster, run the following:
-microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de
-Use the '--worker' flag to join a node as a worker not running the control plane, eg:
-microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de --worker
-If the node you are adding is not reachable through the default interface you can use one of the following:
-microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de
-pi@ubuntu:~$
-</sxh>
-== Join the cluster with second pi ==
-On the second pi, in same network
-<code>
-sudo microk8s join 192.168.191.34:25000/11b28f8dcbd84f96a3e59637d55f94d6/88f6162716de
-</code>
-== Debug check success ==
-To check the status of cluster do
-<code>
-sudo microk8s.status
-</code>
-Sometimes, if the above command doesn’t respond. There might be something or other wrong. Checking on the kubelet with journalctl is a good place to start looking for info.
-Or maybe do a common error check
-all errors
-<code>
-journalctl -p err -b
-</code>
-<sxh shell>
-Sep 30 07:01:59 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:02:27 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:02:37 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:02:45 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:03:10 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-Sep 30 07:04:30 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
-</sxh>
-An easy enough debugging step is to run a stop and start.
-<code>
-sudo microk8s.stop
-sudo microk8s.start
-</code>
-Look at the cluster info and make sure it’s up:
-<code>
-$ sudo microk8s.kubectl cluster-info
-</code>
-<sxh shell>
-$ sudo microk8s.kubectl cluster-info
-Kubernetes master is running at https://127.0.0.1:16443
-To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
-</sxh>
-== Reset the k8s ==
-You can clean-up MicroK8s installation by running the following command:
-This command is used to return the MicroK8s node to the default initial state. This process may take some time and will remove any resources, authentication, running services, pods and optionally, storage. All addons will be disabled and the configuration will be reinitialised.
-This commands makes it easy to revert your MicroK8s to an ‘install fresh’ state wihout having to reinstall anything.
-<code>
-sudo microk8s.reset
-</code>
-<sxh shell>
-Cleaning resources in namespace default
-No resources found
-endpoints "kubernetes" deleted
-secret "default-token-5lqdh" deleted
-serviceaccount "default" deleted
-service "kubernetes" deleted
-</sxh>
@@ Line 929: / Line 724: @@
 http://www.armand.nz/2022/06/01/install-k3s-kubernetes-on-raspberry-pi.html
 https://www.puzzle.ch/de/blog/articles/2020/10/13/k3s-on-raspberry-pi
-  * Kubernetes: **MicroK8s** is a low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT. It was only supporting a single node but recently they added clustering.
@@ Line 975: / Line 765: @@
 Sep 30 07:04:30 ubuntu kernel: hwmon hwmon1: Undervoltage detected!
 </sxh>
-<code>
-journalctl -xeu snap.microk8s.daemon-containerd.service
-systemctl status snap.microk8s.daemon-containerd.service
-</code>
@@ Line 1011: / Line 795: @@
 as in https://k3s.io/
+Start installation
 <code>
+# seem to have been required to set ownership on configs in the past https://0to1.nl/post/k3s-kubectl-permission/
 export K3S_KUBECONFIG_MODE="644"
-curl -sfL https://get.k3s.io
-# Check for Ready node, takes ~30 seconds
-# Kubeconfig is written to /etc/rancher/k3s/k3s.yaml
+# the command might use prev. environment variable
-sudo k3s kubectl get node
+curl -sfL https://get.k3s.io | sh -
 </code>
@@ Line 1046: / Line 830: @@
-Run Server
+Check for Ready node, takes ~30 seconds
+<code>
+sudo systemctl status k3s
+</code>
+<sxh shell>
+pi@pi2:~$ sudo systemctl status k3s
+● k3s.service - Lightweight Kubernetes
+     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: e>
+     Active: active (running) since Wed 2022-10-05 06:46:54 UTC; 2min 7s ago
+       Docs: https://k3s.io
+    Process: 6146 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --qu>
+    Process: 6148 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status>
+    Process: 6149 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SU>
+   Main PID: 6150 (k3s-server)
+      Tasks: 119
+     Memory: 558.5M
+        CPU: 4min 13.990s
+     CGroup: /system.slice/k3s.service
+             ├─6150 "/usr/local/bin/k3s server"
+             ├─6185 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/con>
+             ├─6879 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─6880 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─6881 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─6882 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             ├─7027 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+             └─7266 /var/lib/rancher/k3s/data/e7bc2e5951d7fc2b8da4bc5419f9c3241>
+Oct 05 06:49:01 pi2 k3s[6150]: I1005 06:49:01.422427    6150 trace.go:205] Trac>
+Oct 05 06:49:01 pi2 k3s[6150]: Trace[1718652963]: ---"Object stored in database>
+</sxh>
+You can check if the master node is working. At this point, there is only one Master node
 <code>
-sudo k3s server &
+k3s kubectl get node
-# Kubeconfig is written to /etc/rancher/k3s/k3s.yaml
+</code>
-sudo k3s kubectl get node
-# On a different node run the below command.
+<sxh shell>
+pi@pi2:~$ k3s kubectl get node
+NAME   STATUS   ROLES                  AGE   VERSION
+pi2    Ready    control-plane,master   12m   v1.24.6+k3s1
+</sxh>
+By default, the Kubernetes API server listens on port **6443**.
+You can check if the k8s server is listening on the port 6443
+<code>
+sudo ss -tulpn | grep :6443
+</code>
+<sxh shell>
+pi@pi2:~$ sudo ss -tulpn | grep :6443
+tcp   LISTEN 0      4096                     *:6443             *:*    users:(("k3s-server",pid=6150,fd=16))
+pi@pi2:~$ sudo ps aux | grep 6150
+root        6150 53.9 52.2 1250788 484324 ?      Ssl  06:46   8:15 /usr/local/bin/k3s server
+pi          9717  2.0  0.1   6420  1824 pts/0    S+   07:01   0:00 grep --color=auto 6150
+pi@pi2:~$ ls /usr/local/bin/
+crictl  ctr  k3s  k3s-killall.sh  k3s-uninstall.sh  kubectl
+</sxh>
+==Attach worker nodes to master==
+On k8s server / mater - get teh token
+<code>
 # NODE_TOKEN comes from /var/lib/rancher/k3s/server/node-token on your server
-sudo k3s agent --server https://myserver:6443 --token ${NODE_TOKEN}
+sudo cat /var/lib/rancher/k3s/server/node-token
 </code>
+<sxh shell>
+pi@pi2:~$ sudo cat /var/lib/rancher/k3s/server/node-token
+T10eb2811f7856b5895acac5179c6ae29787957f5289bb00737cf144ac58da851c4::server:9acae609ca530528b4bcc3219720705b
+</sxh>
+Figure out the ip of your server
+<code>
+ip a
+</code>
+The **eth0** adapter is relevant for me.
+<sxh shell>
+pi@pi2:~$ ip a
+: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
+    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
+    inet 127.0.0.1/8 scope host lo
+       valid_lft forever preferred_lft forever
+    inet6 ::1/128 scope host
+       valid_lft forever preferred_lft forever
+: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
+    link/ether b8:27:eb:c8:d0:18 brd ff:ff:ff:ff:ff:ff
+    inet 192.168.191.32/24 metric 100 brd 192.168.191.255 scope global dynamic eth0
+       valid_lft 82237sec preferred_lft 82237sec
+    inet6 fe80::ba27:ebff:fec8:d018/64 scope link
+       valid_lft forever preferred_lft forever
+: wlan0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
+    link/ether b8:27:eb:9d:85:4d brd ff:ff:ff:ff:ff:ff
+: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN group default
+    link/ether 3a:27:9b:9b:d2:5f brd ff:ff:ff:ff:ff:ff
+    inet 10.42.0.0/32 scope global flannel.1
+       valid_lft forever preferred_lft forever
+    inet6 fe80::3827:9bff:fe9b:d25f/64 scope link
+       valid_lft forever preferred_lft forever
+: cni0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
+    link/ether 2e:80:11:68:72:4e brd ff:ff:ff:ff:ff:ff
+    inet 10.42.0.1/24 brd 10.42.0.255 scope global cni0
+       valid_lft forever preferred_lft forever
+    inet6 fe80::2c80:11ff:fe68:724e/64 scope link
+       valid_lft forever preferred_lft forever
+: vethf8a47774@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 22:9e:ba:f3:6d:7b brd ff:ff:ff:ff:ff:ff link-netns cni-afc93733-4c10-7c13-e702-e2159f2115e0
+    inet6 fe80::209e:baff:fef3:6d7b/64 scope link
+       valid_lft forever preferred_lft forever
+: veth4dd4db4c@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 92:68:4b:e9:89:af brd ff:ff:ff:ff:ff:ff link-netns cni-520c6f0a-d043-1c4e-10cd-1877f56186fa
+    inet6 fe80::9068:4bff:fee9:89af/64 scope link
+       valid_lft forever preferred_lft forever
+: veth82490627@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 92:49:38:ab:7a:a6 brd ff:ff:ff:ff:ff:ff link-netns cni-69a4e084-4fd7-25df-0237-d034c7549c13
+    inet6 fe80::9049:38ff:feab:7aa6/64 scope link
+       valid_lft forever preferred_lft forever
+: veth99adcc15@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether f2:a9:ea:60:33:79 brd ff:ff:ff:ff:ff:ff link-netns cni-d01d4a95-b65c-54c4-76c2-0752a4639660
+    inet6 fe80::50b3:2bff:fe69:7914/64 scope link
+       valid_lft forever preferred_lft forever
+: veth748d4e42@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue master cni0 state UP group default
+    link/ether 32:16:9b:b6:f4:9f brd ff:ff:ff:ff:ff:ff link-netns cni-099b5741-6df4-3801-6694-a6357459a63d
+    inet6 fe80::3016:9bff:feb6:f49f/64 scope link
+       valid_lft forever preferred_lft forever
+</sxh>
+On a different node run the below command.
+<code>
+export K3S_KUBECONFIG_MODE="644"
+export K3S_URL="https://192.168.191.32:6443"
+export K3S_TOKEN="T10eb2811f7856b5895acac5179c6ae29787957f5289bb00737cf144ac58da851c4::server:9acae609ca530528b4bcc3219720705b"
+curl -sfL https://get.k3s.io | sh -
+</code>
+==== Remote install of image per ssh ====
+see
+https://weberblog.net/reinstall-your-raspberry-over-the-network/
+  * Login via SSH to the system on which the image is located and read it via dd. The stdout is tunneled through SSH to the local system.
+  * Using pv you can see how much data is transferred. (You may need to install it with apt-get, but you can also omit this.)
+  * funzip extracts the first file and sends it to stdout.
+  * dd writes it to the SD-card.
+<code>
+#This will enable the reset of the Raspberry without regular commands.
+echo 1 > /proc/sys/kernel/sysrq
+#download image
+curl -L http://downloads.raspberrypi.org/raspbian_lite_latest | funzip | dd bs=4M of=/dev/mmcblk0
+# restart the pi (Sofortiger Neustart des Systems (ohne Synchronisation der Speichermedien!))
+echo b > /proc/sysrq-trigger
+</code>