Cannot reach a service deployed on kubernetes - can someone help with network config?


#1

Hi all,
I installed kubernetes on 4 raspberry pi 3’s several time with the rak8s playbooks.
The last time today with the 2018-06-27-raspbian-stretch-lite as operating system. But I got the same issue also with the 2018-04-18 raspian and the 2018-03-13 raspian.

Running the playbooks works fine. After the installation I configure the cluster directly from my master node which is raspic0. pi is not in the docker group thus I use kubectl with sudo.
“sudo kubectl get nodes” shows all 4 nodes running.

Now my problem:
I cannot reach deployed services neither from within the cluster (thus from my master) nor from outside the cluster. For a check I deployed a hypriot busybox as in https://blog.hypriot.com/post/setup-kubernetes-raspberry-pi-cluster/

I assume a network configuration problem due to the following:
“sudo kubectl get po --all-namespaces” reports
NAMESPACE NAME READY STATUS RESTARTS AGE
default hypriot-587768b4f5-5cxs9 1/1 Running 0 26m
default hypriot-587768b4f5-ccj74 1/1 Running 0 26m
default hypriot-587768b4f5-n5z7h 1/1 Running 0 26m
kube-system etcd-raspic0 1/1 Running 1 53m
kube-system kube-apiserver-raspic0 1/1 Running 1 53m
kube-system kube-controller-manager-raspic0 1/1 Running 1 54m
kube-system kube-dns-7b6ff86f69-4lkpb 3/3 Running 3 53m
kube-system kube-flannel-ds-bbtlh 1/1 Running 5 53m
kube-system kube-flannel-ds-jdp6l 1/1 Running 5 53m
kube-system kube-flannel-ds-nzpr7 1/1 Running 3 52m
kube-system kube-flannel-ds-qvtkc 1/1 Running 4 53m
kube-system kube-proxy-kjs7k 1/1 Running 1 53m
kube-system kube-proxy-kk7gs 1/1 Running 1 52m
kube-system kube-proxy-t4wht 1/1 Running 1 53m
kube-system kube-proxy-z8c8x 1/1 Running 1 53m
kube-system kube-scheduler-raspic0 1/1 Running 1 54m
kube-system kubernetes-dashboard-7fcc5cb979-2nr7m 0/1 CrashLoopBackOff 16 53m

So I see that the apiserver and dns are running.

But “sudo kubectl get endpoints --all-namespaces” reportsNAMESPACE NAME ENDPOINTS AGE
default hypriot 172.30.1.4:80,172.30.2.2:80,172.30.3.2:80 28m
default kubernetes 192.168.1.120:6443 57m
kube-system kube-controller-manager 57m
kube-system kube-dns 172.30.0.3:53,172.30.0.3:53 56m
kube-system kube-scheduler 57m
kube-system kubernetes-dashboard 56m

The network address of my raspic0 is 192.168.1.120.
According to the playbook kubeadm init is executed with a --pod-network-cidr=172.30.0.0/16
That would fit to the endpoint ips of hypriot and the kube-dns

In investigated the logs of kubernetes-dashboard with “sudo kubectl logs kubernetes-dashboard-7fcc5cb979-2nr7m --namespace=kube-system” and got
018/08/25 20:34:41 Starting overwatch
2018/08/25 20:34:41 Using in-cluster config to connect to apiserver
2018/08/25 20:34:41 Using service account token for csrf signing
2018/08/25 20:34:41 No request provided. Skipping authorization
2018/08/25 20:35:11 Error while initializing connection to Kubernetes apiserver. This most likely means that the cluster is misconfigured (e.g., it has invalid apiserver certificates or service accounts configuration) or the --apiserver-host param points to a server that does not exist. Reason: Get https://10.96.0.1:443/version: dial tcp 10.96.0.1:443: i/o timeout

It seems that the dashboard searches for the apiserver on 10.96.0.1

I directly attached to the pods of hypriot and it turned out that from within the pod the nameserver is neither able to resolve the service hypriot nor the dns.

According to https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ the dns service is running
“sudo kubectl get svc --namespace=kube-system” reports
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns ClusterIP 10.96.0.10 53/UDP,53/TCP 1h
kubernetes-dashboard ClusterIP 10.104.188.48 443/TCP 1h

According to https://kubernetes.io/docs/tasks/debug-application-cluster/debug-service/ I checked from within one of my hypriot pods whether the dns is visible but no success:

pi@raspic0:~ $ sudo kubectl exec hypriot-587768b4f5-5cxs9 busybox nslookup kubernetes.default
nslookup: can’t resolve ‘kubernetes.default’
Server: 10.96.0.10
Address 1: 10.96.0.10

command terminated with exit code 1

I am not really familiar with the network setup in kubernetes so every help is appreciated.
Thanks
Heinz