General overview ================= - etcd services (worth checking both ports) etcdctl3 --endpoints="192.168.213.1:2379" member list - doesn't check health only reports members oc get cs - only etcd (other services will fail on Openshift) - All nodes and pods are fine and running and all pvc are bound oc get nodes oc get pods --all-namespaces -o wide oc get pvc --all-namespaces -o wide - API health check curl -k https://apiserver.kube-service-catalog.svc/healthz Storage ======= - Heketi status heketi-cli -s http://heketi-storage.glusterfs.svc.cluster.local:8080 --user admin --secret "$(oc get secret heketi-storage-admin-secret -n glusterfs -o jsonpath='{.data.key}' | base64 -d)" topology info - Status of Gluster Volume (and its bricks which with heketi fails often) gluster volume info ./gluster.sh info all_heketi - Check available storage space on system partition and LVM volumes (docker, heketi, ands) Run 'df -h' and 'lvdisplay' on each node - Check status of hardware raids /opt/MegaRAID/storcli/storcli64 /c0/v0 show all Networking ========== - Check that correct upstream name servers are listed for both DNSMasq (host) and SkyDNS (pods). If not fix and restart 'origin-node' and 'dnsmasq'. * '/etc/dnsmasq.d/origin-upstream-dns.conf' * '/etc/origin/node/resolv.conf' - Check that both internal and external addresses are resolvable from all hosts. * I.e. we should be able to resolve 'google.com' * And we should be able to resolve 'heketi-storage.glusterfs.svc.cluster.local' - Check that keepalived service is up and the corresponding ip's are really assigned to one of the nodes (vagrant provisioner would remove keepalived tracked ips, but keepalived will continue running without noticing it) - Ensure, we don't have override of cluster_name to first master (which we do during the provisioning of OpenShift plays) ADEI ==== - No caching pods are hung (for whatever reason)