(formerly known as kubectl-enter)
Start a root shell in the node's host OS running. Uses an alpine pod with nsenter for Linux nodes and a HostProcess pod with PowerShell for Windows nodes.
using krew:
Plugin can be installed from the official krew repository:
kubectl krew install node-shell
Or from our own krew repository:
kubectl krew index add kvaps https://github.com/kvaps/krew-index kubectl krew install kvaps/node-shell
or using curl:
curl -LO https://github.com/kvaps/kubectl-node-shell/raw/master/kubectl-node_shell
chmod +x ./kubectl-node_shell
sudo mv ./kubectl-node_shell /usr/local/bin/kubectl-node_shell# Get standard bash shell
kubectl node-shell <node>
# Use custom image for pod
kubectl node-shell <node> --image <image>
# Use X-mode (mount /host, and do not enter host namespace)
kubectl node-shell -x <node>
# Skip specific namespace types to enter, choose any of ipc, mount, pid, net, uts
kubectl node-shell <node> --no-ipc
# Execute custom command
kubectl node-shell <node> -- echo 123
# Use stdin
cat /etc/passwd | kubectl node-shell <node> -- sh -c 'cat > /tmp/passwd'
# Run oneliner script
kubectl node-shell <node> -- sh -c 'cat /tmp/passwd; rm -f /tmp/passwd'X-mode can be useful for debugging minimal systems that do not have a built-in shell (eg. Talos).
Here's an example of how you can debug the network for a rootless kube-apiserver container without a filesystem:
kubectl node-shell -x <node>
# Download crictl
wget https://github.com/kubernetes-sigs/cri-tools/releases/download/v1.28.0/crictl-v1.28.0-linux-amd64.tar.gz -O- | \
tar -xzf- -C /usr/local/bin/
# Setup CRI endpoint
export CONTAINER_RUNTIME_ENDPOINT=unix://github.com/host/run/containerd/containerd.sock
# Find your container
crictl ps | grep kube-apiserver
#3ff4626a9f10e e7972205b6614 6 hours ago Running kube-apiserver 0 215107b47bd7e kube-apiserver-talos-rzq-nkg
# Find pid of the container
crictl inspect 3ff4626a9f10e | grep pid
# "pid": 2152,
# "pid": 1
# "type": "pid"
# "getpid",
# "getppid",
# "pidfd_open",
# "pidfd_send_signal",
# "waitpid",
# Go to network namespace of the pid, but keep mount namespace of the debug container
nsenter -t 2152 -nYou need to be able to start privileged containers for that.
You can mount volumes from your CSI storage layer using the -m flag. This allows you to move data to/from node devices seamlessly. The PVC will be mounted at /opt-pvc. This is useful for failover in minimal systems that do not have a built in shell (eg. Talos).
Here is an example of how you can retrieve zfs/lvm data from a volume on a failed CSI node and put it back in your distributed storage layer:
k node-shell -n <namespace> -x <node_with_data> -m <pvc_name>
# install rsync
apk add rsync
# Add lvm/zfs libs
# ZFS
mount -o bind /host/dev /dev
mount -o bind /host/usr/local /usr/local
touch /lib/libuuid.so.1
mount -o bind /host/lib/libuuid.so.1 /lib/libuuid.so.1
touch /lib/libuuid.so.1.3.0
mount -o bind /host/lib/libuuid.so.1.3.0 /lib/libuuid.so.1.3.0
touch /lib/libblkid.so.1
mount -o bind /host/lib/libblkid.so.1 /lib/libblkid.so.1
touch /lib/libblkid.so.1.1.0
mount -o bind /host/lib/libblkid.so.1.1.0 /lib/libblkid.so.1.1.0
#LVM
touch /usr/lib/libaio.so.1
mount -o bind /host/usr/lib/libaio.so.1.0.2 /usr/lib/libaio.so.1
touch /usr/lib/libudev.so.1
mount -o bind /host/usr/lib/libudev.so.1 /usr/lib/libudev.so.1
export PATH=$PATH:/host/sbin
mkdir /lib/modules
mount -o bind /host/lib/modules /lib/modules
# look for data to recover
zfs list
NAME USED AVAIL REFER MOUNTPOINT
hdd-1 15.9T 7.52T 96K /hdd-1
hdd-1/SOME-OLD-PVC-FROM-PREVIOUS-NODE-INSTALL 361G 7.52T 361G - -
# mount the failed volume
zfs set mountpoint=/mnt hdd-1/SOME-OLD-PVC-FROM-PREVIOUS-NODE-INSTALL
zfs mount /hdd-1/SOME-OLD-PVC-FROM-PREVIOUS-NODE-INSTALL
# recover the data : copy it to the mounted CSI volume
rsync -avh --info=progress2 /mnt/ /opt-pvc/the above exemple assumes pvc_name already exists in namespace. You need to be able to start privileged containers.