Bash script to kill TPUv5e(TPUv5lite) consuming process

Issue

TPUv5 does not live at /dev/accel0 — where TPUv3 and TPUv4 live

→ sudo fuser -k /dev/accel0 does not work

Solution

TPUv5e(TPUv5lite) lives at /dev/vfio/0

Bash script with gcloud command to kill TPU consuming process


# seq len 2048
export TPU_NAME='v5e-256'
export ZONE='europe-west4-b'

echo "[local] Killing TPU"
gcloud compute tpus tpu-vm ssh $TPU_NAME \
    --zone $ZONE --worker=all --command "sudo fuser -k /dev/vfio/0" # TPUv5에서는 /dev/vfio/0가 장비 이름이다.

echo "[local] Removing TPU Lock"
gcloud compute tpus tpu-vm ssh $TPU_NAME \
    --zone $ZONE --worker=all --command "sudo rm -f /tmp/libtpu_lockfile"