Google recommends creating separate GPU node pools, rather than adding GPUs to your general pool. The reason is cost and control. GPU machines are expensive, so you want them to exist only when something needs them, and you want your ordinary workloads to stay off them. Separate pools make both possible.<\/p>\r\n
This walks through adding a GPU node pool in the console, then with one gcloud command, and covers drivers, quota, how pods ask for a GPU, and how to keep the cost down.<\/p>\r\n
Prerequisites:<\/strong><\/p>\r\n Not every GPU type is offered in every zone. List the accelerator types that exist, and check the zone column against the zone your cluster uses:<\/p>\r\n gcloud compute accelerator-types list<\/p>\r\n\r\n\r\n\r\n Open your cluster in the console and click Add node pool.<\/p>\r\n\r\n\r\n\r\n Give the pool a name and choose the node version. Choose the size of the pool, and tick Enable autoscaling. Setting the minimum number of nodes to zero lets the pool shrink to nothing when no pod needs a GPU, which is the main way to avoid paying for idle GPUs. Under Automation, tick Enable auto upgrade and Enable auto repair.<\/p>\r\n\r\n\r\n\r\n Choose the operating system for the nodes.<\/p>\r\n\r\n\r\n\r\n In Machine configuration, click GPU. Choose the GPU type and how many GPUs each node should have. Then choose the machine configuration for the nodes, and click Create. The GPU pool is created.<\/p>\r\n The original page also says to click Enable Virtual Workstation and tick I understand the limitations. That option selects a GPU variant meant for graphics and remote workstation workloads, and it typically costs more. For machine learning and other compute work, leave it off.<\/p>\r\n\r\n\r\n\r\n This is the command from the original page, with the flags corrected. Each flag has two real hyphens, and the last two flags are no longer joined together:<\/p>\r\n gcloud container node-pools create gpu-pool –accelerator type=nvidia-tesla-p4,count=1 –zone us-central1-c –cluster gke-cluster-1 –num-nodes 2 –min-nodes 0 –max-nodes 3 –enable-autoscaling –machine-type n1-standard-1<\/p>\r\n It creates a pool called gpu-pool with 2 nodes to start, that can scale between 0 and 3, each with one GPU. If you leave out the zone, gcloud may ask you for one.<\/p>\r\n\r\n\r\n\r\n A GPU is useless to your containers until the NVIDIA driver is installed on the node. Google’s gcloud reference lets you choose how, with a driver version option added to the accelerator flag, for example:<\/p>\r\n –accelerator type=nvidia-tesla-t4,count=1,gpu-driver-version=default<\/p>\r\n This is why a 2021 style guide that never mentions drivers is incomplete today. On recent versions GKE does this for you, but it is worth knowing the option exists.<\/p>\r\n GKE automatically adds a taint to GPU nodes so that only pods that ask for a GPU are scheduled on them, and your ordinary pods stay off the expensive machines. To ask for one, a pod requests the GPU resource in its container settings, under the resource limits:<\/p>\r\n resources: When a pod like this is waiting and no GPU node is free, the autoscaler can add a node to the pool, up to the maximum you set.<\/p>\r\n GPU nodes bill for the whole machine, GPU included, for as long as they exist. Two habits keep this under control. First, set the pool’s minimum to zero nodes, so it costs nothing when no pod needs a GPU. Second, delete the pool when you finish practising. I have not tested this command, so check the names first:<\/p>\r\n gcloud container node-pools delete gpu-pool –cluster=gke-cluster-1 –zone=us-central1-c<\/p>\r\n The Creating and Deleting Budget guide in this series explains why a budget alert only notifies you, and does not stop spending.<\/p>\r\n\r\n
Step by Step Process of Adding a GPU Node Pool<\/h2>\r\n
Step 1: See Which GPUs Are Available in Your Zone<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-556.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 2: Add a Node Pool<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-557.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 3: Name It and Set the Size<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-558.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 4: Choose the Operating System<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-559.png\")
<\/figure>\r\n\r\n\r\n\r\nStep 5: Choose the GPU and the Machine<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/image-560.png\")
<\/figure>\r\n\r\n\r\n\r\nThe Same Thing With One Command<\/h3>\r\n
![\"\"](\"https:\/\/prwatech.in\/blog\/wp-content\/uploads\/2021\/05\/49-1024x172.jpg\")
<\/figure>\r\nGPU Drivers<\/h2>\r\n
\r\n
How a Pod Asks for a GPU<\/h2>\r\n
\u00a0\u00a0limits:
\u00a0\u00a0\u00a0\u00a0nvidia.com\/gpu: 1<\/p>\r\nWhat a GPU Pool Costs<\/h2>\r\n
Common Mistakes to Avoid<\/h2>\r\n