AWS EKS Cluster without Node Pod Limit with Terraform

Caution: If you are using an AWS managed node group, before switching to the launch template below, AWS will delete the current node group and then create the new node group based on the template. Your pod and cluster configuration should not be deleted, but the pods will not be provisioned for the duration of the node change.

Since I didn't find the appropriate Terraform resource to start AWS EKS nodes without the AWS-defined pod limits per node, here it is. Why AWS defines limits and how to configure your Amazon VPC CNI plugin can be found in this blog post from AWS. I will show the Terraform Launch Template resource needed to start the kubelet without the pod limit defined by AWS.

After you start using the launch template, the following changes to the template will result in rolling node group updates. So AWS will start the new version of the launch template and move the whole cluster to it before shutting down the "old" nodes.

The interesting part of the Terraform resources is the user_data part. This is the script AWS calls on node creation. So this is where one can customize one's nodes.

/etc/eks/bootstrap.sh main \
  --kubelet-extra-args '--max-pods=100' \
  --use-max-pods false

More information about valid parameters can be found on GitHub.

Terraform Resources

The launch template will always get the newest version of the EKS' optimized Amazon Linux 2 images. The Kubernetes version is pulled from the aws_eks_cluster.this resource.

locals {
  cluster_name = "my-eks-cluster"
}

resource "aws_eks_cluster" "this" {
  name                      = local.cluster_name
  role_arn                  = aws_iam_role.this.arn
  enabled_cluster_log_types = ["api", "audit"]
  version                   = "1.28"

  vpc_config {
    subnet_ids              = <SUBNET_IDS>
    endpoint_private_access = true
    endpoint_public_access  = false
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
  # Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.
  depends_on = [
    aws_iam_role_policy_attachment.this_cluster_policy,
  ]
}

data "aws_ssm_parameter" "eks_ami_release_version" {
  name = "/aws/service/eks/optimized-ami/${aws_eks_cluster.this.version}/amazon-linux-2/recommended/image_id"
}

resource "aws_launch_template" "this" {
  name = "eks-launch-template"

  vpc_security_group_ids = [aws_eks_cluster.this.vpc_config[0].cluster_security_group_id, aws_security_group.eks.id]

  block_device_mappings {
    device_name = "/dev/xvda"

    ebs {
      volume_size = 20
      volume_type = "gp2"
    }
  }

  image_id      = data.aws_ssm_parameter.eks_ami_release_version.insecure_value
  instance_type = "t3.medium"
  user_data = base64encode(<<-EOF
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="==MYBOUNDARY=="

--==MYBOUNDARY==
Content-Type: text/x-shellscript; charset="us-ascii"

#!/bin/bash
set -ex
/etc/eks/bootstrap.sh main \
  --kubelet-extra-args '--max-pods=100' \
  --use-max-pods false

--==MYBOUNDARY==--
  EOF
  )

  tag_specifications {
    resource_type = "instance"

    tags = {
      Name = "kubernetes.ec2"
    }
  }
}