Teleport EKS Auto-Discovery

EKS Auto-Discovery can automatically discover any EKS cluster and enroll it in Teleport if its tags match the configured labels.

Teleport cluster auto-discovery involves two components:

1. The Teleport Discovery Service that watches for new clusters or changes to previously discovered clusters. It dynamically registers each discovered cluster as a kube_cluster resource in your Teleport cluster. It does not need connectivity to the clusters it discovers.
2. The Teleport Kubernetes Service that monitors the dynamic kube_cluster resources registered by the Discovery Service. It proxies communications between users and the cluster.

How it works

The Teleport Discovery Service scans configured cloud providers, including AWS, for Kubernetes clusters that match specified filtering labels, creating dynamic resources within Teleport for any new clusters identified. The Teleport Kubernetes Service monitors these dynamic resources, forwarding requests to the corresponding Kubernetes clusters. Both services require access to the AWS API to perform their functions.

Additionally, the Kubernetes Service needs direct access to the target clusters and the necessary permissions to forward requests.

Prerequisites

• A running Teleport cluster version 16.4.7 or above. If you want to get started with Teleport, sign up for a free trial or set up a demo environment.

• The tctl admin tool and tsh client tool.

  Visit Installation for instructions on downloading tctl and tsh.

• An AWS account with permissions to create and attach IAM policies.
• A host to run the Teleport Discovery and Kubernetes services.
• One or more EKS clusters running.

Step 1/3. Set up AWS IAM credentials

Create and attach the following AWS IAM policy to the identity of the instance running the Teleport Discovery Service:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "EKSDiscovery",
            "Effect": "Allow",
            "Action": [
              "eks:DescribeCluster",
              "eks:ListClusters"
            ],
            "Resource": "*"
        },
        {
            "Sid": "EKSManageAccess",
            "Effect": "Allow",
            "Action": [
              "eks:AssociateAccessPolicy",
              "eks:CreateAccessEntry",
              "eks:DeleteAccessEntry",
              "eks:DescribeAccessEntry",
              "eks:TagResource",
              "eks:UpdateAccessEntry"
            ],
            "Resource": "*"
        }
    ]
}

You can use a list of ARNs and narrow the scope of the permissions to specific regions or EKS clusters instead of using a wildcard. The resource ARN has the following format:

arn:{Partition}:eks:{Region}:{Account}:cluster/{ClusterName}

The permissions in the EKSManageAccess statement are optional because the Discovery Service will discover EKS clusters even when it cannot ensure that the Teleport Kubernetes Service has access to the clusters it discovers. If you omit any of the EKSManageAccess permissions, then it is your responsibility to ensure that the Teleport Kubernetes Service can access each EKS cluster.

Step 2/3. Configure EKS cluster authorization

When the Kubernetes Service uses an IAM role that is different from the one that creates the clusters, you need to configure the mapping between the Teleport IAM Role and the Kubernetes RBAC permissions by editing the aws-auth Configmap on each of the discovered clusters.

To forward requests to the Kubernetes cluster, the Teleport Kubernetes Service requires cluster-wide permissions to Impersonate RBAC users and groups, to create SelfSubjectAccessReviews and SelfSubjectRulesReviews, and, finally, read access to Pods.

If your Kubernetes cluster does not have an RBAC group with the required permissions, you can create the ClusterRole, ClusterRoleBinding, and the mapping by following the Creating RBAC group guide. If your cluster already has an RBAC group that satisfies the required permissions, you can reuse it and map it into the IAM Role used by the Teleport Kubernetes Service. For simplicity, it is also possible to map the Teleport IAM role onto a built-in Kubernetes RBAC group like system:masters, but not recommended in production.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: teleport
rules:
- apiGroups:
  - ""
  resources:
  - users
  - groups
  - serviceaccounts
  verbs:
  - impersonate
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - get
- apiGroups:
  - "authorization.k8s.io"
  resources:
  - selfsubjectaccessreviews
  - selfsubjectrulesreviews
  verbs:
  - create

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: teleport
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: teleport
subjects:
- kind: Group
  name: teleport
  apiGroup: rbac.authorization.k8s.io

apiVersion: v1
data:
  mapRoles: |
    - groups:
      - teleport
      rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role
      username: teleport

aws eks create-access-entry \  --cluster-name eks-cluster \  --region eu-west-1 \  --principal-arn arn:aws:iam::222222222222:role/teleport-role \  --kubernetes-groups teleport

{  ...}

apiVersion: v1
data:
  mapRoles: |
  ...
    - groups:
      - {rbac_group}
      rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role
      username: teleport

apiVersion: v1
data:
  mapRoles: |
  ...
    - groups:
      - system:masters
      rolearn: {teleport_aws_iam_role} # e.g. arn:aws:iam::222222222222:role/teleport-role
      username: teleport

aws eks create-access-entry \  --cluster-name eks-cluster \  --region eu-west-1 \  --principal-arn arn:aws:iam::222222222222:role/teleport-role

{  ...}
aws eks associate-access-policy \  --cluster-name eks-cluster \  --region eu-west-1 \  --principal-arn arn:aws:iam::222222222222:role/teleport-role
  --policy-arn "arn:aws:eks::aws:cluster-access-policy/AmazonEKSClusterAdminPolicy" \  --access-scope type=cluster
{  ...}

Step 3/3. Configure Teleport to discover EKS clusters

Get a join token

Teleport EKS Auto-Discovery requires a valid Teleport auth token for the Discovery and Kubernetes services to join the cluster. Generate one by running the following command against your Teleport Auth Service and save it in /tmp/token on the machine that will run Kubernetes Discovery:

tctl tokens add --type=discovery,kube

Configure the Teleport Kubernetes and Discovery Services

Enabling EKS Auto-Discovery requires that the discovery_service.aws section include at least one entry and that discovery_service.aws.types include eks. It also requires configuring the kubernetes_service.resources.tags to use the same labels configured at discovery_service.aws.tags or a subset of them to make the Kubernetes Service listen to the dynamic resources created by the Discovery Service.

version: v3
teleport:
  join_params:
    token_name: "/tmp/token"
    method: token
  proxy_server: teleport.example.com:443
auth_service:
  enabled: off
proxy_service:
  enabled: off
ssh_service:
  enabled: off
discovery_service:
  enabled: "yes"
  discovery_group: "aws-prod"
  aws:
   - types: ["eks"]
     regions: ["*"]
     tags:
       "env": "prod" # Match EKS cluster tags where tag:env=prod
kubernetes_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod" # Match Kubernetes Cluster labels specified earlier

Start the Kubernetes and Discovery Services

Grant the Kubernetes and Discovery Services access to credentials that it can use to authenticate to AWS.

• If you are running the Kubernetes and Discovery Services on an EC2 instance, you may use the EC2 Instance Metadata Service method
• If you are running the Kubernetes and Discovery Services in Kubernetes, you can use IAM Roles for Service Accounts (IRSA)
• Otherwise, you must use environment variables

AWS_ACCESS_KEY_ID=00000000000000000000
AWS_SECRET_ACCESS_KEY=0000000000000000000000000000000000000000
AWS_DEFAULT_REGION=<YOUR_REGION>

Configure the Kubernetes and Discovery Services to start automatically when the host boots up by creating a systemd service for it. The instructions depend on how you installed the Kubernetes and Discovery Services.

sudo systemctl enable teleport
sudo systemctl start teleport

sudo teleport install systemd -o /etc/systemd/system/teleport.service
sudo systemctl enable teleport
sudo systemctl start teleport

You can check the status of the Kubernetes and Discovery Services with systemctl status teleport and view its logs with journalctl -fu teleport.

Once the Kubernetes and Discovery Services start, EKS clusters matching the tags and regions specified in the AWS section will be added to the Teleport cluster automatically.

Troubleshooting

Discovery Service troubleshooting

First, check if any Kubernetes clusters have been discovered. To do this, you can use the tctl get kube_cluster command and check if the expected Kubernetes clusters have already been registered with your Teleport cluster.

If some Kubernetes clusters do not appear in the list, check if the Discovery Service selector labels match the missing Kubernetes cluster tags or look into the Discovery Service logs for permission errors.

Check that the Discovery Service is running with credentials for the correct AWS account. It can discover resources in another AWS account, but it must be configured to assume a role in the other AWS account if that's the case.

Check if there is more than one Discovery Services running:

tctl inventory status --connected

If you are running multiple Discovery Services, you must ensure that each service is configured with the same discovery_group value if they are watching the same cloud Kubernetes clusters or a different value if they are watching different cloud Kubernetes clusters. If this is not configured correctly, a typical symptom is kube_cluster resources being intermittently deleted from your Teleport cluster's registry.

Kubernetes Service troubleshooting

If the tctl get kube_cluster command returns the discovered clusters, but the tctl kube ls command does not include them, check that you have set the kubernetes_service.resources section correctly.

kubernetes_service:
  enabled: "yes"
  resources:
  - labels:
      "env": "prod"

If the section is correctly configured, but clusters still do not appear or return authentication errors, please check if permissions have been correctly configured in your target cluster or that you have the correct permissions to list Kubernetes clusters in Teleport.