diff --git a/charts/karpenter/Chart.yaml b/charts/karpenter/Chart.yaml index 218265b..29dbc2b 100644 --- a/charts/karpenter/Chart.yaml +++ b/charts/karpenter/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: karpenter description: A Helm chart for Karpenter, an open-source node provisioning project built for Kubernetes. type: application -version: 0.28.0 -appVersion: 0.28.0 +version: 0.32.3 +appVersion: 0.32.3 keywords: - cluster - node diff --git a/charts/karpenter/README.md b/charts/karpenter/README.md index eaa5cec..8bcc9e3 100644 --- a/charts/karpenter/README.md +++ b/charts/karpenter/README.md @@ -2,25 +2,24 @@ A Helm chart for Karpenter, an open-source node provisioning project built for Kubernetes. -![Version: 0.28.0](https://img.shields.io/badge/Version-0.28.0-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.28.0](https://img.shields.io/badge/AppVersion-0.28.0-informational?style=flat-square) +![Version: 0.32.3](https://img.shields.io/badge/Version-0.32.3-informational?style=flat-square) ![Type: application](https://img.shields.io/badge/Type-application-informational?style=flat-square) ![AppVersion: 0.32.3](https://img.shields.io/badge/AppVersion-0.32.3-informational?style=flat-square) ## Documentation -For full Karpenter documentation please checkout [https://karpenter.sh](https://karpenter.sh/v0.28.0/). +For full Karpenter documentation please checkout [https://karpenter.sh](https://karpenter.sh/docs/). ## Installing the Chart -You can follow the detailed installation instruction in the [documentation](https://karpenter.sh/v0.28.0/getting-started/getting-started-with-karpenter/#install) which covers the Karpenter prerequisites and installation options. The outcome of these instructions should result in something like the following command. +You can follow the detailed installation instruction in the [documentation](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/#install) which covers the Karpenter prerequisites and installation options. The outcome of these instructions should result in something like the following command. ```bash helm upgrade --install --namespace karpenter --create-namespace \ karpenter oci://public.ecr.aws/karpenter/karpenter \ - --version v0.28.0 \ + --version v0.32.3 \ --set serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN} \ - --set settings.aws.clusterName=${CLUSTER_NAME} \ - --set settings.aws.clusterEndpoint=${CLUSTER_ENDPOINT} \ - --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \ - --set settings.aws.interruptionQueueName=${CLUSTER_NAME} \ + --set settings.clusterName=${CLUSTER_NAME} \ + --set settings.clusterEndpoint=${CLUSTER_ENDPOINT} \ + --set settings.interruptionQueue=${CLUSTER_NAME} \ --wait ``` @@ -31,21 +30,16 @@ helm upgrade --install --namespace karpenter --create-namespace \ | additionalAnnotations | object | `{}` | Additional annotations to add into metadata. | | additionalClusterRoleRules | list | `[]` | Specifies additional rules for the core ClusterRole. | | additionalLabels | object | `{}` | Additional labels to add into metadata. | -| affinity | object | `{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"karpenter.sh/provisioner-name","operator":"DoesNotExist"}]}]}},"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"topologyKey":"kubernetes.io/hostname"}]}}` | Affinity rules for scheduling the pod. If an explicit label selector is not provided for pod affinity or pod anti-affinity one will be created from the pod selector labels. | +| affinity | object | `{"nodeAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":{"nodeSelectorTerms":[{"matchExpressions":[{"key":"karpenter.sh/provisioner-name","operator":"DoesNotExist"},{"key":"karpenter.sh/nodepool","operator":"DoesNotExist"}]}]}},"podAntiAffinity":{"requiredDuringSchedulingIgnoredDuringExecution":[{"topologyKey":"kubernetes.io/hostname"}]}}` | Affinity rules for scheduling the pod. If an explicit label selector is not provided for pod affinity or pod anti-affinity one will be created from the pod selector labels. | | controller.env | list | `[]` | Additional environment variables for the controller pod. | | controller.envFrom | list | `[]` | | -| controller.errorOutputPaths | list | `["stderr"]` | Controller errorOutputPaths - default to stderr only | | controller.extraVolumeMounts | list | `[]` | Additional volumeMounts for the controller pod. | | controller.healthProbe.port | int | `8081` | The container port to use for http health probe. | -| controller.image.digest | string | `"sha256:4194fc219d5a1019cd8d7d69f518b903e972966cad2e9420ccbe9dca149f179d"` | SHA256 digest of the controller image. | +| controller.image.digest | string | `"sha256:afa0d0fd5ac375859dc3d239ec992f197cdf01f6c8e3413e3845a43c2434621e"` | SHA256 digest of the controller image. | | controller.image.repository | string | `"public.ecr.aws/karpenter/controller"` | Repository path to the controller image. | -| controller.image.tag | string | `"v0.28.0"` | Tag of the controller image. | -| controller.logEncoding | string | `""` | Controller log encoding, defaults to the global log encoding | -| controller.logLevel | string | `""` | Controller log level, defaults to the global log level | +| controller.image.tag | string | `"v0.32.3"` | Tag of the controller image. | | controller.metrics.port | int | `8000` | The container port to use for metrics. | -| controller.outputPaths | list | `["stdout"]` | Controller outputPaths - default to stdout only | | controller.resources | object | `{}` | Resources for the controller pod. | -| controller.securityContext | object | `{}` | SecurityContext for the controller container. | | controller.sidecarContainer | list | `[]` | Additional sidecarContainer config | | controller.sidecarVolumeMounts | list | `[]` | Additional volumeMounts for the sidecar - this will be added to the volume mounts on top of extraVolumeMounts | | dnsConfig | object | `{}` | Configure DNS Config for the pod | @@ -55,7 +49,15 @@ helm upgrade --install --namespace karpenter --create-namespace \ | hostNetwork | bool | `false` | Bind the pod to the host network. This is required when using a custom CNI. | | imagePullPolicy | string | `"IfNotPresent"` | Image pull policy for Docker images. | | imagePullSecrets | list | `[]` | Image pull secrets for Docker images. | -| logEncoding | string | `"console"` | Global log encoding | +| logConfig | object | `{"enabled":true,"errorOutputPaths":["stderr"],"logEncoding":"json","logLevel":{"controller":"debug","global":"debug","webhook":"error"},"outputPaths":["stdout"]}` | Log configuration (Deprecated: Logging configuration will be dropped by v1, use logLevel instead) | +| logConfig.enabled | bool | `true` | Whether to enable provisioning and mounting the log ConfigMap | +| logConfig.errorOutputPaths | list | `["stderr"]` | Log errorOutputPaths - defaults to stderr only | +| logConfig.logEncoding | string | `"json"` | Log encoding - defaults to json - must be one of 'json', 'console' | +| logConfig.logLevel | object | `{"controller":"debug","global":"debug","webhook":"error"}` | Component-based log configuration | +| logConfig.logLevel.controller | string | `"debug"` | Controller log level, defaults to 'debug' | +| logConfig.logLevel.global | string | `"debug"` | Global log level, defaults to 'debug' | +| logConfig.logLevel.webhook | string | `"error"` | Error log level, defaults to 'error' | +| logConfig.outputPaths | list | `["stdout"]` | Log outputPaths - defaults to stdout only | | logLevel | string | `"debug"` | Global log level | | nameOverride | string | `""` | Overrides the chart's name. | | nodeSelector | object | `{"kubernetes.io/os":"linux"}` | Node selectors to schedule the pod to nodes with labels. | @@ -63,7 +65,6 @@ helm upgrade --install --namespace karpenter --create-namespace \ | podDisruptionBudget.maxUnavailable | int | `1` | | | podDisruptionBudget.name | string | `"karpenter"` | | | podLabels | object | `{}` | Additional labels for the pod. | -| podSecurityContext | object | `{"fsGroup":1000}` | SecurityContext for the pod. | | priorityClassName | string | `"system-cluster-critical"` | PriorityClass name for the pod. | | replicas | int | `2` | Number of replicas. | | revisionHistoryLimit | int | `10` | The number of old ReplicaSets to retain to allow rollback. | @@ -73,24 +74,25 @@ helm upgrade --install --namespace karpenter --create-namespace \ | serviceMonitor.additionalLabels | object | `{}` | Additional labels for the ServiceMonitor. | | serviceMonitor.enabled | bool | `false` | Specifies whether a ServiceMonitor should be created. | | serviceMonitor.endpointConfig | object | `{}` | Endpoint configuration for the ServiceMonitor. | -| settings | object | `{"aws":{"clusterEndpoint":"","clusterName":"","defaultInstanceProfile":"","enableENILimitedPodDensity":true,"enablePodENI":false,"interruptionQueueName":"","isolatedVPC":false,"tags":null,"vmMemoryOverheadPercent":0.075},"batchIdleDuration":"1s","batchMaxDuration":"10s","featureGates":{"driftEnabled":false}}` | Global Settings to configure Karpenter | -| settings.aws | object | `{"clusterEndpoint":"","clusterName":"","defaultInstanceProfile":"","enableENILimitedPodDensity":true,"enablePodENI":false,"interruptionQueueName":"","isolatedVPC":false,"tags":null,"vmMemoryOverheadPercent":0.075}` | AWS-specific configuration values | -| settings.aws.clusterEndpoint | string | `""` | Cluster endpoint. If not set, will be discovered during startup (EKS only) | -| settings.aws.clusterName | string | `""` | Cluster name. | -| settings.aws.defaultInstanceProfile | string | `""` | The default instance profile to use when launching nodes | -| settings.aws.enableENILimitedPodDensity | bool | `true` | Indicates whether new nodes should use ENI-based pod density DEPRECATED: Use `.spec.kubeletConfiguration.maxPods` to set pod density on a per-provisioner basis | -| settings.aws.enablePodENI | bool | `false` | If true then instances that support pod ENI will report a vpc.amazonaws.com/pod-eni resource | -| settings.aws.interruptionQueueName | string | `""` | interruptionQueueName is disabled if not specified. Enabling interruption handling may require additional permissions on the controller service account. Additional permissions are outlined in the docs. | -| settings.aws.isolatedVPC | bool | `false` | If true then assume we can't reach AWS services which don't have a VPC endpoint This also has the effect of disabling look-ups to the AWS pricing endpoint | -| settings.aws.tags | string | `nil` | The global tags to use on all AWS infrastructure resources (launch templates, instances, etc.) across node templates | -| settings.aws.vmMemoryOverheadPercent | float | `0.075` | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types | +| settings | object | `{"assumeRoleARN":"","assumeRoleDuration":"15m","aws":{},"batchIdleDuration":"1s","batchMaxDuration":"10s","clusterCABundle":"","clusterEndpoint":"","clusterName":"","featureGates":{"drift":false},"interruptionQueue":"","isolatedVPC":false,"reservedENIs":"0","vmMemoryOverheadPercent":0.075}` | Global Settings to configure Karpenter | +| settings.assumeRoleARN | string | `""` | Role to assume for calling AWS services. | +| settings.assumeRoleDuration | string | `"15m"` | Duration of assumed credentials in minutes. Default value is 15 minutes. Not used unless assumeRoleARN set. | +| settings.aws | object | `{}` | AWS-specific settings (Deprecated: The AWS block inside of settings was flattened into settings) | | settings.batchIdleDuration | string | `"1s"` | The maximum amount of time with no new ending pods that if exceeded ends the current batching window. If pods arrive faster than this time, the batching window will be extended up to the maxDuration. If they arrive slower, the pods will be batched separately. | | settings.batchMaxDuration | string | `"10s"` | The maximum length of a batch window. The longer this is, the more pods we can consider for provisioning at one time which usually results in fewer but larger nodes. | -| settings.featureGates | object | `{"driftEnabled":false}` | Feature Gate configuration values. Feature Gates will follow the same graduation process and requirements as feature gates in Kubernetes. More information here https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features | -| settings.featureGates.driftEnabled | bool | `false` | driftEnabled is in ALPHA and is disabled by default. Setting driftEnabled to true enables the drift deprovisioner to watch for drift between currently deployed nodes and the desired state of nodes set in provisioners and node templates | +| settings.clusterCABundle | string | `""` | Cluster CA bundle for TLS configuration of provisioned nodes. If not set, this is taken from the controller's TLS configuration for the API server. | +| settings.clusterEndpoint | string | `""` | Cluster endpoint. If not set, will be discovered during startup (EKS only) | +| settings.clusterName | string | `""` | Cluster name. | +| settings.featureGates | object | `{"drift":false}` | Feature Gate configuration values. Feature Gates will follow the same graduation process and requirements as feature gates in Kubernetes. More information here https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features | +| settings.featureGates.drift | bool | `false` | drift is in ALPHA and is disabled by default. Setting drift to true enables the drift disruption method to watch for drift between currently deployed nodes and the desired state of nodes set in provisioners and node templates | +| settings.interruptionQueue | string | `""` | interruptionQueue is disabled if not specified. Enabling interruption handling may require additional permissions on the controller service account. Additional permissions are outlined in the docs. | +| settings.isolatedVPC | bool | `false` | If true then assume we can't reach AWS services which don't have a VPC endpoint This also has the effect of disabling look-ups to the AWS pricing endpoint | +| settings.reservedENIs | string | `"0"` | Reserved ENIs are not included in the calculations for max-pods or kube-reserved This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html | +| settings.vmMemoryOverheadPercent | float | `0.075` | The VM memory overhead as a percent that will be subtracted from the total memory for all instance types | | strategy | object | `{"rollingUpdate":{"maxUnavailable":1}}` | Strategy for updating the pod. | | terminationGracePeriodSeconds | string | `nil` | Override the default termination grace period for the pod. | | tolerations | list | `[{"key":"CriticalAddonsOnly","operator":"Exists"}]` | Tolerations to allow the pod to be scheduled to nodes with taints. | | topologySpreadConstraints | list | `[{"maxSkew":1,"topologyKey":"topology.kubernetes.io/zone","whenUnsatisfiable":"ScheduleAnyway"}]` | Topology spread constraints to increase the controller resilience by distributing pods across the cluster zones. If an explicit label selector is not provided one will be created from the pod selector labels. | -| webhook.logLevel | string | `"error"` | | +| webhook.enabled | bool | `true` | Whether to enable the webhooks and webhook permissions. | +| webhook.metrics.port | int | `8001` | The container port to use for webhook metrics. | | webhook.port | int | `8443` | The container port to use for the webhook. | diff --git a/charts/karpenter/README.md.gotmpl b/charts/karpenter/README.md.gotmpl index 7ec4d42..75d5e12 100644 --- a/charts/karpenter/README.md.gotmpl +++ b/charts/karpenter/README.md.gotmpl @@ -5,21 +5,20 @@ ## Documentation -For full Karpenter documentation please checkout [https://karpenter.sh](https://karpenter.sh/v{{ template "chart.version" . }}/). +For full Karpenter documentation please checkout [https://karpenter.sh](https://karpenter.sh/docs/). ## Installing the Chart -You can follow the detailed installation instruction in the [documentation](https://karpenter.sh/v{{ template "chart.version" . }}/getting-started/getting-started-with-karpenter/#install) which covers the Karpenter prerequisites and installation options. The outcome of these instructions should result in something like the following command. +You can follow the detailed installation instruction in the [documentation](https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/#install) which covers the Karpenter prerequisites and installation options. The outcome of these instructions should result in something like the following command. ```bash helm upgrade --install --namespace karpenter --create-namespace \ karpenter oci://public.ecr.aws/karpenter/{{ template "chart.name" . }} \ --version v{{ template "chart.version" . }} \ --set serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${KARPENTER_IAM_ROLE_ARN} \ - --set settings.aws.clusterName=${CLUSTER_NAME} \ - --set settings.aws.clusterEndpoint=${CLUSTER_ENDPOINT} \ - --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \ - --set settings.aws.interruptionQueueName=${CLUSTER_NAME} \ + --set settings.clusterName=${CLUSTER_NAME} \ + --set settings.clusterEndpoint=${CLUSTER_ENDPOINT} \ + --set settings.interruptionQueue=${CLUSTER_NAME} \ --wait ``` diff --git a/charts/karpenter/crds/karpenter.k8s.aws_ec2nodeclasses.yaml b/charts/karpenter/crds/karpenter.k8s.aws_ec2nodeclasses.yaml new file mode 100644 index 0000000..2daaacb --- /dev/null +++ b/charts/karpenter/crds/karpenter.k8s.aws_ec2nodeclasses.yaml @@ -0,0 +1 @@ +../pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml diff --git a/charts/karpenter/crds/karpenter.sh_nodeclaims.yaml b/charts/karpenter/crds/karpenter.sh_nodeclaims.yaml new file mode 100644 index 0000000..9ab4cbe --- /dev/null +++ b/charts/karpenter/crds/karpenter.sh_nodeclaims.yaml @@ -0,0 +1 @@ +../pkg/apis/crds/karpenter.sh_nodeclaims.yaml diff --git a/charts/karpenter/crds/karpenter.sh_nodepools.yaml b/charts/karpenter/crds/karpenter.sh_nodepools.yaml new file mode 100644 index 0000000..efc1eb2 --- /dev/null +++ b/charts/karpenter/crds/karpenter.sh_nodepools.yaml @@ -0,0 +1 @@ +../pkg/apis/crds/karpenter.sh_nodepools.yaml diff --git a/charts/karpenter/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml b/charts/karpenter/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml new file mode 100644 index 0000000..ae31eb2 --- /dev/null +++ b/charts/karpenter/pkg/apis/crds/karpenter.k8s.aws_ec2nodeclasses.yaml @@ -0,0 +1,555 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.13.0 + name: ec2nodeclasses.karpenter.k8s.aws +spec: + group: karpenter.k8s.aws + names: + categories: + - karpenter + kind: EC2NodeClass + listKind: EC2NodeClassList + plural: ec2nodeclasses + shortNames: + - ec2nc + - ec2ncs + singular: ec2nodeclass + scope: Cluster + versions: + - name: v1beta1 + schema: + openAPIV3Schema: + description: EC2NodeClass is the Schema for the EC2NodeClass API + properties: + apiVersion: + description: + "APIVersion defines the versioned schema of this representation + of an object. Servers should convert recognized schemas to the latest + internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources" + type: string + kind: + description: + "Kind is a string value representing the REST resource this + object represents. Servers may infer this from the endpoint the client + submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds" + type: string + metadata: + type: object + spec: + description: + EC2NodeClassSpec is the top level specification for the AWS + Karpenter Provider. This will contain configuration necessary to launch + instances in AWS. + properties: + amiFamily: + description: AMIFamily is the AMI family that instances use. + enum: + - AL2 + - Bottlerocket + - Ubuntu + - Custom + - Windows2019 + - Windows2022 + type: string + amiSelectorTerms: + description: + AMISelectorTerms is a list of or ami selector terms. + The terms are ORed. + items: + description: + AMISelectorTerm defines selection logic for an ami + used by Karpenter to launch nodes. If multiple fields are used + for selection, the requirements are ANDed. + properties: + id: + description: ID is the ami id in EC2 + pattern: ami-[0-9a-z]+ + type: string + name: + description: + Name is the ami name in EC2. This value is the + name field, which is different from the name tag. + type: string + owner: + description: + Owner is the owner for the ami. You can specify + a combination of AWS account IDs, "self", "amazon", and "aws-marketplace" + type: string + tags: + additionalProperties: + type: string + description: + Tags is a map of key/value tags used to select + subnets Specifying '*' for a value selects all values for + a given tag key. + maxProperties: 20 + type: object + x-kubernetes-validations: + - message: empty tag keys or values aren't supported + rule: self.all(k, k != '' && self[k] != '') + type: object + maxItems: 30 + type: array + x-kubernetes-validations: + - message: expected at least one, got none, ['tags', 'id', 'name'] + rule: self.all(x, has(x.tags) || has(x.id) || has(x.name)) + - message: + "'id' is mutually exclusive, cannot be set with a combination + of other fields in amiSelectorTerms" + rule: + "!self.all(x, has(x.id) && (has(x.tags) || has(x.name) || + has(x.owner)))" + blockDeviceMappings: + description: BlockDeviceMappings to be applied to provisioned nodes. + items: + properties: + deviceName: + description: The device name (for example, /dev/sdh or xvdh). + type: string + ebs: + description: + EBS contains parameters used to automatically set + up EBS volumes when an instance is launched. + properties: + deleteOnTermination: + description: + DeleteOnTermination indicates whether the EBS + volume is deleted on instance termination. + type: boolean + encrypted: + description: + Encrypted indicates whether the EBS volume + is encrypted. Encrypted volumes can only be attached to + instances that support Amazon EBS encryption. If you are + creating a volume from a snapshot, you can't specify an + encryption value. + type: boolean + iops: + description: + "IOPS is the number of I/O operations per second + (IOPS). For gp3, io1, and io2 volumes, this represents + the number of IOPS that are provisioned for the volume. + For gp2 volumes, this represents the baseline performance + of the volume and the rate at which the volume accumulates + I/O credits for bursting. \n The following are the supported + values for each volume type: \n * gp3: 3,000-16,000 IOPS + \n * io1: 100-64,000 IOPS \n * io2: 100-64,000 IOPS \n + For io1 and io2 volumes, we guarantee 64,000 IOPS only + for Instances built on the Nitro System (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#ec2-nitro-instances). + Other instance families guarantee performance up to 32,000 + IOPS. \n This parameter is supported for io1, io2, and + gp3 volumes only. This parameter is not supported for + gp2, st1, sc1, or standard volumes." + format: int64 + type: integer + kmsKeyID: + description: + KMSKeyID (ARN) of the symmetric Key Management + Service (KMS) CMK used for encryption. + type: string + snapshotID: + description: SnapshotID is the ID of an EBS snapshot + type: string + throughput: + description: + "Throughput to provision for a gp3 volume, + with a maximum of 1,000 MiB/s. Valid Range: Minimum value + of 125. Maximum value of 1000." + format: int64 + type: integer + volumeSize: + allOf: + - pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + - pattern: ^((?:[1-9][0-9]{0,3}|[1-4][0-9]{4}|[5][0-8][0-9]{3}|59000)Gi|(?:[1-9][0-9]{0,3}|[1-5][0-9]{4}|[6][0-3][0-9]{3}|64000)G|([1-9]||[1-5][0-7]|58)Ti|([1-9]||[1-5][0-9]|6[0-3]|64)T)$ + anyOf: + - type: integer + - type: string + description: + "VolumeSize in `Gi`, `G`, `Ti`, or `T`. You + must specify either a snapshot ID or a volume size. The + following are the supported volumes sizes for each volume + type: \n * gp2 and gp3: 1-16,384 \n * io1 and io2: 4-16,384 + \n * st1 and sc1: 125-16,384 \n * standard: 1-1,024" + x-kubernetes-int-or-string: true + volumeType: + description: + VolumeType of the block device. For more information, + see Amazon EBS volume types (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolumeTypes.html) + in the Amazon Elastic Compute Cloud User Guide. + enum: + - standard + - io1 + - io2 + - gp2 + - sc1 + - st1 + - gp3 + type: string + type: object + x-kubernetes-validations: + - message: snapshotID or volumeSize must be defined + rule: has(self.snapshotID) || has(self.volumeSize) + rootVolume: + description: + RootVolume is a flag indicating if this device + is mounted as kubelet root dir. You can configure at most + one root volume in BlockDeviceMappings. + type: boolean + type: object + maxItems: 50 + type: array + x-kubernetes-validations: + - message: must have only one blockDeviceMappings with rootVolume + rule: + self.filter(x, has(x.rootVolume)?x.rootVolume==true:false).size() + <= 1 + context: + description: Context is a Reserved field in EC2 APIs https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateFleet.html + type: string + detailedMonitoring: + description: + DetailedMonitoring controls if detailed monitoring is + enabled for instances that are launched + type: boolean + instanceProfile: + description: + InstanceProfile is the AWS entity that instances use. + This field is mutually exclusive from role. The instance profile + should already have a role assigned to it that Karpenter has PassRole + permission on for instance launch using this instanceProfile to + succeed. + type: string + x-kubernetes-validations: + - message: instanceProfile cannot be empty + rule: self != '' + metadataOptions: + default: + httpEndpoint: enabled + httpProtocolIPv6: disabled + httpPutResponseHopLimit: 2 + httpTokens: required + description: + "MetadataOptions for the generated launch template of + provisioned nodes. \n This specifies the exposure of the Instance + Metadata Service to provisioned EC2 nodes. For more information, + see Instance Metadata and User Data (https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html) + in the Amazon Elastic Compute Cloud User Guide. \n Refer to recommended, + security best practices (https://aws.github.io/aws-eks-best-practices/security/docs/iam/#restrict-access-to-the-instance-profile-assigned-to-the-worker-node) + for limiting exposure of Instance Metadata and User Data to pods. + If omitted, defaults to httpEndpoint enabled, with httpProtocolIPv6 + disabled, with httpPutResponseLimit of 2, and with httpTokens required." + properties: + httpEndpoint: + default: enabled + description: + "HTTPEndpoint enables or disables the HTTP metadata + endpoint on provisioned nodes. If metadata options is non-nil, + but this parameter is not specified, the default state is \"enabled\". + \n If you specify a value of \"disabled\", instance metadata + will not be accessible on the node." + enum: + - enabled + - disabled + type: string + httpProtocolIPv6: + default: disabled + description: + HTTPProtocolIPv6 enables or disables the IPv6 endpoint + for the instance metadata service on provisioned nodes. If metadata + options is non-nil, but this parameter is not specified, the + default state is "disabled". + enum: + - enabled + - disabled + type: string + httpPutResponseHopLimit: + default: 2 + description: + HTTPPutResponseHopLimit is the desired HTTP PUT response + hop limit for instance metadata requests. The larger the number, + the further instance metadata requests can travel. Possible + values are integers from 1 to 64. If metadata options is non-nil, + but this parameter is not specified, the default value is 2. + format: int64 + maximum: 64 + minimum: 1 + type: integer + httpTokens: + default: required + description: + "HTTPTokens determines the state of token usage for + instance metadata requests. If metadata options is non-nil, + but this parameter is not specified, the default state is \"required\". + \n If the state is optional, one can choose to retrieve instance + metadata with or without a signed token header on the request. + If one retrieves the IAM role credentials without a token, the + version 1.0 role credentials are returned. If one retrieves + the IAM role credentials using a valid signed token, the version + 2.0 role credentials are returned. \n If the state is \"required\", + one must send a signed token header with any instance metadata + retrieval requests. In this state, retrieving the IAM role credentials + always returns the version 2.0 credentials; the version 1.0 + credentials are not available." + enum: + - required + - optional + type: string + type: object + role: + description: + Role is the AWS identity that nodes use. This field is + immutable. This field is mutually exclusive from instanceProfile. + Marking this field as immutable avoids concerns around terminating + managed instance profiles from running instances. This field may + be made mutable in the future, assuming the correct garbage collection + and drift handling is implemented for the old instance profiles + on an update. + type: string + x-kubernetes-validations: + - message: role cannot be empty + rule: self != '' + - message: immutable field changed + rule: self == oldSelf + securityGroupSelectorTerms: + description: + SecurityGroupSelectorTerms is a list of or security group + selector terms. The terms are ORed. + items: + description: + SecurityGroupSelectorTerm defines selection logic for + a security group used by Karpenter to launch nodes. If multiple + fields are used for selection, the requirements are ANDed. + properties: + id: + description: ID is the security group id in EC2 + pattern: sg-[0-9a-z]+ + type: string + name: + description: + Name is the security group name in EC2. This value + is the name field, which is different from the name tag. + type: string + tags: + additionalProperties: + type: string + description: + Tags is a map of key/value tags used to select + subnets Specifying '*' for a value selects all values for + a given tag key. + maxProperties: 20 + type: object + x-kubernetes-validations: + - message: empty tag keys or values aren't supported + rule: self.all(k, k != '' && self[k] != '') + type: object + maxItems: 30 + type: array + x-kubernetes-validations: + - message: securityGroupSelectorTerms cannot be empty + rule: self.size() != 0 + - message: expected at least one, got none, ['tags', 'id', 'name'] + rule: self.all(x, has(x.tags) || has(x.id) || has(x.name)) + - message: + "'id' is mutually exclusive, cannot be set with a combination + of other fields in securityGroupSelectorTerms" + rule: "!self.all(x, has(x.id) && (has(x.tags) || has(x.name)))" + - message: + "'name' is mutually exclusive, cannot be set with a combination + of other fields in securityGroupSelectorTerms" + rule: "!self.all(x, has(x.name) && (has(x.tags) || has(x.id)))" + subnetSelectorTerms: + description: + SubnetSelectorTerms is a list of or subnet selector terms. + The terms are ORed. + items: + description: + SubnetSelectorTerm defines selection logic for a subnet + used by Karpenter to launch nodes. If multiple fields are used + for selection, the requirements are ANDed. + properties: + id: + description: ID is the subnet id in EC2 + pattern: subnet-[0-9a-z]+ + type: string + tags: + additionalProperties: + type: string + description: + Tags is a map of key/value tags used to select + subnets Specifying '*' for a value selects all values for + a given tag key. + maxProperties: 20 + type: object + x-kubernetes-validations: + - message: empty tag keys or values aren't supported + rule: self.all(k, k != '' && self[k] != '') + type: object + maxItems: 30 + type: array + x-kubernetes-validations: + - message: subnetSelectorTerms cannot be empty + rule: self.size() != 0 + - message: expected at least one, got none, ['tags', 'id'] + rule: self.all(x, has(x.tags) || has(x.id)) + - message: + "'id' is mutually exclusive, cannot be set with a combination + of other fields in subnetSelectorTerms" + rule: "!self.all(x, has(x.id) && has(x.tags))" + tags: + additionalProperties: + type: string + description: + Tags to be applied on ec2 resources like instances and + launch templates. + type: object + x-kubernetes-validations: + - message: empty tag keys aren't supported + rule: self.all(k, k != '') + - message: tag contains a restricted tag matching kubernetes.io/cluster/ + rule: self.all(k, !k.startsWith('kubernetes.io/cluster') ) + - message: tag contains a restricted tag matching karpenter.sh/provisioner-name + rule: self.all(k, k != 'karpenter.sh/provisioner-name') + - message: tag contains a restricted tag matching karpenter.sh/nodepool + rule: self.all(k, k != 'karpenter.sh/nodepool') + - message: tag contains a restricted tag matching karpenter.sh/managed-by + rule: self.all(k, k !='karpenter.sh/managed-by') + userData: + description: + UserData to be applied to the provisioned nodes. It must + be in the appropriate format based on the AMIFamily in use. Karpenter + will merge certain fields into this UserData to ensure nodes are + being provisioned with the correct configuration. + type: string + required: + - amiFamily + - securityGroupSelectorTerms + - subnetSelectorTerms + type: object + x-kubernetes-validations: + - message: amiSelectorTerms is required when amiFamily == 'Custom' + rule: + "self.amiFamily == 'Custom' ? self.amiSelectorTerms.size() != + 0 : true" + - message: must specify exactly one of ['role', 'instanceProfile'] + rule: + (has(self.role) && !has(self.instanceProfile)) || (!has(self.role) + && has(self.instanceProfile)) + - message: + changing from 'instanceProfile' to 'role' is not supported. + You must delete and recreate this node class if you want to change + this. + rule: + (has(oldSelf.role) && has(self.role)) || (has(oldSelf.instanceProfile) + && has(self.instanceProfile)) + status: + description: EC2NodeClassStatus contains the resolved state of the EC2NodeClass + properties: + amis: + description: + AMI contains the current AMI values that are available + to the cluster under the AMI selectors. + items: + description: + AMI contains resolved AMI selector values utilized + for node launch + properties: + id: + description: ID of the AMI + type: string + name: + description: Name of the AMI + type: string + requirements: + description: + Requirements of the AMI to be utilized on an instance + type + items: + description: + A node selector requirement is a selector that + contains values, a key, and an operator that relates the + key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + operator: + description: + Represents a key's relationship to a set + of values. Valid operators are In, NotIn, Exists, DoesNotExist. + Gt, and Lt. + type: string + values: + description: + An array of string values. If the operator + is In or NotIn, the values array must be non-empty. + If the operator is Exists or DoesNotExist, the values + array must be empty. If the operator is Gt or Lt, the + values array must have a single element, which will + be interpreted as an integer. This array is replaced + during a strategic merge patch. + items: + type: string + type: array + required: + - key + - operator + type: object + type: array + required: + - id + - requirements + type: object + type: array + instanceProfile: + description: + InstanceProfile contains the resolved instance profile + for the role + type: string + securityGroups: + description: + SecurityGroups contains the current Security Groups values + that are available to the cluster under the SecurityGroups selectors. + items: + description: + SecurityGroup contains resolved SecurityGroup selector + values utilized for node launch + properties: + id: + description: ID of the security group + type: string + name: + description: Name of the security group + type: string + required: + - id + type: object + type: array + subnets: + description: + Subnets contains the current Subnet values that are available + to the cluster under the subnet selectors. + items: + description: + Subnet contains resolved Subnet selector values utilized + for node launch + properties: + id: + description: ID of the subnet + type: string + zone: + description: The associated availability zone + type: string + required: + - id + - zone + type: object + type: array + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/charts/karpenter/pkg/apis/crds/karpenter.sh_nodeclaims.yaml b/charts/karpenter/pkg/apis/crds/karpenter.sh_nodeclaims.yaml new file mode 100644 index 0000000..3d5d08c --- /dev/null +++ b/charts/karpenter/pkg/apis/crds/karpenter.sh_nodeclaims.yaml @@ -0,0 +1,366 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.13.0 + name: nodeclaims.karpenter.sh +spec: + group: karpenter.sh + names: + categories: + - karpenter + kind: NodeClaim + listKind: NodeClaimList + plural: nodeclaims + singular: nodeclaim + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .metadata.labels.node\.kubernetes\.io/instance-type + name: Type + type: string + - jsonPath: .metadata.labels.topology\.kubernetes\.io/zone + name: Zone + type: string + - jsonPath: .status.nodeName + name: Node + type: string + - jsonPath: .status.conditions[?(@.type=="Ready")].status + name: Ready + type: string + - jsonPath: .metadata.creationTimestamp + name: Age + type: date + - jsonPath: .metadata.labels.karpenter\.sh/capacity-type + name: Capacity + priority: 1 + type: string + - jsonPath: .metadata.labels.karpenter\.sh/nodepool + name: NodePool + priority: 1 + type: string + - jsonPath: .spec.nodeClassRef.name + name: NodeClass + priority: 1 + type: string + name: v1beta1 + schema: + openAPIV3Schema: + description: NodeClaim is the Schema for the NodeClaims API + properties: + apiVersion: + description: "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources" + type: string + kind: + description: "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds" + type: string + metadata: + type: object + spec: + description: NodeClaimSpec describes the desired state of the NodeClaim + properties: + kubelet: + description: Kubelet defines args to be used when configuring kubelet on provisioned nodes. They are a subset of the upstream types, recognizing not all options may be supported. Wherever possible, the types and names should reflect the upstream kubelet types. + properties: + clusterDNS: + description: clusterDNS is a list of IP addresses for the cluster DNS server. Note that not all providers may use all addresses. + items: + type: string + type: array + cpuCFSQuota: + description: CPUCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits. + type: boolean + evictionHard: + additionalProperties: + type: string + pattern: ^((\d{1,2}(\.\d{1,2})?|100(\.0{1,2})?)%||(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?)$ + description: EvictionHard is the map of signal names to quantities that define hard eviction thresholds + type: object + x-kubernetes-validations: + - message: valid keys for evictionHard are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'] + rule: self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']) + evictionMaxPodGracePeriod: + description: EvictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use when terminating pods in response to soft eviction thresholds being met. + format: int32 + type: integer + evictionSoft: + additionalProperties: + type: string + pattern: ^((\d{1,2}(\.\d{1,2})?|100(\.0{1,2})?)%||(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?)$ + description: EvictionSoft is the map of signal names to quantities that define soft eviction thresholds + type: object + x-kubernetes-validations: + - message: valid keys for evictionSoft are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'] + rule: self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']) + evictionSoftGracePeriod: + additionalProperties: + type: string + description: EvictionSoftGracePeriod is the map of signal names to quantities that define grace periods for each eviction signal + type: object + x-kubernetes-validations: + - message: valid keys for evictionSoftGracePeriod are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'] + rule: self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']) + imageGCHighThresholdPercent: + description: ImageGCHighThresholdPercent is the percent of disk usage after which image garbage collection is always run. The percent is calculated by dividing this field value by 100, so this field must be between 0 and 100, inclusive. When specified, the value must be greater than ImageGCLowThresholdPercent. + format: int32 + maximum: 100 + minimum: 0 + type: integer + imageGCLowThresholdPercent: + description: ImageGCLowThresholdPercent is the percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. The percent is calculated by dividing this field value by 100, so the field value must be between 0 and 100, inclusive. When specified, the value must be less than imageGCHighThresholdPercent + format: int32 + maximum: 100 + minimum: 0 + type: integer + kubeReserved: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: KubeReserved contains resources reserved for Kubernetes system components. + type: object + x-kubernetes-validations: + - message: valid keys for kubeReserved are ['cpu','memory','ephemeral-storage','pid'] + rule: self.all(x, x=='cpu' || x=='memory' || x=='ephemeral-storage' || x=='pid') + - message: kubeReserved value cannot be a negative resource quantity + rule: self.all(x, !self[x].startsWith('-')) + maxPods: + description: MaxPods is an override for the maximum number of pods that can run on a worker node instance. + format: int32 + minimum: 0 + type: integer + podsPerCore: + description: PodsPerCore is an override for the number of pods that can run on a worker node instance based on the number of cpu cores. This value cannot exceed MaxPods, so, if MaxPods is a lower value, that value will be used. + format: int32 + minimum: 0 + type: integer + systemReserved: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: SystemReserved contains resources reserved for OS system daemons and kernel memory. + type: object + x-kubernetes-validations: + - message: valid keys for systemReserved are ['cpu','memory','ephemeral-storage','pid'] + rule: self.all(x, x=='cpu' || x=='memory' || x=='ephemeral-storage' || x=='pid') + - message: systemReserved value cannot be a negative resource quantity + rule: self.all(x, !self[x].startsWith('-')) + type: object + x-kubernetes-validations: + - message: imageGCHighThresholdPercent must be greater than imageGCLowThresholdPercent + rule: "has(self.imageGCHighThresholdPercent) && has(self.imageGCLowThresholdPercent) ? self.imageGCHighThresholdPercent > self.imageGCLowThresholdPercent : true" + - message: evictionSoft OwnerKey does not have a matching evictionSoftGracePeriod + rule: has(self.evictionSoft) ? self.evictionSoft.all(e, (e in self.evictionSoftGracePeriod)):true + - message: evictionSoftGracePeriod OwnerKey does not have a matching evictionSoft + rule: has(self.evictionSoftGracePeriod) ? self.evictionSoftGracePeriod.all(e, (e in self.evictionSoft)):true + nodeClassRef: + description: NodeClassRef is a reference to an object that defines provider specific configuration + properties: + apiVersion: + description: API version of the referent + type: string + kind: + description: 'Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"' + type: string + name: + description: "Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names" + type: string + required: + - name + type: object + requirements: + description: Requirements are layered with GetLabels and applied to every node. + items: + description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + x-kubernetes-validations: + - message: label domain "kubernetes.io" is restricted + rule: self in ["beta.kubernetes.io/instance-type", "failure-domain.beta.kubernetes.io/region", "beta.kubernetes.io/os", "beta.kubernetes.io/arch", "failure-domain.beta.kubernetes.io/zone", "topology.kubernetes.io/zone", "topology.kubernetes.io/region", "node.kubernetes.io/instance-type", "kubernetes.io/arch", "kubernetes.io/os", "node.kubernetes.io/windows-build"] || self.find("^([^/]+)").endsWith("node.kubernetes.io") || self.find("^([^/]+)").endsWith("node-restriction.kubernetes.io") || !self.find("^([^/]+)").endsWith("kubernetes.io") + - message: label domain "k8s.io" is restricted + rule: self.find("^([^/]+)").endsWith("kops.k8s.io") || !self.find("^([^/]+)").endsWith("k8s.io") + - message: label domain "karpenter.sh" is restricted + rule: self in ["karpenter.sh/capacity-type", "karpenter.sh/nodepool"] || !self.find("^([^/]+)").endsWith("karpenter.sh") + - message: label "kubernetes.io/hostname" is restricted + rule: self != "kubernetes.io/hostname" + - message: label domain "karpenter.k8s.aws" is restricted + rule: self in ["karpenter.k8s.aws/instance-encryption-in-transit-supported", "karpenter.k8s.aws/instance-category", "karpenter.k8s.aws/instance-hypervisor", "karpenter.k8s.aws/instance-family", "karpenter.k8s.aws/instance-generation", "karpenter.k8s.aws/instance-local-nvme", "karpenter.k8s.aws/instance-size", "karpenter.k8s.aws/instance-cpu","karpenter.k8s.aws/instance-memory", "karpenter.k8s.aws/instance-network-bandwidth", "karpenter.k8s.aws/instance-gpu-name", "karpenter.k8s.aws/instance-gpu-manufacturer", "karpenter.k8s.aws/instance-gpu-count", "karpenter.k8s.aws/instance-gpu-memory", "karpenter.k8s.aws/instance-accelerator-name", "karpenter.k8s.aws/instance-accelerator-manufacturer", "karpenter.k8s.aws/instance-accelerator-count"] || !self.find("^([^/]+)").endsWith("karpenter.k8s.aws") + operator: + description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + enum: + - In + - NotIn + - Exists + - DoesNotExist + - Gt + - Lt + values: + description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. + items: + type: string + type: array + maxLength: 63 + pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$ + required: + - key + - operator + type: object + maxItems: 30 + type: array + x-kubernetes-validations: + - message: requirements with operator 'In' must have a value defined + rule: "self.all(x, x.operator == 'In' ? x.values.size() != 0 : true)" + - message: requirements operator 'Gt' or 'Lt' must have a single positive integer value + rule: "self.all(x, (x.operator == 'Gt' || x.operator == 'Lt') ? (x.values.size() == 1 && int(x.values[0]) >= 0) : true)" + resources: + description: Resources models the resource requirements for the NodeClaim to launch + properties: + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Requests describes the minimum required resources for the NodeClaim to launch + type: object + type: object + startupTaints: + description: StartupTaints are taints that are applied to nodes upon startup which are expected to be removed automatically within a short period of time, typically by a DaemonSet that tolerates the taint. These are commonly used by daemonsets to allow initialization and enforce startup ordering. StartupTaints are ignored for provisioning purposes in that pods are not required to tolerate a StartupTaint in order to have nodes provisioned for them. + items: + description: The node this Taint is attached to has the "effect" on any pod that does not tolerate the Taint. + properties: + effect: + description: Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. + type: string + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + key: + description: Required. The taint key to be applied to a node. + type: string + minLength: 1 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + timeAdded: + description: TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints. + format: date-time + type: string + value: + description: The taint value corresponding to the taint key. + type: string + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + required: + - effect + - key + type: object + type: array + taints: + description: Taints will be applied to the NodeClaim's node. + items: + description: The node this Taint is attached to has the "effect" on any pod that does not tolerate the Taint. + properties: + effect: + description: Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. + type: string + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + key: + description: Required. The taint key to be applied to a node. + type: string + minLength: 1 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + timeAdded: + description: TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints. + format: date-time + type: string + value: + description: The taint value corresponding to the taint key. + type: string + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + required: + - effect + - key + type: object + type: array + required: + - nodeClassRef + - requirements + type: object + status: + description: NodeClaimStatus defines the observed state of NodeClaim + properties: + allocatable: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Allocatable is the estimated allocatable capacity of the node + type: object + capacity: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Capacity is the estimated full capacity of the node + type: object + conditions: + description: Conditions contains signals for health and readiness + items: + description: "Condition defines a readiness condition for a Knative resource. See: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#typical-status-properties" + properties: + lastTransitionTime: + description: LastTransitionTime is the last time the condition transitioned from one status to another. We use VolatileTime in place of metav1.Time to exclude this from creating equality.Semantic differences (all other things held constant). + type: string + message: + description: A human readable message indicating details about the transition. + type: string + reason: + description: The reason for the condition's last transition. + type: string + severity: + description: Severity with which to treat failures of this type of condition. When this is not specified, it defaults to Error. + type: string + status: + description: Status of the condition, one of True, False, Unknown. + type: string + type: + description: Type of condition. + type: string + required: + - status + - type + type: object + type: array + imageID: + description: ImageID is an identifier for the image that runs on the node + type: string + nodeName: + description: NodeName is the name of the corresponding node object + type: string + providerID: + description: ProviderID of the corresponding node object + type: string + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/charts/karpenter/pkg/apis/crds/karpenter.sh_nodepools.yaml b/charts/karpenter/pkg/apis/crds/karpenter.sh_nodepools.yaml new file mode 100644 index 0000000..9276b75 --- /dev/null +++ b/charts/karpenter/pkg/apis/crds/karpenter.sh_nodepools.yaml @@ -0,0 +1,386 @@ +--- +apiVersion: apiextensions.k8s.io/v1 +kind: CustomResourceDefinition +metadata: + annotations: + controller-gen.kubebuilder.io/version: v0.13.0 + name: nodepools.karpenter.sh +spec: + group: karpenter.sh + names: + categories: + - karpenter + kind: NodePool + listKind: NodePoolList + plural: nodepools + singular: nodepool + scope: Cluster + versions: + - additionalPrinterColumns: + - jsonPath: .spec.template.spec.nodeClassRef.name + name: NodeClass + type: string + - jsonPath: .spec.weight + name: Weight + priority: 1 + type: string + name: v1beta1 + schema: + openAPIV3Schema: + description: NodePool is the Schema for the NodePools API + properties: + apiVersion: + description: "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources" + type: string + kind: + description: "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds" + type: string + metadata: + type: object + spec: + description: NodePoolSpec is the top level provisioner specification. Provisioners launch nodes in response to pods that are unschedulable. A single provisioner is capable of managing a diverse set of nodes. Node properties are determined from a combination of provisioner and pod scheduling constraints. + properties: + disruption: + default: + consolidationPolicy: WhenUnderutilized + expireAfter: 720h + description: Disruption contains the parameters that relate to Karpenter's disruption logic + properties: + consolidateAfter: + description: ConsolidateAfter is the duration the controller will wait before attempting to terminate nodes that are underutilized. Refer to ConsolidationPolicy for how underutilization is considered. + pattern: ^(([0-9]+(s|m|h))+)|(Never)$ + type: string + consolidationPolicy: + default: WhenUnderutilized + description: ConsolidationPolicy describes which nodes Karpenter can disrupt through its consolidation algorithm. This policy defaults to "WhenUnderutilized" if not specified + enum: + - WhenEmpty + - WhenUnderutilized + type: string + expireAfter: + default: 720h + description: ExpireAfter is the duration the controller will wait before terminating a node, measured from when the node is created. This is useful to implement features like eventually consistent node upgrade, memory leak protection, and disruption testing. + pattern: ^(([0-9]+(s|m|h))+)|(Never)$ + type: string + type: object + x-kubernetes-validations: + - message: consolidateAfter cannot be combined with consolidationPolicy=WhenUnderutilized + rule: "has(self.consolidateAfter) ? self.consolidationPolicy != 'WhenUnderutilized' || self.consolidateAfter == 'Never' : true" + - message: consolidateAfter must be specified with consolidationPolicy=WhenEmpty + rule: "self.consolidationPolicy == 'WhenEmpty' ? has(self.consolidateAfter) : true" + limits: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Limits define a set of bounds for provisioning capacity. + type: object + template: + description: Template contains the template of possibilities for the provisioning logic to launch a NodeClaim with. NodeClaims launched from this NodePool will often be further constrained than the template specifies. + properties: + metadata: + properties: + annotations: + additionalProperties: + type: string + description: "Annotations is an unstructured key value map stored with a resource that may be set by external tools to store and retrieve arbitrary metadata. They are not queryable and should be preserved when modifying objects. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations" + type: object + labels: + additionalProperties: + type: string + maxLength: 63 + pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$ + description: "Map of string keys and values that can be used to organize and categorize (scope and select) objects. May match selectors of replication controllers and services. More info: https://kubernetes.io/docs/concepts/overview/working-with-objects/labels" + type: object + maxProperties: 100 + x-kubernetes-validations: + - message: label domain "kubernetes.io" is restricted + rule: self.all(x, x in ["beta.kubernetes.io/instance-type", "failure-domain.beta.kubernetes.io/region", "beta.kubernetes.io/os", "beta.kubernetes.io/arch", "failure-domain.beta.kubernetes.io/zone", "topology.kubernetes.io/zone", "topology.kubernetes.io/region", "kubernetes.io/arch", "kubernetes.io/os", "node.kubernetes.io/windows-build"] || x.find("^([^/]+)").endsWith("node.kubernetes.io") || x.find("^([^/]+)").endsWith("node-restriction.kubernetes.io") || !x.find("^([^/]+)").endsWith("kubernetes.io")) + - message: label domain "k8s.io" is restricted + rule: self.all(x, x.find("^([^/]+)").endsWith("kops.k8s.io") || !x.find("^([^/]+)").endsWith("k8s.io")) + - message: label domain "karpenter.sh" is restricted + rule: self.all(x, x in ["karpenter.sh/capacity-type", "karpenter.sh/nodepool"] || !x.find("^([^/]+)").endsWith("karpenter.sh")) + - message: label "karpenter.sh/nodepool" is restricted + rule: self.all(x, x != "karpenter.sh/nodepool") + - message: label "kubernetes.io/hostname" is restricted + rule: self.all(x, x != "kubernetes.io/hostname") + - message: label domain "karpenter.k8s.aws" is restricted + rule: self.all(x, x in ["karpenter.k8s.aws/instance-encryption-in-transit-supported", "karpenter.k8s.aws/instance-category", "karpenter.k8s.aws/instance-hypervisor", "karpenter.k8s.aws/instance-family", "karpenter.k8s.aws/instance-generation", "karpenter.k8s.aws/instance-local-nvme", "karpenter.k8s.aws/instance-size", "karpenter.k8s.aws/instance-cpu","karpenter.k8s.aws/instance-memory", "karpenter.k8s.aws/instance-network-bandwidth", "karpenter.k8s.aws/instance-gpu-name", "karpenter.k8s.aws/instance-gpu-manufacturer", "karpenter.k8s.aws/instance-gpu-count", "karpenter.k8s.aws/instance-gpu-memory", "karpenter.k8s.aws/instance-accelerator-name", "karpenter.k8s.aws/instance-accelerator-manufacturer", "karpenter.k8s.aws/instance-accelerator-count"] || !x.find("^([^/]+)").endsWith("karpenter.k8s.aws")) + type: object + spec: + description: NodeClaimSpec describes the desired state of the NodeClaim + properties: + kubelet: + description: Kubelet defines args to be used when configuring kubelet on provisioned nodes. They are a subset of the upstream types, recognizing not all options may be supported. Wherever possible, the types and names should reflect the upstream kubelet types. + properties: + clusterDNS: + description: clusterDNS is a list of IP addresses for the cluster DNS server. Note that not all providers may use all addresses. + items: + type: string + type: array + cpuCFSQuota: + description: CPUCFSQuota enables CPU CFS quota enforcement for containers that specify CPU limits. + type: boolean + evictionHard: + additionalProperties: + type: string + pattern: ^((\d{1,2}(\.\d{1,2})?|100(\.0{1,2})?)%||(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?)$ + description: EvictionHard is the map of signal names to quantities that define hard eviction thresholds + type: object + x-kubernetes-validations: + - message: valid keys for evictionHard are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'] + rule: self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']) + evictionMaxPodGracePeriod: + description: EvictionMaxPodGracePeriod is the maximum allowed grace period (in seconds) to use when terminating pods in response to soft eviction thresholds being met. + format: int32 + type: integer + evictionSoft: + additionalProperties: + type: string + pattern: ^((\d{1,2}(\.\d{1,2})?|100(\.0{1,2})?)%||(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?)$ + description: EvictionSoft is the map of signal names to quantities that define soft eviction thresholds + type: object + x-kubernetes-validations: + - message: valid keys for evictionSoft are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'] + rule: self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']) + evictionSoftGracePeriod: + additionalProperties: + type: string + description: EvictionSoftGracePeriod is the map of signal names to quantities that define grace periods for each eviction signal + type: object + x-kubernetes-validations: + - message: valid keys for evictionSoftGracePeriod are ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available'] + rule: self.all(x, x in ['memory.available','nodefs.available','nodefs.inodesFree','imagefs.available','imagefs.inodesFree','pid.available']) + imageGCHighThresholdPercent: + description: ImageGCHighThresholdPercent is the percent of disk usage after which image garbage collection is always run. The percent is calculated by dividing this field value by 100, so this field must be between 0 and 100, inclusive. When specified, the value must be greater than ImageGCLowThresholdPercent. + format: int32 + maximum: 100 + minimum: 0 + type: integer + imageGCLowThresholdPercent: + description: ImageGCLowThresholdPercent is the percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. The percent is calculated by dividing this field value by 100, so the field value must be between 0 and 100, inclusive. When specified, the value must be less than imageGCHighThresholdPercent + format: int32 + maximum: 100 + minimum: 0 + type: integer + kubeReserved: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: KubeReserved contains resources reserved for Kubernetes system components. + type: object + x-kubernetes-validations: + - message: valid keys for kubeReserved are ['cpu','memory','ephemeral-storage','pid'] + rule: self.all(x, x=='cpu' || x=='memory' || x=='ephemeral-storage' || x=='pid') + - message: kubeReserved value cannot be a negative resource quantity + rule: self.all(x, !self[x].startsWith('-')) + maxPods: + description: MaxPods is an override for the maximum number of pods that can run on a worker node instance. + format: int32 + minimum: 0 + type: integer + podsPerCore: + description: PodsPerCore is an override for the number of pods that can run on a worker node instance based on the number of cpu cores. This value cannot exceed MaxPods, so, if MaxPods is a lower value, that value will be used. + format: int32 + minimum: 0 + type: integer + systemReserved: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: SystemReserved contains resources reserved for OS system daemons and kernel memory. + type: object + x-kubernetes-validations: + - message: valid keys for systemReserved are ['cpu','memory','ephemeral-storage','pid'] + rule: self.all(x, x=='cpu' || x=='memory' || x=='ephemeral-storage' || x=='pid') + - message: systemReserved value cannot be a negative resource quantity + rule: self.all(x, !self[x].startsWith('-')) + type: object + x-kubernetes-validations: + - message: imageGCHighThresholdPercent must be greater than imageGCLowThresholdPercent + rule: "has(self.imageGCHighThresholdPercent) && has(self.imageGCLowThresholdPercent) ? self.imageGCHighThresholdPercent > self.imageGCLowThresholdPercent : true" + - message: evictionSoft OwnerKey does not have a matching evictionSoftGracePeriod + rule: has(self.evictionSoft) ? self.evictionSoft.all(e, (e in self.evictionSoftGracePeriod)):true + - message: evictionSoftGracePeriod OwnerKey does not have a matching evictionSoft + rule: has(self.evictionSoftGracePeriod) ? self.evictionSoftGracePeriod.all(e, (e in self.evictionSoft)):true + nodeClassRef: + description: NodeClassRef is a reference to an object that defines provider specific configuration + properties: + apiVersion: + description: API version of the referent + type: string + kind: + description: 'Kind of the referent; More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds"' + type: string + name: + description: "Name of the referent; More info: http://kubernetes.io/docs/user-guide/identifiers#names" + type: string + required: + - name + type: object + requirements: + description: Requirements are layered with GetLabels and applied to every node. + items: + description: A node selector requirement is a selector that contains values, a key, and an operator that relates the key and values. + properties: + key: + description: The label key that the selector applies to. + type: string + maxLength: 316 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + x-kubernetes-validations: + - message: label domain "kubernetes.io" is restricted + rule: self in ["beta.kubernetes.io/instance-type", "failure-domain.beta.kubernetes.io/region", "beta.kubernetes.io/os", "beta.kubernetes.io/arch", "failure-domain.beta.kubernetes.io/zone", "topology.kubernetes.io/zone", "topology.kubernetes.io/region", "node.kubernetes.io/instance-type", "kubernetes.io/arch", "kubernetes.io/os", "node.kubernetes.io/windows-build"] || self.find("^([^/]+)").endsWith("node.kubernetes.io") || self.find("^([^/]+)").endsWith("node-restriction.kubernetes.io") || !self.find("^([^/]+)").endsWith("kubernetes.io") + - message: label domain "k8s.io" is restricted + rule: self.find("^([^/]+)").endsWith("kops.k8s.io") || !self.find("^([^/]+)").endsWith("k8s.io") + - message: label domain "karpenter.sh" is restricted + rule: self in ["karpenter.sh/capacity-type", "karpenter.sh/nodepool"] || !self.find("^([^/]+)").endsWith("karpenter.sh") + - message: label "karpenter.sh/nodepool" is restricted + rule: self != "karpenter.sh/nodepool" + - message: label "kubernetes.io/hostname" is restricted + rule: self != "kubernetes.io/hostname" + - message: label domain "karpenter.k8s.aws" is restricted + rule: self in ["karpenter.k8s.aws/instance-encryption-in-transit-supported", "karpenter.k8s.aws/instance-category", "karpenter.k8s.aws/instance-hypervisor", "karpenter.k8s.aws/instance-family", "karpenter.k8s.aws/instance-generation", "karpenter.k8s.aws/instance-local-nvme", "karpenter.k8s.aws/instance-size", "karpenter.k8s.aws/instance-cpu","karpenter.k8s.aws/instance-memory", "karpenter.k8s.aws/instance-network-bandwidth", "karpenter.k8s.aws/instance-gpu-name", "karpenter.k8s.aws/instance-gpu-manufacturer", "karpenter.k8s.aws/instance-gpu-count", "karpenter.k8s.aws/instance-gpu-memory", "karpenter.k8s.aws/instance-accelerator-name", "karpenter.k8s.aws/instance-accelerator-manufacturer", "karpenter.k8s.aws/instance-accelerator-count"] || !self.find("^([^/]+)").endsWith("karpenter.k8s.aws") + operator: + description: Represents a key's relationship to a set of values. Valid operators are In, NotIn, Exists, DoesNotExist. Gt, and Lt. + type: string + enum: + - In + - NotIn + - Exists + - DoesNotExist + - Gt + - Lt + values: + description: An array of string values. If the operator is In or NotIn, the values array must be non-empty. If the operator is Exists or DoesNotExist, the values array must be empty. If the operator is Gt or Lt, the values array must have a single element, which will be interpreted as an integer. This array is replaced during a strategic merge patch. + items: + type: string + type: array + maxLength: 63 + pattern: ^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$ + required: + - key + - operator + type: object + maxItems: 30 + type: array + x-kubernetes-validations: + - message: requirements with operator 'In' must have a value defined + rule: "self.all(x, x.operator == 'In' ? x.values.size() != 0 : true)" + - message: requirements operator 'Gt' or 'Lt' must have a single positive integer value + rule: "self.all(x, (x.operator == 'Gt' || x.operator == 'Lt') ? (x.values.size() == 1 && int(x.values[0]) >= 0) : true)" + resources: + description: Resources models the resource requirements for the NodeClaim to launch + properties: + requests: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Requests describes the minimum required resources for the NodeClaim to launch + type: object + type: object + startupTaints: + description: StartupTaints are taints that are applied to nodes upon startup which are expected to be removed automatically within a short period of time, typically by a DaemonSet that tolerates the taint. These are commonly used by daemonsets to allow initialization and enforce startup ordering. StartupTaints are ignored for provisioning purposes in that pods are not required to tolerate a StartupTaint in order to have nodes provisioned for them. + items: + description: The node this Taint is attached to has the "effect" on any pod that does not tolerate the Taint. + properties: + effect: + description: Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. + type: string + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + key: + description: Required. The taint key to be applied to a node. + type: string + minLength: 1 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + timeAdded: + description: TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints. + format: date-time + type: string + value: + description: The taint value corresponding to the taint key. + type: string + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + required: + - effect + - key + type: object + type: array + taints: + description: Taints will be applied to the NodeClaim's node. + items: + description: The node this Taint is attached to has the "effect" on any pod that does not tolerate the Taint. + properties: + effect: + description: Required. The effect of the taint on pods that do not tolerate the taint. Valid effects are NoSchedule, PreferNoSchedule and NoExecute. + type: string + enum: + - NoSchedule + - PreferNoSchedule + - NoExecute + key: + description: Required. The taint key to be applied to a node. + type: string + minLength: 1 + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + timeAdded: + description: TimeAdded represents the time at which the taint was added. It is only written for NoExecute taints. + format: date-time + type: string + value: + description: The taint value corresponding to the taint key. + type: string + pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*(\/))?([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]$ + required: + - effect + - key + type: object + type: array + required: + - nodeClassRef + - requirements + type: object + required: + - spec + type: object + weight: + description: Weight is the priority given to the provisioner during scheduling. A higher numerical weight indicates that this provisioner will be ordered ahead of other provisioners with lower weights. A provisioner with no weight will be treated as if it is a provisioner with a weight of 0. + format: int32 + maximum: 100 + minimum: 1 + type: integer + required: + - template + type: object + status: + description: NodePoolStatus defines the observed state of NodePool + properties: + resources: + additionalProperties: + anyOf: + - type: integer + - type: string + pattern: ^(\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))(([KMGTPE]i)|[numkMGTPE]|([eE](\+|-)?(([0-9]+(\.[0-9]*)?)|(\.[0-9]+))))?$ + x-kubernetes-int-or-string: true + description: Resources is the list of resources that have been provisioned. + type: object + type: object + type: object + served: true + storage: true + subresources: + status: {} diff --git a/charts/karpenter/templates/_helpers.tpl b/charts/karpenter/templates/_helpers.tpl index d78659a..34af864 100644 --- a/charts/karpenter/templates/_helpers.tpl +++ b/charts/karpenter/templates/_helpers.tpl @@ -141,40 +141,20 @@ This works because Helm treats dictionaries as mutable objects and allows passin {{- end }} {{- end }} -{{/* -Flatten Settings Map using "." syntax -*/}} -{{- define "flattenSettings" -}} -{{- $map := first . -}} -{{- $label := last . -}} -{{- range $key := (keys $map | uniq | sortAlpha) }} - {{- $sublabel := $key -}} - {{- $val := (get $map $key) -}} - {{- if $label -}} - {{- $sublabel = list $label $key | join "." -}} - {{- end -}} - {{/* Special-case "tags" since we want this to be a JSON object */}} - {{- if eq $key "tags" -}} - {{- if not (kindIs "invalid" $val) -}} - {{- $sublabel | quote | nindent 2 }}: {{ $val | toJson | quote }} - {{- end -}} - {{- else if kindOf $val | eq "map" -}} - {{- list $val $sublabel | include "flattenSettings" -}} - {{- else -}} - {{- if not (kindIs "invalid" $val) -}} - {{- $sublabel | quote | nindent 2 -}}: {{ $val | quote }} - {{- end -}} -{{- end -}} -{{- end -}} -{{- end -}} - {{/* Flatten the stdout logging outputs from args provided */}} -{{- define "karpenter.controller.outputPathsList" -}} +{{- define "karpenter.outputPathsList" -}} {{ $paths := list -}} {{- range .Values.controller.outputPaths -}} - {{- $paths = printf "%s" . | quote | append $paths -}} + {{- if not (has (printf "%s" . | quote) $paths) -}} + {{- $paths = printf "%s" . | quote | append $paths -}} + {{- end -}} +{{- end -}} +{{- range .Values.logConfig.outputPaths -}} + {{- if not (has (printf "%s" . | quote) $paths) -}} + {{- $paths = printf "%s" . | quote | append $paths -}} + {{- end -}} {{- end -}} {{ $paths | join ", " }} {{- end -}} @@ -182,10 +162,17 @@ Flatten the stdout logging outputs from args provided {{/* Flatten the stderr logging outputs from args provided */}} -{{- define "karpenter.controller.errorOutputPathsList" -}} +{{- define "karpenter.errorOutputPathsList" -}} {{ $paths := list -}} {{- range .Values.controller.errorOutputPaths -}} - {{- $paths = printf "%s" . | quote | append $paths -}} + {{- if not (has (printf "%s" . | quote) $paths) -}} + {{- $paths = printf "%s" . | quote | append $paths -}} + {{- end -}} +{{- end -}} +{{- range .Values.logConfig.errorOutputPaths -}} + {{- if not (has (printf "%s" . | quote) $paths) -}} + {{- $paths = printf "%s" . | quote | append $paths -}} + {{- end -}} {{- end -}} {{ $paths | join ", " }} {{- end -}} \ No newline at end of file diff --git a/charts/karpenter/templates/aggregate-clusterrole.yaml b/charts/karpenter/templates/aggregate-clusterrole.yaml index 155ccce..b46cfd8 100644 --- a/charts/karpenter/templates/aggregate-clusterrole.yaml +++ b/charts/karpenter/templates/aggregate-clusterrole.yaml @@ -13,6 +13,12 @@ rules: - apiGroups: ["karpenter.sh"] resources: ["provisioners", "provisioners/status", "machines", "machines/status"] verbs: ["get", "list", "watch", "create", "delete", "patch"] + - apiGroups: ["karpenter.sh"] + resources: ["nodepools", "nodepools/status", "nodeclaims", "nodeclaims/status"] + verbs: ["get", "list", "watch", "create", "delete", "patch"] - apiGroups: ["karpenter.k8s.aws"] resources: ["awsnodetemplates"] verbs: ["get", "list", "watch", "create", "delete", "patch"] + - apiGroups: ["karpenter.k8s.aws"] + resources: ["ec2nodeclasses"] + verbs: ["get", "list", "watch", "create", "delete", "patch"] diff --git a/charts/karpenter/templates/clusterrole-core.yaml b/charts/karpenter/templates/clusterrole-core.yaml index d090527..45d9ccc 100644 --- a/charts/karpenter/templates/clusterrole-core.yaml +++ b/charts/karpenter/templates/clusterrole-core.yaml @@ -32,6 +32,9 @@ rules: - apiGroups: ["karpenter.sh"] resources: ["provisioners", "provisioners/status", "machines", "machines/status"] verbs: ["get", "list", "watch"] + - apiGroups: ["karpenter.sh"] + resources: ["nodepools", "nodepools/status", "nodeclaims", "nodeclaims/status"] + verbs: ["get", "list", "watch"] - apiGroups: [""] resources: ["pods", "nodes", "persistentvolumes", "persistentvolumeclaims", "replicationcontrollers", "namespaces"] verbs: ["get", "list", "watch"] @@ -41,16 +44,27 @@ rules: - apiGroups: ["apps"] resources: ["daemonsets", "deployments", "replicasets", "statefulsets"] verbs: ["list", "watch"] +{{- if .Values.webhook.enabled }} - apiGroups: ["admissionregistration.k8s.io"] resources: ["validatingwebhookconfigurations", "mutatingwebhookconfigurations"] verbs: ["get", "watch", "list"] - - apiGroups: [ "policy" ] - resources: [ "poddisruptionbudgets" ] - verbs: [ "get", "list", "watch" ] +{{- end }} + - apiGroups: ["policy"] + resources: ["poddisruptionbudgets"] + verbs: ["get", "list", "watch"] # Write - apiGroups: ["karpenter.sh"] - resources: ["provisioners/status", "machines", "machines/status"] - verbs: ["create", "delete", "patch"] + resources: ["machines", "machines/status"] + verbs: ["create", "delete", "update", "patch"] + - apiGroups: ["karpenter.sh"] + resources: ["provisioners", "provisioners/status"] + verbs: ["update", "patch"] + - apiGroups: ["karpenter.sh"] + resources: ["nodeclaims", "nodeclaims/status"] + verbs: ["create", "delete", "update", "patch"] + - apiGroups: ["karpenter.sh"] + resources: ["nodepools", "nodepools/status"] + verbs: ["update", "patch"] - apiGroups: [""] resources: ["events"] verbs: ["create", "patch"] @@ -60,10 +74,13 @@ rules: - apiGroups: [""] resources: ["pods/eviction"] verbs: ["create"] +{{- if .Values.webhook.enabled }} - apiGroups: ["admissionregistration.k8s.io"] resources: ["validatingwebhookconfigurations"] verbs: ["update"] resourceNames: ["validation.webhook.karpenter.sh", "validation.webhook.config.karpenter.sh"] +{{- end }} {{- with .Values.additionalClusterRoleRules -}} {{ toYaml . | nindent 2 }} - {{- end -}} \ No newline at end of file + {{- end -}} + \ No newline at end of file diff --git a/charts/karpenter/templates/clusterrole.yaml b/charts/karpenter/templates/clusterrole.yaml index cf32e62..8580931 100644 --- a/charts/karpenter/templates/clusterrole.yaml +++ b/charts/karpenter/templates/clusterrole.yaml @@ -30,8 +30,13 @@ metadata: rules: # Read - apiGroups: ["karpenter.k8s.aws"] - resources: ["awsnodetemplates"] + resources: ["awsnodetemplates", "ec2nodeclasses"] verbs: ["get", "list", "watch"] + # Write + - apiGroups: ["karpenter.k8s.aws"] + resources: ["awsnodetemplates", "awsnodetemplates/status", "ec2nodeclasses", "ec2nodeclasses/status"] + verbs: ["patch", "update"] +{{- if .Values.webhook.enabled }} - apiGroups: ["admissionregistration.k8s.io"] resources: ["validatingwebhookconfigurations"] verbs: ["update"] @@ -40,7 +45,4 @@ rules: resources: ["mutatingwebhookconfigurations"] verbs: ["update"] resourceNames: ["defaulting.webhook.karpenter.k8s.aws"] - # Write - - apiGroups: ["karpenter.k8s.aws"] - resources: ["awsnodetemplates/status"] - verbs: ["patch", "update"] \ No newline at end of file +{{- end }} diff --git a/charts/karpenter/templates/configmap-logging.yaml b/charts/karpenter/templates/configmap-logging.yaml index b2d34c6..0de88c1 100644 --- a/charts/karpenter/templates/configmap-logging.yaml +++ b/charts/karpenter/templates/configmap-logging.yaml @@ -1,3 +1,4 @@ +{{- if .Values.logConfig.enabled }} apiVersion: v1 kind: ConfigMap metadata: @@ -13,7 +14,7 @@ data: # https://github.com/uber-go/zap/blob/aa3e73ec0896f8b066ddf668597a02f89628ee50/config.go zap-logger-config: | { - "level": "{{ .Values.logLevel }}", + "level": "{{ .Values.logConfig.logLevel.global }}", "development": false, "disableStacktrace": true, "disableCaller": true, @@ -21,9 +22,9 @@ data: "initial": 100, "thereafter": 100 }, - "outputPaths": [{{ include "karpenter.controller.outputPathsList" . }}], - "errorOutputPaths": [{{ include "karpenter.controller.errorOutputPathsList" . }}], - "encoding": "{{ .Values.logEncoding }}", + "outputPaths": [{{ include "karpenter.outputPathsList" . }}], + "errorOutputPaths": [{{ include "karpenter.errorOutputPathsList" . }}], + "encoding": "{{ or .Values.logEncoding .Values.logConfig.logEncoding }}", "encoderConfig": { "timeKey": "time", "levelKey": "level", @@ -35,9 +36,6 @@ data: "timeEncoder": "iso8601" } } -{{- with .Values.controller.logLevel }} - loglevel.controller: {{ . | quote }} + loglevel.controller: {{ or .Values.controller.logLevel .Values.logConfig.logLevel.controller }} + loglevel.webhook: {{ or .Values.webhook.logLevel .Values.logConfig.logLevel.webhook }} {{- end }} -{{- with .Values.webhook.logLevel }} - loglevel.webhook: {{ . | quote }} -{{- end }} \ No newline at end of file diff --git a/charts/karpenter/templates/configmap.yaml b/charts/karpenter/templates/configmap.yaml index 1d31257..362673e 100644 --- a/charts/karpenter/templates/configmap.yaml +++ b/charts/karpenter/templates/configmap.yaml @@ -10,4 +10,51 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} data: - {{- list .Values.settings "" | include "flattenSettings" | indent 2 }} \ No newline at end of file +{{- with .Values.settings.batchMaxDuration }} + batchMaxDuration: "{{ . }}" +{{- end }} +{{- with .Values.settings.batchIdleDuration }} + batchIdleDuration: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.assumeRoleARN }} + aws.assumeRoleARN: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.assumeRoleDuration }} + aws.assumeRoleDuration: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.clusterCABundle }} + aws.clusterCABundle: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.clusterName }} + aws.clusterName: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.clusterEndpoint }} + aws.clusterEndpoint: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.defaultInstanceProfile }} + aws.defaultInstanceProfile: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.enablePodENI }} + aws.enablePodENI: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.enableENILimitedPodDensity }} + aws.enableENILimitedPodDensity: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.isolatedVPC }} + aws.isolatedVPC: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.vmMemoryOverheadPercent }} + aws.vmMemoryOverheadPercent: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.interruptionQueueName }} + aws.interruptionQueueName: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.tags }} + aws.tags: "{{ . }}" +{{- end }} +{{- with .Values.settings.aws.reservedENIs }} + aws.reservedENIs: "{{ . }}" +{{- end }} +{{- with .Values.settings.featureGates.driftEnabled }} + featureGates.driftEnabled: "{{ . }}" +{{- end }} diff --git a/charts/karpenter/templates/deployment.yaml b/charts/karpenter/templates/deployment.yaml index f8bd3c8..9cc2026 100644 --- a/charts/karpenter/templates/deployment.yaml +++ b/charts/karpenter/templates/deployment.yaml @@ -37,10 +37,6 @@ spec: {{- toYaml . | nindent 8 }} {{- end }} serviceAccountName: {{ include "karpenter.serviceAccountName" . }} - {{- with .Values.podSecurityContext }} - securityContext: - {{- toYaml . | nindent 8 }} - {{- end }} {{- with .Values.priorityClassName }} priorityClassName: {{ . | quote }} {{- end }} @@ -59,10 +55,17 @@ spec: {{- end }} containers: - name: controller - {{- with .Values.controller.securityContext }} securityContext: - {{- toYaml . | nindent 12 }} - {{- end }} + runAsUser: 65536 + runAsGroup: 65536 + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + readOnlyRootFilesystem: true image: {{ include "karpenter.controller.image" . }} imagePullPolicy: {{ .Values.imagePullPolicy }} env: @@ -70,8 +73,19 @@ spec: value: "1.19.0-0" - name: KARPENTER_SERVICE value: {{ include "karpenter.fullname" . }} + {{- if .Values.webhook.enabled }} - name: WEBHOOK_PORT value: "{{ .Values.webhook.port }}" + - name: WEBHOOK_METRICS_PORT + value: "{{ .Values.webhook.metrics.port }}" + {{- else }} + - name: DISABLE_WEBHOOK + value: "true" + {{- end }} + {{- with .Values.logLevel }} + - name: LOG_LEVEL + value: "{{ . }}" + {{- end }} - name: METRICS_PORT value: "{{ .Values.controller.metrics.port }}" - name: HEALTH_PROBE_PORT @@ -86,6 +100,52 @@ spec: containerName: controller divisor: "0" resource: limits.memory + - name: FEATURE_GATES + value: "Drift={{ or .Values.settings.featureGates.driftEnabled .Values.settings.featureGates.drift }}" + {{- with .Values.settings.batchMaxDuration }} + - name: BATCH_MAX_DURATION + value: "{{ . }}" + {{- end }} + {{- with .Values.settings.batchIdleDuration }} + - name: BATCH_IDLE_DURATION + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.assumeRoleARN .Values.settings.assumeRoleARN }} + - name: ASSUME_ROLE_ARN + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.assumeRoleDuration .Values.settings.assumeRoleDuration }} + - name: ASSUME_ROLE_DURATION + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.clusterCABundle .Values.settings.clusterCABundle }} + - name: CLUSTER_CA_BUNDLE + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.clusterName .Values.settings.clusterName }} + - name: CLUSTER_NAME + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.clusterEndpoint .Values.settings.clusterEndpoint }} + - name: CLUSTER_ENDPOINT + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.isolatedVPC .Values.settings.isolatedVPC }} + - name: ISOLATED_VPC + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.vmMemoryOverheadPercent .Values.settings.vmMemoryOverheadPercent }} + - name: VM_MEMORY_OVERHEAD_PERCENT + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.interruptionQueueName .Values.settings.interruptionQueue }} + - name: INTERRUPTION_QUEUE + value: "{{ . }}" + {{- end }} + {{- with or .Values.settings.aws.reservedENIs .Values.settings.reservedENIs }} + - name: RESERVED_ENIS + value: "{{ . }}" + {{- end }} {{- with .Values.controller.env }} {{- toYaml . | nindent 12 }} {{- end }} @@ -97,12 +157,17 @@ spec: - name: http-metrics containerPort: {{ .Values.controller.metrics.port }} protocol: TCP - - name: http - containerPort: {{ .Values.controller.healthProbe.port }} + {{- if .Values.webhook.enabled }} + - name: webhook-metrics + containerPort: {{ .Values.webhook.metrics.port }} protocol: TCP - name: https-webhook containerPort: {{ .Values.webhook.port }} protocol: TCP + {{- end }} + - name: http + containerPort: {{ .Values.controller.healthProbe.port }} + protocol: TCP livenessProbe: initialDelaySeconds: 30 timeoutSeconds: 30 @@ -110,6 +175,7 @@ spec: path: /healthz port: http readinessProbe: + initialDelaySeconds: 5 timeoutSeconds: 30 httpGet: path: /readyz @@ -118,22 +184,28 @@ spec: resources: {{- toYaml . | nindent 12 }} {{- end }} - {{- with .Values.controller.extraVolumeMounts }} + {{- if or (.Values.logConfig.enabled) (.Values.controller.extraVolumeMounts) }} volumeMounts: + {{- if .Values.logConfig.enabled }} + - name: config-logging + mountPath: /etc/karpenter/logging + {{- end }} + {{- with .Values.controller.extraVolumeMounts }} {{- toYaml . | nindent 12 }} {{- end }} + {{- end }} {{- with .Values.controller.sidecarContainer }} - {{- toYaml . | nindent 8 }} + {{- toYaml . | nindent 8 }} {{- end }} {{- if and (.Values.controller.sidecarContainer) (or .Values.controller.extraVolumeMounts .Values.controller.sidecarVolumeMounts) }} volumeMounts: {{- with .Values.controller.extraVolumeMounts }} - {{- toYaml . | nindent 12 }} + {{- toYaml . | nindent 12 }} {{- end }} {{- with .Values.controller.sidecarVolumeMounts }} - {{- toYaml . | nindent 12 }} + {{- toYaml . | nindent 12 }} {{- end }} - {{- end }} + {{- end }} {{- with .Values.nodeSelector }} nodeSelector: {{- toYaml . | nindent 8 }} @@ -154,7 +226,15 @@ spec: tolerations: {{- toYaml . | nindent 8 }} {{- end }} - {{- with .Values.extraVolumes }} + {{- if or (.Values.logConfig.enabled) (.Values.extraVolumes) }} volumes: + {{- if .Values.logConfig.enabled }} + - name: config-logging + configMap: + name: config-logging + {{- end }} + {{- with .Values.extraVolumes }} {{- toYaml . | nindent 8 }} - {{- end }} \ No newline at end of file + {{- end }} + {{- end }} + \ No newline at end of file diff --git a/charts/karpenter/templates/poddisruptionbudget.yaml b/charts/karpenter/templates/poddisruptionbudget.yaml index 528b907..4827f38 100644 --- a/charts/karpenter/templates/poddisruptionbudget.yaml +++ b/charts/karpenter/templates/poddisruptionbudget.yaml @@ -13,4 +13,5 @@ spec: maxUnavailable: {{ .Values.podDisruptionBudget.maxUnavailable }} selector: matchLabels: - {{- include "karpenter.selectorLabels" . | nindent 6 }} \ No newline at end of file + {{- include "karpenter.selectorLabels" . | nindent 6 }} + \ No newline at end of file diff --git a/charts/karpenter/templates/role.yaml b/charts/karpenter/templates/role.yaml index 589f632..926c81d 100644 --- a/charts/karpenter/templates/role.yaml +++ b/charts/karpenter/templates/role.yaml @@ -33,10 +33,16 @@ rules: verbs: ["patch", "update"] resourceNames: - "karpenter-leader-election" +{{- if .Values.webhook.enabled }} + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["patch", "update"] + resourceNames: - "webhook.configmapwebhook.00-of-01" - "webhook.defaultingwebhook.00-of-01" - "webhook.validationwebhook.00-of-01" - "webhook.webhookcertificates.00-of-01" +{{- end }} # Cannot specify resourceNames on create # https://kubernetes.io/docs/reference/access-authn-authz/rbac/#referring-to-resources - apiGroups: ["coordination.k8s.io"] @@ -62,4 +68,26 @@ rules: - apiGroups: [""] resources: ["services"] resourceNames: ["kube-dns"] - verbs: ["get"] \ No newline at end of file + verbs: ["get"] +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: Role +metadata: + name: {{ include "karpenter.fullname" . }}-lease + namespace: kube-node-lease + labels: + {{- include "karpenter.labels" . | nindent 4 }} + {{- with .Values.additionalAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +rules: + # Read + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["get", "list", "watch"] + # Write + - apiGroups: ["coordination.k8s.io"] + resources: ["leases"] + verbs: ["delete"] + \ No newline at end of file diff --git a/charts/karpenter/templates/rolebinding.yaml b/charts/karpenter/templates/rolebinding.yaml index ba54e18..cdffdee 100644 --- a/charts/karpenter/templates/rolebinding.yaml +++ b/charts/karpenter/templates/rolebinding.yaml @@ -36,4 +36,25 @@ roleRef: subjects: - kind: ServiceAccount name: {{ template "karpenter.serviceAccountName" . }} - namespace: {{ .Release.Namespace }} \ No newline at end of file + namespace: {{ .Release.Namespace }} +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: RoleBinding +metadata: + name: {{ include "karpenter.fullname" . }}-lease + namespace: kube-node-lease + labels: + {{- include "karpenter.labels" . | nindent 4 }} + {{- with .Values.additionalAnnotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +roleRef: + apiGroup: rbac.authorization.k8s.io + kind: Role + name: {{ include "karpenter.fullname" . }}-lease +subjects: + - kind: ServiceAccount + name: {{ template "karpenter.serviceAccountName" . }} + namespace: {{ .Release.Namespace }} + \ No newline at end of file diff --git a/charts/karpenter/templates/secret-webhook-cert.yaml b/charts/karpenter/templates/secret-webhook-cert.yaml index 3eb1004..b17309f 100644 --- a/charts/karpenter/templates/secret-webhook-cert.yaml +++ b/charts/karpenter/templates/secret-webhook-cert.yaml @@ -1,3 +1,4 @@ +{{- if .Values.webhook.enabled }} apiVersion: v1 kind: Secret metadata: @@ -9,4 +10,5 @@ metadata: annotations: {{- toYaml . | nindent 4 }} {{- end }} -# data: {} # Injected by karpenter-webhook \ No newline at end of file +# data: {} # Injected by karpenter-webhook +{{- end }} diff --git a/charts/karpenter/templates/service.yaml b/charts/karpenter/templates/service.yaml index 1be4435..91a68a6 100644 --- a/charts/karpenter/templates/service.yaml +++ b/charts/karpenter/templates/service.yaml @@ -16,9 +16,16 @@ spec: port: {{ .Values.controller.metrics.port }} targetPort: http-metrics protocol: TCP + {{- if .Values.webhook.enabled }} + - name: webhook-metrics + port: {{ .Values.webhook.metrics.port }} + targetPort: webhook-metrics + protocol: TCP - name: https-webhook port: {{ .Values.webhook.port }} targetPort: https-webhook protocol: TCP + {{- end }} selector: - {{- include "karpenter.selectorLabels" . | nindent 4 }} \ No newline at end of file + {{- include "karpenter.selectorLabels" . | nindent 4 }} + \ No newline at end of file diff --git a/charts/karpenter/templates/serviceaccount.yaml b/charts/karpenter/templates/serviceaccount.yaml index 2ae6779..0141afc 100644 --- a/charts/karpenter/templates/serviceaccount.yaml +++ b/charts/karpenter/templates/serviceaccount.yaml @@ -15,4 +15,4 @@ metadata: {{- toYaml . | nindent 4 }} {{- end }} {{- end }} -{{- end -}} \ No newline at end of file +{{- end -}} diff --git a/charts/karpenter/templates/servicemonitor.yaml b/charts/karpenter/templates/servicemonitor.yaml index d053cfd..70f0760 100644 --- a/charts/karpenter/templates/servicemonitor.yaml +++ b/charts/karpenter/templates/servicemonitor.yaml @@ -26,4 +26,4 @@ spec: {{- with .Values.serviceMonitor.endpointConfig }} {{- toYaml . | nindent 6 }} {{- end }} -{{- end -}} \ No newline at end of file +{{- end -}} diff --git a/charts/karpenter/templates/webhooks-core.yaml b/charts/karpenter/templates/webhooks-core.yaml index 8d49f8d..391f22c 100644 --- a/charts/karpenter/templates/webhooks-core.yaml +++ b/charts/karpenter/templates/webhooks-core.yaml @@ -1,3 +1,4 @@ +{{- if .Values.webhook.enabled }} apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: @@ -23,12 +24,35 @@ webhooks: - karpenter.sh apiVersions: - v1alpha5 + operations: + - CREATE + - UPDATE resources: - provisioners - provisioners/status + scope: '*' + - apiGroups: + - karpenter.sh + apiVersions: + - v1beta1 + operations: + - CREATE + - UPDATE + resources: + - nodeclaims + - nodeclaims/status + scope: '*' + - apiGroups: + - karpenter.sh + apiVersions: + - v1beta1 operations: - CREATE - UPDATE + resources: + - nodepools + - nodepools/status + scope: '*' --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration @@ -52,4 +76,5 @@ webhooks: sideEffects: None objectSelector: matchLabels: - app.kubernetes.io/part-of: {{ template "karpenter.name" . }} \ No newline at end of file + app.kubernetes.io/part-of: {{ template "karpenter.name" . }} +{{- end }} diff --git a/charts/karpenter/templates/webhooks.yaml b/charts/karpenter/templates/webhooks.yaml index 8cbd744..1aa7326 100644 --- a/charts/karpenter/templates/webhooks.yaml +++ b/charts/karpenter/templates/webhooks.yaml @@ -1,3 +1,4 @@ +{{- if .Values.webhook.enabled }} apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: @@ -30,16 +31,28 @@ webhooks: - awsnodetemplates - awsnodetemplates/status scope: '*' + - apiGroups: + - karpenter.k8s.aws + apiVersions: + - v1beta1 + operations: + - CREATE + - UPDATE + resources: + - ec2nodeclasses + - ec2nodeclasses/status + scope: '*' - apiGroups: - karpenter.sh apiVersions: - v1alpha5 - resources: - - provisioners - - provisioners/status operations: - CREATE - UPDATE + resources: + - provisioners + - provisioners/status + scope: '*' --- apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration @@ -73,13 +86,26 @@ webhooks: - awsnodetemplates - awsnodetemplates/status scope: '*' + - apiGroups: + - karpenter.k8s.aws + apiVersions: + - v1beta1 + operations: + - CREATE + - UPDATE + resources: + - ec2nodeclasses + - ec2nodeclasses/status + scope: '*' - apiGroups: - karpenter.sh apiVersions: - v1alpha5 + operations: + - CREATE + - UPDATE resources: - provisioners - provisioners/status - operations: - - CREATE - - UPDATE \ No newline at end of file + scope: '*' +{{- end }} diff --git a/charts/karpenter/values.yaml b/charts/karpenter/values.yaml index 49ad8ae..93d8825 100644 --- a/charts/karpenter/values.yaml +++ b/charts/karpenter/values.yaml @@ -44,9 +44,6 @@ podAnnotations: {} podDisruptionBudget: name: karpenter maxUnavailable: 1 -# -- SecurityContext for the pod. -podSecurityContext: - fsGroup: 1000 # -- PriorityClass name for the pod. priorityClassName: system-cluster-critical # -- Override the default termination grace period for the pod. @@ -72,6 +69,8 @@ affinity: - matchExpressions: - key: karpenter.sh/provisioner-name operator: DoesNotExist + - key: karpenter.sh/nodepool + operator: DoesNotExist podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - topologyKey: "kubernetes.io/hostname" @@ -99,11 +98,9 @@ controller: # -- Repository path to the controller image. repository: public.ecr.aws/karpenter/controller # -- Tag of the controller image. - tag: v0.28.0 + tag: v0.32.3 # -- SHA256 digest of the controller image. - digest: sha256:4194fc219d5a1019cd8d7d69f518b903e972966cad2e9420ccbe9dca149f179d - # -- SecurityContext for the controller container. - securityContext: {} + digest: sha256:afa0d0fd5ac375859dc3d239ec992f197cdf01f6c8e3413e3845a43c2434621e # -- Additional environment variables for the controller pod. env: [] # - name: AWS_REGION @@ -122,16 +119,6 @@ controller: # cpu: 1 # memory: 1Gi - # -- Controller outputPaths - default to stdout only - outputPaths: - - stdout - # -- Controller errorOutputPaths - default to stderr only - errorOutputPaths: - - stderr - # -- Controller log level, defaults to the global log level - logLevel: "" - # -- Controller log encoding, defaults to the global log encoding - logEncoding: "" # -- Additional volumeMounts for the controller pod. extraVolumeMounts: [] # - name: aws-iam-token @@ -148,15 +135,39 @@ controller: # -- The container port to use for http health probe. port: 8081 webhook: - logLevel: error + # -- Whether to enable the webhooks and webhook permissions. + enabled: true # -- The container port to use for the webhook. port: 8443 + metrics: + # -- The container port to use for webhook metrics. + port: 8001 # -- Global log level logLevel: debug -# -- Global log encoding -logEncoding: console +# -- Log configuration (Deprecated: Logging configuration will be dropped by v1, use logLevel instead) +logConfig: + # -- Whether to enable provisioning and mounting the log ConfigMap + enabled: true + # -- Log outputPaths - defaults to stdout only + outputPaths: + - stdout + # -- Log errorOutputPaths - defaults to stderr only + errorOutputPaths: + - stderr + # -- Log encoding - defaults to json - must be one of 'json', 'console' + logEncoding: json + # -- Component-based log configuration + logLevel: + # -- Global log level, defaults to 'debug' + global: debug + # -- Controller log level, defaults to 'debug' + controller: debug + # -- Error log level, defaults to 'error' + webhook: error # -- Global Settings to configure Karpenter settings: + # -- AWS-specific settings (Deprecated: The AWS block inside of settings was flattened into settings) + aws: {} # -- The maximum length of a batch window. The longer this is, the more pods we can consider for provisioning at one # time which usually results in fewer but larger nodes. batchMaxDuration: 10s @@ -164,33 +175,31 @@ settings: # faster than this time, the batching window will be extended up to the maxDuration. If they arrive slower, the pods # will be batched separately. batchIdleDuration: 1s - # -- AWS-specific configuration values - aws: - # -- Cluster name. - clusterName: "" - # -- Cluster endpoint. If not set, will be discovered during startup (EKS only) - clusterEndpoint: "" - # -- The default instance profile to use when launching nodes - defaultInstanceProfile: "" - # -- If true then instances that support pod ENI will report a vpc.amazonaws.com/pod-eni resource - enablePodENI: false - # -- Indicates whether new nodes should use ENI-based pod density - # DEPRECATED: Use `.spec.kubeletConfiguration.maxPods` to set pod density on a per-provisioner basis - enableENILimitedPodDensity: true - # -- If true then assume we can't reach AWS services which don't have a VPC endpoint - # This also has the effect of disabling look-ups to the AWS pricing endpoint - isolatedVPC: false - # -- The VM memory overhead as a percent that will be subtracted from the total memory for all instance types - vmMemoryOverheadPercent: 0.075 - # -- interruptionQueueName is disabled if not specified. Enabling interruption handling may - # require additional permissions on the controller service account. Additional permissions are outlined in the docs. - interruptionQueueName: "" - # -- The global tags to use on all AWS infrastructure resources (launch templates, instances, etc.) across node templates - tags: + # -- Role to assume for calling AWS services. + assumeRoleARN: "" + # -- Duration of assumed credentials in minutes. Default value is 15 minutes. Not used unless assumeRoleARN set. + assumeRoleDuration: 15m + # -- Cluster CA bundle for TLS configuration of provisioned nodes. If not set, this is taken from the controller's TLS configuration for the API server. + clusterCABundle: "" + # -- Cluster name. + clusterName: "" + # -- Cluster endpoint. If not set, will be discovered during startup (EKS only) + clusterEndpoint: "" + # -- If true then assume we can't reach AWS services which don't have a VPC endpoint + # This also has the effect of disabling look-ups to the AWS pricing endpoint + isolatedVPC: false + # -- The VM memory overhead as a percent that will be subtracted from the total memory for all instance types + vmMemoryOverheadPercent: 0.075 + # -- interruptionQueue is disabled if not specified. Enabling interruption handling may + # require additional permissions on the controller service account. Additional permissions are outlined in the docs. + interruptionQueue: "" + # -- Reserved ENIs are not included in the calculations for max-pods or kube-reserved + # This is most often used in the VPC CNI custom networking setup https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html + reservedENIs: "0" # -- Feature Gate configuration values. Feature Gates will follow the same graduation process and requirements as feature gates # in Kubernetes. More information here https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/#feature-gates-for-alpha-or-beta-features featureGates: - # -- driftEnabled is in ALPHA and is disabled by default. - # Setting driftEnabled to true enables the drift deprovisioner to watch for drift between currently deployed nodes + # -- drift is in ALPHA and is disabled by default. + # Setting drift to true enables the drift disruption method to watch for drift between currently deployed nodes # and the desired state of nodes set in provisioners and node templates - driftEnabled: false + drift: false