[sc-59294] Creating the httpscaledobject file for the Keda project (#127

)

* Creating the httpscaledobject file the for Keda project

Dockerfile

@@ -16,8 +16,8 @@ FROM alpine
 
 ARG DEFAULT_HELM_REPO_URL
 
-ENV KUBECTL_VERSION=v1.22.5
-ENV KUSTOMIZE_VERSION=v4.5.5
+ENV KUBECTL_VERSION=v1.24.11
+ENV KUSTOMIZE_VERSION=v5.0.1
 ENV KR_BASE_OVERLAY_PATH=/usr/local/kube-review/deploy/resources/base
 
 # Default packages #
@@ -34,7 +34,7 @@ RUN curl -LO --silent https://storage.googleapis.com/kubernetes-release/release/
   && chmod +x /usr/local/bin/kubectl
 
 # Kustomize
-RUN curl -L --silent https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz -o kustomize.tar.gz \
+RUN curl --proto "=https" -L --silent https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/${KUSTOMIZE_VERSION}/kustomize_${KUSTOMIZE_VERSION}_linux_amd64.tar.gz -o kustomize.tar.gz \
   && tar -zxf kustomize.tar.gz \
   && mv ./kustomize /usr/local/bin/kustomize \
   && rm -f kustomize.tar.gz

README.md

@@ -30,8 +30,9 @@ These are some of the features supported by *Kube-Review*:
 - Extension through pre and post install hooks;
 - Scalable and lean infra using Nginx Ingress and Let's Encrypt WildCards domains;
 - Environment isolation through namespaces;
-- Connection test after deployment.
-- [Vertical Pod Autoscaling](https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler)
+- Connection test after deployment;
+- [Vertical Pod Autoscaling](https://cloud.google.com/kubernetes-engine/docs/concepts/verticalpodautoscaler);
+- [Scaling From/To zero with Keda HTTP Add-On](https://github.com/kedacore/charts/tree/main/http-add-on);
 
 ## Documentation
 

docs/customization.md

@@ -69,14 +69,15 @@ If one needs to inject dynamic variables in the resources, that can be done by u
 ]
 ```
 
-Finally, in order to use the overlay one has to specify the `KR_OVERLAY_PATH` variable and the env vars to be injected in the resources:
+Finally, in order to use the overlay one has to specify the `KR_OVERLAY_PATH` and `KR_OVERLAY_TARGET_DIR` variables and the env vars to be injected in the resources:
 
 ```
 KR_ID=nginx \
 KR_IMAGE=nginx:latest \
 KR_DOMAIN="my-domain.io" \
 KR_CONTAINER_PORT="80" \
 KR_OVERLAY_PATH=src/deploy/resources/example \
+KR_OVERLAY_TARGET_DIR=example \
 LABEL=6.2.1 \
 src/deploy/deploy
 ```

docs/reference.md

@@ -44,7 +44,8 @@ The `deploy` command contains many options which can be passed as environment va
 | KR_PRE_HOOK | A shell command to be executed before the deployment starts. | - | false |
 | KR_POST_HOOK | A shell command to be executed after the deployment is finished. | - | false |
 | KR_BASE_OVERLAY_PATH | The path containing the base kustomize overlay to be used. | src/deploy/resources/base | false |
-| KR_OVERLAY_PATH | The path containing a kustomize overlay to be used. | - | false |
+| KR_OVERLAY_PATH | The path containing a kustomize overlay files to be used. | - | false |
+| KR_OVERLAY_TARGET_DIR | The folder name inside of the `KR_OVERLAY_PATH` path containing a kustomize file to be applied, for example: `$KR_BASE_OVERLAY_PATH/example`. | - | false |
 | KR_VERBOSE | Prints verbose or debug messages. Should not be used in production. | false | false |
 | KR_SHARED_PATH | The path of a folder that will be copied to `__shared_path/` folder into the overlay path. Can be used to load files from the host as configMaps, for example. | - | false |
 

docs/tutorial.md

@@ -11,6 +11,7 @@
     + [Install](#install-1)
     + [Configure](#configure-2)
     + [Vertical Pod autoscaling - VPA](#vertical-pod-autoscaling---vpa)
+  * [Scaling From or To zero with Keda](#scaling-from-or-to-zero-with-keda)
   * [Deploying an environment](#deploying-an-environment)
   * [Kudos](#kudos)
 
@@ -46,7 +47,7 @@ The following command creates a EKS Cluster and node group named `tutorial`:
 ```shell
 eksctl create cluster \
 --name tutorial \
---version 1.19 \
+--version 1.24 \
 --with-oidc \
 --nodegroup-name tutorial \
 --node-type m5.large \
@@ -85,7 +86,7 @@ helm install \
   cert-manager jetstack/cert-manager \
   --create-namespace
   --namespace cert-manager \
-  --version v1.7.1 \
+  --version v1.9.1 \
   --set installCRDs=true \
   --set 'extraArgs={--dns01-recursive-nameservers-only}'
 ```
@@ -135,6 +136,7 @@ kubectl describe certificate --namespace cert-manager
 
 If everything went well, you should see something like:
 
+```shell
    ...
     Status:
     Conditions:
@@ -149,6 +151,7 @@ If everything went well, you should see something like:
     Renewal Time:            2021-08-06T14:00:55Z
     Events:                    <none>
     ...
+```
 
 For more info about this, check the certmanager's [official docs](https://cert-manager.io/docs/configuration/acme/).
 
@@ -168,7 +171,7 @@ helm install \
   --create-namespace
   --wait \
   --namespace nginx-ingress \
-  --version 4.1.3 \
+  --version 4.4.0 \
   --set rbac.create=true \
   --set controller.extraArgs.default-ssl-certificate=cert-manager/tutorial-prd
 ```
@@ -204,6 +207,46 @@ For more information, you can also check this [guide](https://github.com/kuberne
 
 We are using the VPA service with `updateMode: "Off"` by default for all containers, including the `kube-review` and `sidecar`. To change these settings, we recommend that you use the [customization](customization.md) page. You can see the file created for this project [here](../src/deploy/resources/base/vpa.yml).
 
+## Scaling From or To zero with Keda
+
+**NOTE: KEDA requires Kubernetes cluster version 1.24 and higher**
+
+We implemented the [Keda - Kubernetes-based Event Driven Autoscaling](https://github.com/kedacore/keda) project as a component in the Kube Review project because the review environments are a temporal environment running for a few days, we considered that saving money is an essential decision in that case. Implementing Keda help us with the possibility of scaling from/to zero the environment through HTTP requests with the [HTTP Add-On](https://github.com/kedacore/http-add-on) project.
+
+To install both components Keda and HTTP Add-On, you can follow their guides below:
+
+**NOTE: We tested the following versions: Keda 2.10.1 and HTTP Add-on 0.4.1**
+
+- [Keda](https://keda.sh/docs/2.10/deploy/)
+- [HTTP Add-on](https://github.com/kedacore/http-add-on/blob/main/docs/install.md)
+
+Using the Keda project in the review environments, we don't need to take care of an ingress setting per each review environment (namespace). We should move it to the Keda namespace where we will have a `wildcard` covering all review environments URLs, then per each review environment (namespace), we just need to have an `HTTPScaledObject` created and used by Keda to collect metrics to scaling up/down the environment checking the HTTP requests.
+
+```yaml
+apiVersion: networking.k8s.io/v1
+kind: Ingress
+metadata:
+  name: kube-review-ingress
+  namespace: keda
+  annotations:
+    kubernetes.io/ingress.class: nginx
+spec:
+  tls:
+    - hosts:
+        - '*.example.com'
+  rules:
+    - host: '*.example.com'
+      http:
+        paths:
+          - path: /
+            pathType: Prefix
+            backend:
+              service:
+                name: keda-add-ons-http-interceptor-proxy
+                port:
+                  number: 8080
+```
+
 ## Deploying an environment
 
 Now that everything is installed and working, we just need to call the deploy script to actually deploy a review env.
@@ -218,6 +261,7 @@ KR_IMAGE=nginx:latest \
 KR_DOMAIN="${MY_DOMAIN}" \
 KR_CONTAINER_PORT="80" \
 KR_OVERLAY_PATH=src/deploy/resources/example \
+KR_OVERLAY_TARGET_DIR=example \
 LABEL=6.2.1 \
 src/deploy/deploy
 ```

src/deploy/deploy

@@ -34,8 +34,9 @@ kube_context=$KR_KUBE_CONTEXT
 test_connection=${KR_TEST_CONNECTION:-true}
 test_connection_url_path=${KR_TEST_CONNECTION_URL_PATH:-"/"}
 shared_path=$KR_SHARED_PATH
-overlay_path=$KR_OVERLAY_PATH
 base_overlay_path=${KR_BASE_OVERLAY_PATH:-"src/deploy/resources/base"}
+overlay_path="$KR_OVERLAY_PATH/."
+overlay_target_dir=$KR_OVERLAY_TARGET_DIR
 delete_before_apply=${KR_DELETE_BEFORE_APPLY:-false}
 
 updated_at=$(date +%s)
@@ -72,7 +73,7 @@ install_resources () {
   # If we don't have a overlay to apply, we just run from base overlay
   cp -R "$base_overlay_path" "$work_dir"
   if [ "$overlay_path" != "" ]; then
-    cp -R "$overlay_path" "$work_dir/overlay"
+    cp -R "$overlay_path" "$work_dir"
   fi
 
   # Apply variable substituion on patches
@@ -85,9 +86,11 @@ install_resources () {
     fi
 	done
 
-  cd "$work_dir/base"
-  if [ "$overlay_path" != "" ]; then
-    cd "$work_dir/overlay"
+  # Accessing the folder with the kustomize files
+  if [ "$overlay_target_dir" != "" ]; then
+    cd "$work_dir/$overlay_target_dir"
+  else
+    cd "$work_dir/base"
   fi
 
   # Copy files to shared path if it is present
@@ -105,6 +108,7 @@ install_resources () {
   app.kubernetes.io/updated_at:"$updated_at" \
   app.kubernetes.io/is_ephemeral:"$is_ephemeral" \
   app.kubernetes.io/job_url:'"'$job_url'"'
+
   if [ "$verbose" = "true" ]; then
     kustomize build
   fi
@@ -120,7 +124,7 @@ install_resources () {
     fresh_install=1; \
   fi
 
-  # Apply changes generated by kustomize,
+  # Apply changes generated by kustomize
   kustomize build | kubectl apply -f - -n "$namespace" --wait=true
 
   if [[ $fresh_install != 1 ]]; then
@@ -156,7 +160,7 @@ test_url() {
   then
     full_url=$url$test_connection_url_path
     echo "Running connection test against: $full_url"
-    output=$(curl --silent --fail --retry 3 "$full_url")
+    output=$(curl --silent --fail --max-time 10 --retry 10 --retry-delay 5 --retry-connrefused --retry-all-errors "$full_url")
     if [[ $? != 0 ]]; then
       echo "Connection test has failed with the following test output: $output";
       exit 1;

src/deploy/resources/base/deployment.yml

@@ -3,7 +3,7 @@ kind: Deployment
 metadata:
   name: deployment
 spec:
-  replicas: 1
+  replicas: 0
   strategy:
     rollingUpdate:
       maxSurge: 1

src/deploy/resources/base/httpscaledobject.yml

@@ -0,0 +1,14 @@
+apiVersion: http.keda.sh/v1alpha1
+kind: HTTPScaledObject
+metadata:
+  name: keda-http
+spec:
+  host: "PLACEHOLDER"
+  targetPendingRequests: 100
+  scaleTargetRef:
+    deployment: kube-review-deployment
+    service: kube-review-service
+    port: 80
+  replicas:
+    min: 0
+    max: 1

src/deploy/resources/base/kustomization.yaml

@@ -7,7 +7,7 @@ resources:
   - namespace.yml
   - serviceaccount.yml
   - service.yml
-  - ingress.yml
+  - httpscaledobject.yml
   - deployment.yml
   - vpa.yml
 
@@ -23,13 +23,12 @@ patches:
   - path: patches/service.patch.json
     target:
       kind: Service
-  - path: patches/ingress.patch.json
+  - path: patches/httpscaledobject.patch.json
     target:
-      kind: Ingress
+      kind: HTTPScaledObject
   - path: patches/deployment.patch.json
     target:
       kind: Deployment
-      name: deployment
   - path: patches/namespace.patch.json
     target:
       kind: Namespace

src/deploy/resources/base/patches/httpscaledobject.patch.json

@@ -0,0 +1,7 @@
+[
+    {
+        "op": "replace",
+        "path": "/spec/host", 
+        "value": "$KR_NAMESPACE.$KR_DOMAIN"
+    }
+]