ETL with Azure Container Registries(ACR) and Azure Kubernetes Servcie (AKS)
## Background
Azure Kubernetes Service (AKS) is a great way to deploy this your-app application. This document describe the process to deploy this app to AKS. It also means to be a basic tutorial.
## Resources Preparation
Before we get started, we assume the `Azure CLI` & `kubectl` has been installed locally. If you have not installed `kubectl`, `Azure CLI` can help you
``` install kubectl
az aks install-cli
```
## Create an Anzure AKS Service
1. log in: portal.azure.com
2. search ‘aks’ and follow the on-screen instruction to set up an AKS service. You can also use this [doc.](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough) as a reference
3. Get credentials to connect to kubernetes cluster using kubectl
```
az aks get-credentials — name YOUR-AKS-SERVICE-NAME — resource-group YOUR-RESOURECE-GROUP
```
4. Show a dashboard of Kubernetes clusters in a web browser
```
az aks browse — name YOUR-AKS-SERVICE-NAME — resource-group YOUR-RESOURECE-GROUP
```
## Using a ConfigMap to define non-sensitive configuration data
ConfigMaps are used in Kubernetes to decouple non-sensitive configuration data from images and templates used to deploy an application. We use ConfigMap object to map the environment variables in the pod specification to the keys defined the ConfigMap.
1. Define a configmap yaml file:
```
apiVersion: v1
kind: ConfigMap
metadata:
name: YOUR-aks-configmap
namespace: default
data:
ROOT_DIR: /app
ENV: dev
SLEEP_INTERVAL: “10”
QUEUE_BATCH_SIZE: “32”
DEQUEUE_COUNT: “1”
```
Save this `ConfigMap` to a `Yaml` file, say, `YOUR-aks-configmap.yml`.
2. Creat cluster config map
``` create k8s cluster configmap
kubectl create — filename YOUR-aks-configmap.yml — record
```
3. To verify
```Get YOUR configmap
kubectl get configmaps YOUR-aks-configmap -o yaml
```
and
```
kubectl describe configmap YOUR-aks-configmap
```
4. Delete
```
kubectl delete configmaps YOUR-aks-configmap
```
## Use a secret in Kubernetes to define sensitive configuration data
In Kubernetes (K8S), a secret is an object that contains a small amount of sensitive data such as passwords, connection strings, OAuth tokens, and SSH keys.
In this application, the storage connection string, queue connection string and OMS secret are sensitive information. The steps to use K8S secret object are
1. create YOUR-k8s-secret.yml
```
apiVersion: v1
kind: Secret
metadata:
name: azure-secret
type: Opaque
data:
AZURE_STORAGE_CONNECTION_STRING: BASE64-ENCODED-STORAGE-CONNECTION-STRING
AZURE_QUEUES_CONNECTION_STRING: BASE64-ENCODED-QUEUES-CONNECTION-STRING
```
In Mac OS, you can use `echo -n ‘YOUR-STRING’ | base64` in a terminal to generate base64 encoded string
2. create k8s cluster secret
```k8s cluster secret
kubectl create — filename YOUR-aks-secret.yml — record
```
You might see a `secret/azure-secret` in stdout. `azure-secret` is the secret file name you need to refer later.
3. verify secret has been created successfully
```
kubectl get secret azure-secret -o jsonpath=”{.data.AZURE_STORAGE_CONNECTION_STRING}” | base64 — decode; echo
```
4. delete secret
In case you need to delete the secret, use
``` delete secret
kubectl delete secret azure-secret
```
There are other ways to coordinate Azure Cosmos DB credentials such as [Open Service Broker for Azure (OSBA)](https://github.com/azure/open-service-broker-azure).
## Set Docker Image secret for AKS
Before our AKS cluster to pull images from Azure Container Registry (ACR), a secret needs to be set. There are a few ways to do so.
1. Create a secret with K8S
This step is similar to the previous secret creation step. For instance, `your-app-azure-docker-secret` is the secret name
``` create secret
kubectl create secret docker-registry your-app-azure-docker-secret — docker-server=<your-registry-server> — docker-username=<your-name> — docker-password=<your-pword> — docker-email=<your-email>
```
You will use this `your-app-azure-docker-secret` when deploying to the AKS. The detail will be shown later.
2. Use Azure Service Principal
Use those links to for the Azure Serivce Principal authentication
[Authenticate with a private Docker container registry](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-authentication)
[Azure Container Registry authentication with service principals](https://docs.microsoft.com/en-us/azure/container-registry/container-registry-auth-service-principal)
## Deploy to AKS from your local computer
In this application, each K8S deployment yaml has been created, all we need to do are to combine them together and use the configmap & secret created in the previous steps accordingly. Here is an exmaple of using configmap & secret
```sample of using configmap & secret
- name: AZURE_QUEUES_CONNECTION_STRING
valueFrom:
secretKeyRef:
name: azure-secret
key: AZURE_QUEUES_CONNECTION_STRING
- name: your_blob_CONTAINER
valueFrom:
configMapKeyRef:
name: YOUR-aks-configmap
key: your_blob_CONTAINER
…
imagePullSecrets:
- name: azure-docker-secret
```
To deploy,
```deploy your-app email ingestion to AKS
kubectl create — filename YOUR-aks-deployment.yml — record
```
You can delete deployed app by
```delete deployed app
kubectl delete deployments.apps your-app
```
To make sure the cluster runs successfully, open the dashboard locally
```
az aks browse — name YOUR-AKS-SERVICE-NAME — resource-group YOUR-RESOURCE-GROUP
```
or visit `portal.azure.com`, and navigate your AKS service. You can also see logs and other more information
# References
[Build and deploy a multi-container
application in Azure Container Service](https://azure.microsoft.com/mediahandler/files/resourcefiles/build-and-deploy-a-multi-container-application-in-azure-container-service/Build_and_deploy_a_multi-container_application_in_Azure_Container_Service.pdf)
