Containers

How to containerize your solution!

Setup Kubernetes cluster to be production ready isn’t a simple task. It requires to take in consideration many topics.

In this walkthrough we will setup the steps required to have a Kubernetes cluster up and running.

This proposal is thought as to be used on Azure, even though most of the options could be reused for any cloud provider like Google Cloud or AWS.

Prerequisites

Azure Subscription to create AKS cluster
kubectl logged into Kubernetes cluster
PowerShell
Postman
Helm
DockerHub account (optional)

Introduction

The setup is spitted into different steps:

Bare components
Security
Scaling
Monitoring
Application

Setup - Overview

During this step we are going to setup Kubernetes cluster tackling the following components:

Setup private Docker images repository
Setup the Kubernetes cluster
Setup the Vault for secrets
Secure secrets
Setup networking
Setup autoscaler

Monitoring

The implementation will monitor both infrastructure and application side.

The monitoring will be implemented using different approach. Some are open source, other don’t.

Open-source Components uses in this demo:

NOTE: The open-source components have the enterprise version. It requires to have active subscription if you plan to use them.

Security and Networking

Security and Networking context will implement all the components that allows to handle:

routing (forward requests to the services throughout a reverse proxy)
termination (secure APIs calls by SSL or TLS)
throttling

Main components are:

Public IP
Vnet and Subnet
Api Gateway Ingress Controller

There are various products out of the box that can be used to implement the solution. All of them have PROS and CONS.

The most used ones are:

In this solution the first choice is based on Kong.

Authentication and authorization are implemented using OAuth2. Kong provides plugin that streamline the implementation.

Autoscaler

There are multiple options for scaling Kubernetes and containers in general.

Here (KEDA) Kubernetes-based Event Driven Autoscaling will be used.

RabbitMQ is used as the event source.

If you wish to use Kubernetes cluster apart from AKS, you can skip the Step 2.1 of provisioning the cluster and install KEDA on your own Kubernetes cluster.

Similarly, if you do not wish to execute the PowerShell scripts, you can execute the commands which are part of those scripts manually.

Code organization

docker-compose

Contains the docker compose files to install a bunch of enterprise level tools.

Please refer to this README for details.

docs

Contains some useful documentation like:

src

Contains the source code for a hypothetical application.

It contains the source code for a model class used by two services:

The Producers: a WebApi that send messages to RabbitMQ cluster
The Internal Service: a WebApi that can be used only inside the cluster
The Consumers: a Worker listening RabbitMQ messages.

Genocs.KubernetesCourse.WebApi contains the code for generating the messages which are published to a RabbitMQ queue, as well as to call an internal WebApi.

Genocs.KubernetesCourse.Internal.WebApi contains the code for the WebApi used only inside the cluster.

Genocs.KubernetesCourse.Worker contains the consumer code processing RabbitMQ messages.

Both the Producer and Consumer uses the common data model.

In order to build these using Dockerfile:

You can build them and they are ready to be pushed to:

Azure Container Registry docker-compose ACR
DockerHub docker-compose Dockerhub

The docker images can be built and push to dockerhub using the following commands:

# Build
docker-compose -f .\src\docker-compose-dockerhub.yaml build

# Push
docker-compose -f .\src\docker-compose-dockerhub.yaml push

NOTE: Please update the yaml files with the correct image version before use them.

PowerShell

Contains the helper PowerShell scripts to:

Provisioning the AKS cluster
To proxy into the Kubernetes control plane
To deploy the application
To delete the application
To delete the resource group

NOTE: AKS is expensive to keep alive

k8s

Contains Kubernetes manifest files for deploying the Producer, Consumer and the Autoscaler components to the Kubernetes cluster.

helm

Contains the Helm RBAC enabling yaml file which add the Cluster Role Binding for RBAC enabled Kubernetes cluster.

This was required before Helm 3.0 for the Tiller service. With helm 3.0, Tiller is no longer required.

terraform

Contains the terraform Kubernetes cluster setup. The setup is primary meant to be used on Azure. Anyway, deploy on other cloud Provider is quite straightforward.

skaffold

The Skaffold and Kaniko Kubernetes cluster setup.

This allows to setup Kubernetes cluster on Google Cloud.

SECTION 2 - Setup

Install Azure Container Registry ACR
Install Kubernetes Cluster AKS
Install Azure Key Vault AKV
Setup Network Infrastructure
Deploy Azure Key Vault Secret
Deploy RabbitMQ node
Deploy MongoDB node
Deploy Prometheus & Grafana
Deploy Jaeger
Deploy KEDA Autoscaler
Deploy Application
Deploy Application AutoScaler
Get list of the resources

2.1 Install Azure Container Registry ACR

The ACR (Azure Container Registry) allows to store Docker images inside a private repository.

Run initializeACR PowerShell script with default values from root directory.

.\Powershell\initializeACR.ps1

2.2 Install Kubernetes Cluster AKS

Run initializeAKS PowerShell script with default values from root directory.

.\Powershell\initializeAKS.ps1

2.3 Install Azure Key Vault AKV

The AKV Azure Key Vault is used to store every secret used by the application in a safe place.

Secret data are:

connection strings
API Key
certificates
public/private keys

Run initializeAKV PowerShell script with default values from root directory.

.\Powershell\initializeAKV.ps1

2.4 Setup Network Infrastructure

The AKS will use a network infrastructure composed by:

Public IP
VNET
AGIC (Application Gateway Ingress Controller)

The PowerShell script will ‘Create peering between AGIC and AKS and vice versa as well.

Run initializeNetwork PowerShell script with default values from root directory.

.\Powershell\initializeNetwork.ps1

2.5 Deploy Azure Key Vault Secret

This step shows how to setup a script file to push Secrets inside AKV.

Run deployAKV-secrets PowerShell script with default values from root directory.

.\Powershell\deployAKV-secrets.ps1

2.6 Deploy RabbitMQ node

This step shows how to setup a RabbitMQ node inside AKS.

Not recommended for Production

Run deployRabbitMQ PowerShell script with default values from root directory.

.\Powershell\deployRabbitMQ.ps1

2.7 Deploy MongoDB node

This step shows how to setup a MongoDb instance inside AKS.

Not recommended for Production

Run deployMongoDB PowerShell script with default values from root directory.

.\Powershell\deployMongoDB.ps1

2.8 Deploy Prometheus & Grafana

This step shows how to setup a Prometheus and Grafana inside AKS.

Run deployPrometheus PowerShell script with default values from root directory.

.\Powershell\deployPrometheus.ps1

2.9 Deploy Jaeger

This step shows how to setup a Jaeger distributed Tracing inside AKS.

Run deployJaeger PowerShell script with default values from root directory.

.\Powershell\deployJaeger.ps1

2.10 Deploy KEDA Autoscaler

This step shall install the KEDA autoscaler inside AKS.

Run deployKEDA powershell script with default values from root directory.

.\Powershell\deployKEDA.ps1

Verify KEDA is installed correctly on the Kubernetes cluster.

kubectl get all -n keda

2.11 Deploy Application

Deploy External WebAPI (Producer), Internal WebAPI and Backgroud Worker Consumer.

Execute the PowerShell script.

# Use this to setup the application with Secret coming from file as Opaque secret
cd Powershell 
.\deployApplications-AKS-SecretFile.ps1
cd ..

# Use this to deploy the application using secret coming from Key Vault
cd Powershell 
.\Powershell\deployApplications-AKS.ps1
cd ..

The deployApplications-AKS PowerShell script deploys the RabbitMQConsumer and RabbitMQProducer in the correct order.

Alternately, all the components can also be deployed directly using the kubectl apply command recursively on the k8s directory as shown below:

# Run the kubectl apply recursively on k8s directory
kubectl apply -R -f .

2.12 Deploy Application AutoScaler

Execute the deployAutoScaler.ps1 PowerShell script.

cd Powershell 
.\deployAutoScaler.ps1
cd ..

Note: The default options can be overwritten by passing arguments to the initializeAKS script. In the below example, we are overriding the number of nodes in the AKS cluster to 4 instead of 3 and resource group name as kedaresgrp.

.\Powershell\initilaizeAKS `
    -workerNodeCount 4 `
    -resourceGroupName "kedaresgrp"

If you do not wish to run the individual PowerShell scripts, you can run one single script which will deploy all the necessary things by running the above scripts in correct order.

.\Powershell\deployAll.ps1

2.13 Get list of the resources

General purpose command to get info about K8S resources

# General purpose command to get all resources
# -A to get all namespaces
# -w to watch for changes

# Get services
kubectl get svc

# Get secrets
kubectl get ingress

# Get pod
kubectl get pod

# Get deployments
kubectl get deployment

# Get secrets
kubectl get secret

# Get custom resource definition
kubectl get crd

As we can see above, RabbitMQ service is available within the Kubernetes cluster and it exposes 4369, 5672, 25672 and 15672 ports. We will be using 15672 port to map to a local port.

Also note the public LoadBalancer IP for the Producer Service. In this case the IP is 52.139.237.252.

Note: This IP will be different when the services are redeployed on a different Kubernetes cluster.

2.11 Watch for deployments

The RabbitMQ ScaledObject will be deployed as part of the deployment. Watch out for the deployments to see the changes in the scaling as the number of messages increases

kubectl get deployment -w
kubectl get deploy -w

Initially there is 1 instance of RabbitMQ-consumer and 2 replicas of the RabbitMQ-producer (Producer) deployed in the cluster.

2.12 Browse RabbitMQ Management UI

RabbitMQ Management UI is enabled by port forwarding.

In order to do this, open a bash shell and run the command.

Please keep the shell open to keep open the port forwarding.

kubectl port-forward svc/rabbitmq 15672:15672

Open the web browser: RabbitMQ

Remember that these were set during the installation of RabbitMQ services using Helm. If you are using any other user, please update the username and password accordingly.

2.13 Generate load using `Visual Studio code Extension`

TBV

2.14 Generate load using `Postman`

I am using Postman to submit a POST request to the API which generates 2000 messages onto a RabbitMQ queue named hello. You can use any other command line tool like CURL to submit a GET request.

Use the EXTERNAL-IP -52.139.237.252 with port 8080 to submit a GET request to the API. http://52.139.237.252:8080/api/TechTalks/Generate?numberOfMessages=2000

Note that we are setting the number of messages to be produced by Producer as 2000 in this case. You can change the number to any other integer value.

After building the GET query, hit the blue Send button on the top right. If everything goes fine, you should receive a 200 OK as status code.

The Producer will produce 2000 messages on the queue named hello. The consumer is configured to process 10 messages in a batch. The Consumer take the message, deserialize it and send the ACK to the message broker.

2.15 Auto-scaling in action

See the number of containers for consumer grow to adjust the messages and also the drop when messages are processed.

On the left-hand side of the screen you can see the pods auto scaled and on the right we see the deployments autoscaled progressively to 2, 4, 8, 16 and 30.

While the messages are being processed, we can also observe the RabbitMQ management UI.

Our consumer processes 50 messages in a batch by prefetching them together. This can be verified by looking at the details of the consumers.

Once all the messages are processed, KEDA will scale down the pods and the deployments.

List Custom Resource Definition

kubectl get crd

As part of the KEDA installation, ScaledObject and TriggerAuthentications are deployed on the Kubernetes cluster.

2.16 Delete all resources

.\Powershell\deleteRG.ps1

Resource Groups

This demo will create all the Azure Resource inside:

rg-aks-genocs
rg-agic-genocs
MC_rg-aks-genocs_aks-genocs_westeurope
DefaultResourceGroup-WEU

1. rg-aks-genocs

aks-genocs - Kubernetes service
genocscontainer - Container registry
kv-genocsakst - Key vault

2. rg-agic-genocs

agic-genocs - Application gateway
agic-pip - Public IP address
agic-vnet - Virtual network

3. MC_rg-aks-genocs_aks-genocs_westeurope

4c0f07f9-b962-478a-a875-be77a79462dc Public IP address West Europe
aks-agentpool-33168902-nsg Network security group West Europe
aks-agentpool-33168902-routetable Route table West Europe
aks-genocs-agentpool Managed Identity West Europe
aks-nodepool1-12518635-vmss Virtual machine scale set West Europe
aks-vnet-33168902 Virtual network West Europe
kubernetes Load balancer West Europe
omsagent-aks-genocs Managed Identity West Europe
pvc-49df2627-0ef7-4264-8441-f0e36b629bf2 Disk West Europe

4. DefaultResourceGroup-WEU

Community

Discord @genocs
Facebook Page @genocs
Youtube Channel @genocs

Contributors

Submit your PR and join the list!

License

This project is licensed with the MIT license.

Support ⭐

Has this project helped you learn something New? or Helped you at work? Here are a few ways by which you can support.

Leave a star! ⭐
Recommend this awesome project to your colleagues. 🥇
Do consider endorsing me on LinkedIn for ASP.NET Core - Connect via LinkedIn 🦸
Or, If you want to support this project in the long run, consider buying me a coffee! ☕

Acknowledgements

A lot of people inspired me and provided unevaluable amount of information:

Here some of them:

References

Hands-on-Kubernetes-on-Azure-Third-Edition

Containers

How to containerize your solution!

Prerequisites #

Introduction #

Setup - Overview #

Monitoring #

Security and Networking #

Autoscaler #

Code organization #

SECTION 2 - Setup #

2.1 Install Azure Container Registry ACR #

2.2 Install Kubernetes Cluster AKS #

2.3 Install Azure Key Vault AKV #

2.4 Setup Network Infrastructure #

2.5 Deploy Azure Key Vault Secret #

2.6 Deploy RabbitMQ node #

2.7 Deploy MongoDB node #

2.8 Deploy Prometheus & Grafana #

2.9 Deploy Jaeger #

2.10 Deploy KEDA Autoscaler #

2.11 Deploy Application #

2.12 Deploy Application AutoScaler #

2.13 Get list of the resources #

2.11 Watch for deployments #

2.12 Browse RabbitMQ Management UI #

2.13 Generate load using Visual Studio code Extension #

2.14 Generate load using Postman #

2.15 Auto-scaling in action #

2.16 Delete all resources #

Resource Groups #

1. rg-aks-genocs #

2. rg-agic-genocs #

3. MC_rg-aks-genocs_aks-genocs_westeurope #

4. DefaultResourceGroup-WEU #

Community #

Contributors #

License #

Support ⭐ #

Acknowledgements #

References #