Target audience

The article is focused on infrastructure cost optimization with the usage of AWS EC2 Spot instances in AWS EKS clusters. Some knowledge and experience with both AWS and Kubernetes environment is desirable to properly understand the presented concepts and solutions.

Introduction to AWS EKS service

AWS EKS which stands for Elastic Kubernetes Service is an AWS-managed Kubernetes service. It is a very popular choice when it comes to running containerized workloads in AWS cloud because it makes it easy to run upstream, secure, and highly available Kubernetes clusters and also it takes off of the user a big part of cluster management work, shifting the focus on actual application development.

Compute power for EKS clusters is delivered through EC2 instances or AWS Fargate service. When it comes to EC2 instances, several pricing models are available. Spot is most interesting in terms of cost efficiency and it is also available for EKS. This type of instance takes advantage of spare compute power capacity in the AWS Cloud and they are available at up to 90% discount compared to On-Demand prices. The only downside of this approach is that such instances can be terminated with 2 minutes’ notice when there will be increased demand for regular instances.

Having this limitation in mind, Spot EC2 instances are an excellent choice when it comes to running stateless, fault-tolerant workloads such as containerized applications, CI/CD workers, Machine Learning, Big Data ETLs, High-Performance Computing workloads, and other flexible apps. A good example would be a containerized stateless API service, because pods can be gracefully terminated and then quickly scheduled again on the other node, in the event of Spot instance termination.

AWS EKS node management

There are 2 approaches when it comes to EKS nodes management – self-managed node group and EKS-managed node group.

With a self-managed node group all of the configuration management lies on the user, which means that it is needed to create some pipeline to build and maintain configuration for EC2 Auto Scaling groups, there is also a problem of handling Spot instances terminations in a good manner (without disruptions to running service) and last but not least all the work related to keeping nodes up to date, eg. updates to the kubelet engine on worker nodes.

On the other side, the second option which seems to be more suitable in most cases, EKS-managed node group takes off of user whole work related to nodes management – all is needed is to set a parameter to let EKS know that it should manage node group and supply list of instance types that we would like to build the cluster on. A very important feature of EKS-managed node groups is that there is no need to install any extra automation tools in order to handle spot interruptions (for self-managed node groups there are dedicated tools for such purpose, like aws-node-termination-handler). How it works in short words – when one of the worker spot instances gets into state with a high risk of being terminated, it gets EC2 rebalance recommendation (it might arrive sooner than the standard 2 minutes spot interruption notice), which then kicks off the process to spin up replacement instances.

In both approaches, if spot instances are used, it is very important to configure more than one (at least a few) instance types, even if we have the same type of workload in the whole cluster – such method enhances the availability of Spot instances configurable for the cluster. Example to explain a problem here – if we would have an EKS cluster which consists of m5.large spot instances (8GB RAM, 2vCPU) and some other on-demand instances, in the event of spot instance termination and lack of compute resources on current on-demand nodes, a new on-demand node would need to be started to migrate pods to this machine. To mitigate such a problem and keep the cost as low as possible we can add a similar instance type to the list of available types, like t3.large (8GB RAM, 2vCPU burst). Then there is a higher chance that spot instances will be requested (there are separate spare capacity pools for each type of instance) and there will be no need to spin up new on-demand instances.

It is also worth mentioning that there are few allocation strategies for Spot instances. Most important are:

• The capacity-optimized strategy ensures that Spot instances will be requested from the most optimal capacity pool, so the pool which currently has the biggest number of instances available. This strategy works towards decreasing the number of Spot interruptions. EKS-managed node groups are configured with this strategy.
• The lowest-price strategy works towards cost optimization – the cheapest available instance will always be requested.

If there are different workloads in a single cluster, like heavy memory usage (eg. in-memory cache) and GPU-optimized calculations suitable approach would be to combine various instance types in a single node group or even create separate node groups for each type of workload – both methods are supported.

Another use case is when we would like to be sure that at least x replicas of a particular service (or some mission-critical component, or some stateful service that should not be interrupted at all) should be scheduled only on on-demand hosts.

High availability and reliability

Last but not least often it is important to design platforms with attention to high availability and reliability, so ideally we would like to have worker nodes spread between multiple availability zones/regions and we would like to make sure that workloads are also split on mentioned nodes, which are physically located in various parts of the world.

The solution for all problems defined above is to use a combination of placement strategies that are delivered by Kubernetes, such as selectors, taint-toleration mechanism, or node/pod affinity & anti-affinity. In order to properly design a placement strategy, nodes labels should be utilized. for EKS-managed node groups, there are some labels added automatically for each node – capacity type (SPOT or ON_DEMAND), node group name, instance type, etc.

Autoscaling

The last topic which is also important when it comes to designing EKS Cluster with Spot instances is autoscaling. There are 2 dimensions in which we can scale in/out:

• compute power which is the count of worker nodes joined to cluster
• service scaling which is the count of replicas for a particular service running on Kubernetes

To achieve the possibility to flexibly adjust size of the cluster in an automated way, cluster-autoscaler addon can be used. This component continuously monitors cluster state when it comes to 2 factors – if there are pods that failed to run due to insufficient resources or if there are nodes that have been underutilized for an extended period of time and pods which are already scheduled on them can be placed on other existing nodes. When at least one of the above conditions is true, cluster-autoscaler triggers the process of scaling in/out the cluster.

To automatically scale the amount of replicas, Horizontal Pod Autoscaler can be used. HPA is Kubernetes native resource which task is to reduce or increase the number of replicas for a particular workload based on observed CPU utilization (There is also the possibility to use application-provided metrics as a trigger for scaling activities) – by default, it fetches metrics collected by metrics-server, which is additional Kubernetes addon.

Summary

Scaling mechanisms mentioned above should be always considered when running containers using Kubernetes as these are big steps towards finding equilibrium between High Availability/High Resilience of service and cost-efficiency of provisioned infrastructure. To reduce cloud cost even further, Spot instances should be always considered but what is most important, these are not suitable for every workload (also containerization as itself is not a golden solution for every problem) – it is essential to understand what the lifecycle of the application looks like, what are the scaling needs and what is the so-called tech-nature of a particular application.

Sources

• https://aws.amazon.com/blogs/containers/amazon-eks-now-supports-provisioning-and-managing-ec2-spot-instances-in-managed-node-groups/
• https://ec2spotworkshops.com/
• https://docs.aws.amazon.com/eks/latest/userguide/managed-node-groups.html
• https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
• https://kubernetes.io/docs/tasks/debug-application-cluster/resource-metrics-pipeline/#metrics-server
• https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler
• https://docs.aws.amazon.com/eks/latest/userguide/cluster-autoscaler.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-fleet.html
• https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/rebalance-recommendations.html
• https://app.cloudcraft.co/
  
  Author: Wojciech Woźniak, Cloud/DevOps Engineer @ Appliscale
  LinkedIn: https://www.linkedin.com/in/wojciech-woźniak-58a402145/