Kubernetes Deployment Strategies: On-Premises vs. EKS vs. AKS

1. Introduction

Kubernetes has become the de facto standard for modern application deployment and operations.
In practice, production environments typically fall into three categories:

• On-premises bare-metal
• Amazon EKS
• Azure AKS

The choice between them affects far more than cost—it also shapes networking, storage, pod density, and autoscaling strategies.
This article examines the unique characteristics and operational approaches of each environment, including autoscaling with HPA, CA, and Karpenter.

2. On-Premises Kubernetes (Bare-Metal)

Key Traits

• Installed directly on physical servers.
• Full control over CPU, memory, disks, and networking.

Operational Approach

• Databases & Messaging: MySQL, PostgreSQL, MongoDB, and Kafka are best deployed on dedicated bare-metal servers or clusters.
• Logging & Search: Can run inside Kubernetes, but isolation on dedicated nodes is recommended.
• Monitoring Tools: Typically deployed as pods, but in on-prem environments it’s more reliable to run monitoring platforms on external bare-metal servers since the cluster itself also needs monitoring.

Pros

• Maximum performance.
• Data sovereignty and security.
• Full customization and tuning.

Cons

• Heavy burden of upgrades and incident response.
• High upfront hardware investment.
• Slow to scale physical resources.

3. Amazon EKS

Key Traits

• Control plane managed by AWS.
• Worker nodes run on EC2 or Fargate.
• Tight integration with IAM, S3, RDS, CloudWatch, and the AWS ecosystem.

Operational Notes

• Networking & Pod Density:
  • AWS VPC CNI assigns pod IPs directly from VPC subnets.
  • Pod density per node depends on instance type and ENI limits.
  • Prefix delegation and secondary CIDRs extend scalability.
• Storage:
  • EBS is AZ-bound.
  • EFS and FSx support multi-AZ.
  • High-performance DB workloads are typically offloaded to RDS or Aurora.
• Network Services:
  • Service Type=LoadBalancer provisions NLB/ALB automatically.
  • Security group and network policy design is critical in multi-tenant clusters.
• Cost Structure:
  • Control plane: ~$0.10/hour.
  • Worker nodes: EC2/Fargate charges.

4. Azure AKS

Key Traits

• Control plane provided in Free, Standard, or Premium tiers.
• Strong integration with Azure AD, Monitor, and DevOps pipelines.
• Best option for Windows container workloads.

Networking & Pod Density

• Azure CNI: Pods receive VNet IPs. Default 30–110 per node, up to 250. Large clusters risk subnet exhaustion.
• Kubenet: Pods use overlay IPs. Default 110, up to 250.
• Azure CNI Overlay: Assigns overlay addresses independent of VNet space—solves subnet exhaustion for very large clusters.

Storage

• Azure Disk: high-performance block storage, AZ-bound.
• Azure Files: shared storage with performance ceilings.
• Managed DBs (Azure Database for MySQL/Postgres) recommended for reliability.

Network Services

• Native VNet integration.
• Requires joint management of NSGs and firewall policies.

Cost Structure

• Free tier: no control plane charges.
• Standard/Premium: SLA and extended support included.
• Node costs are simply VM usage.

5. Autoscaling: HPA, CA, Karpenter

HPA (Horizontal Pod Autoscaler)

• Scales pods based on CPU, memory, or custom metrics.
• Example: spikes in traffic automatically trigger pod increases.

CA (Cluster Autoscaler)

• Adds nodes when unschedulable pods appear.
• In cloud: automatically provisions VMs, then scales down idle ones.
• On-prem: limited by physical servers, making elasticity harder.

Karpenter

• AWS-only next-gen autoscaler.
• Works at the pod scheduling level instead of node pools.
• Selects the most optimal instance type (on-demand, spot, GPU, etc.) in real time.
• Delivers cost and performance efficiency beyond traditional CA.

6. Placement Strategy for Databases, Middleware, and Monitoring

7. Cost Considerations

• On-Premises: High upfront cost for servers and storage, but lower recurring expenses over time.
• EKS: Adds control plane costs; EC2/Fargate usage varies month to month.
• AKS: Free control plane option; only VM costs apply.

8. Choosing the Right Path

• On-Premises: Best for high-performance databases, Kafka, or strict data sovereignty.
• EKS: Ideal for teams leveraging AWS services with global scaling needs.
• AKS: Strong fit for Microsoft-centric organizations seeking cost efficiency.

9. Conclusion

While Kubernetes is Kubernetes everywhere, operating strategies differ radically across environments:

• On-Premises: ultimate control and performance.
• EKS: seamless AWS integration and global scalability.
• AKS: flexible networking models with strong cost efficiency.

Layering in autoscaling solutions such as HPA, CA, and Karpenter enables highly adaptive architectures tailored to each environment.

Personal Take:
Running Kubernetes on bare-metal is the ultimate challenge. With unmatched freedom comes a flood of unpredictable events and issues—daunting, yet invaluable for an engineer’s growth. In my view, the depth of understanding gained from years of on-prem Kubernetes work is distinct. Engineers who have faced those battles carry a level of mastery that sets them apart from peers who have only operated EKS or AKS.