Kubernetes Cluster design

Here are some pros and cons of having Single Cluster and Multiple kubernetes clusters. 

1. Single Cluster

Pros: 

1. If you have only one Kubernetes cluster, you need to have only one copy of all the resources that are needed to run and manage a Kubernetes cluster. But this also includes other cluster-wide services, such as load balancers, Ingress controllers, authentication, logging, and monitoring. If you have only a single cluster, you can reuse these services for all your workloads, and you don't need to have multiple copies of them for multiple clusters.
2. As a consequence of the above point, fewer clusters are usually cheaper, because the resource overhead with larger numbers of clusters costs money
3. Administrating a single cluster is easier than administrating many clusters.
   

Cons: 

1. If you have only one cluster and if that cluster breaks, then all your workloads are down. 
2. A single incident like this can produce major damage across all your workloads if you have only a single shared cluster.
3. If multiple apps run in the same Kubernetes cluster, this means that these apps share the hardware, network, and operating system on the nodes of the cluster. This may be an issue from a security point of view. 
4. If you use a single cluster for all your workload, this cluster will probably be rather large (in terms of nodes and Pods). The reason is that larger clusters put a higher strain on the Kubernetes control plane, which requires careful planning to keep the cluster functional and efficient.
   

2. Many small single-use clusters

Pros: 

With this approach, you use a separate Kubernetes cluster for every deployment unit

1. If a cluster breaks, the damage is limited to only the workloads that run on this cluster . All the other workloads are unaffected.
   
2. The workloads running in the individual clusters don't share any resources, such as CPU, memory, the operating system, network, or other services.
   
3. This provides strong isolation between unrelated applications, which may be a big plus for the security of these applications.
   
4. If every cluster runs only a small set of workload, then fewer people need to have access to this cluster.

Cons: 

1. Each Kubernetes cluster requires a set of management resources, such as the master nodes, control plane components, monitoring and logging solutions.
   
2. If you have many small clusters, you have to sacrifice a higher percentage of the total resources for these management functions.

3. Cluster per environment

With this approach, you have a separate cluster for each environment:

Pros: 

1. In general, this approach isolates all the environments from each other but, in practice, this especially matters for the prod environment.
   
2. The production versions of your app are now not affected by whatever happens in any of the other clusters and application environments.
3. Nobody really needs to do work on the prod cluster, so you can make access to it very restrictive.
   
4. You can go as far as not granting access to the prod cluster to any humans at all — the deployments to this cluster can be made through an automated CI/CD tool.
   
5. This should greatly decrease the risk of human error in the prod cluster, which is where it matters most
   

Cons:

1. The main disadvantage of this approach is the missing hardware and resource isolation between apps.
   
2. Unrelated apps share cluster resources, such as the operating system kernel, CPU, memory, and several other services.
3. If an app has special requirements, then these requirements must be satisfied in all clusters.