Kubernetes Requests and Limits explained

During my work as IT freelancer I frequently I encounter common misunderstandings about the kubernetes requests and limits which lead to unstable systems and wasted resources.

So this blogpost explains what the meaning of the different values is and how to set them properly.

TLDR;

Set your CPU request to 200m, set your CPU limit to at least 4000m, 8000m or 16000m
Set your Memory request to 120-130% of your application needs and set your memory limit to the same as the request

Setting resource requests and limits

A pod can contain multiple containers and you can set the requests and limits for every container. In total the sum of resources requested by all the containers of a pod determine if it can be scheduled on a node.

https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/

Requests

In Kubernetes you can request a certain amount of resources. The requested resources are guaranteed to the container to be supplied and the pod will not be scheduled if the resources of its containers are not available in the cluster. Also no other container can take or share these resources.

Overprovisioning

When the the pods which are placed on a node in total are allowed to use more resources than the node can supply, then this is called overprovisioning. The containers then share the resources and compete with each other for the resources. How this works in detail and how the consequences are different for CPU and memory resources is explained below.

Limits

To allow for overprovisioning kubernetes has the concept of limits. Limits are a hard upper bound. A container can never take more resources (CPU / Memory) than its limit. The resources between the amount guaranteed by the requests and the upper bound set by the limits are shared resources (overprovisioning). Multiple containers can compete for these resources. There is a catch though besides the competition with other pods. When the memory limit is set higher than the memory request and the node - for whatever reason - gets under memory pressure. It starts terminating containers which are over their requests.

CPU

Overprovisioning CPU is fairly safe and easy. All CPU which is not bound by a request is available as shared CPU and if all containers are running very CPU hungry programs, then every container gets a piece of the shared CPU in the same ratio as their CPU requests relate to each other.

Example (note that the values must be strings, so lone numbers are quoted):

We have 3 containers

# Container A
cpu requests: 200m
cpu limits: "4" (or 4000m)

# Container B
cpu requests: 200m
cpu limits: "4" (or 4000m)

# Container C
cpu requests: 400m
cpu limits: "4" (or 4000m)

Now imagine we have a node with 4 CPUs (or 4000m (4000 millicpus)). 800m are fixed assignedto the containers (200 + 200 + 400), this leaves us with 3200m which the containers can compete against with each other. If no other container is requiring CPU over their requests, then one container is allowed to take all of the CPU. (e.g. Container A can take 200m + 3200m = 3400m, or Container C can take 400m + 3200m = 3600m).

If other containers also are running on full load (e.g. multiple machine learning models training on the same node). Then the Pods are given CPU time based on their CPU requests relative to the total CPU requests of all pods on the node. So a pod with more CPU requests (compared to others) also gets more of the shared CPU.

Memory

Overprovisioning Memory is unsafe and must be considered thoroughly. Kubernetes handles memory resources differently than CPU resources because memory cannot be overcommitted in the same way CPU can. Memory is not a compressible resource, which means that if a container tries to use more memory than its limit, it cannot be throttled like CPU can; instead, the container is terminated or killed by the Kubernetes system (or more specifically, by the underlying Linux kernel’s Out-Of-Memory (OOM) Killer) if it exceeds its memory limit and additional memory cannot be reclaimed.

If the kubernetes nodes comes under memory pressure it can decide at any point to terminate any pods which use more memory than their requests.

Conclusion

**CPU:**Most data transferring processes use very little CPU as they only use the CPU to move data from one location (disk, memory, network) to another location which takes a very long time (compared to a cpu cycle), so the CPU is quick and then only waits for the transfer. If CPU requests are set high (e.g. 1000m (== 1 CPU) this is much wasted CPU which never gets used. Additionally if a node has only a few CPUs but plenty of memory this can prevent the pod getting scheduled on a node, so also memory can get wasted additionally. Unless you have certain usecases which rely heavily on calculations (e.g. machinelearning model training, optimization calculations, etc.) you probably should set a relatively low amount of requested CPU and a high CPU limit (usually I set it to the maximum cores available).

Memory: I recommend usually to set the memory limit equal to the memory request for applications with predictable memory load. Otherwise this leads to unpredictable random crashes and general instability (the pod can get killed any time). While there is a certain resilience expected from cloud applications I seldomly have encountered applications which can handle a random killing. If you are running databases or long running, expensive to restart, tasks (machine learning) this is even more important. Certain types of application (Java) don’t necessarily free unused memory at once. They have a upper heap limit (Xmx) which they frequently reach/get near to and then free memory in bulk via garbage collection. If the heap (Xmx) is higher than the request, this means they are very frequently inside the “kill-zone”.

Then is there any situation where it makes sense to set the memory limit higher than the requests?

If the application doesn’t have a base load, but unpredictable short spike memory load or just needs much memory for startup this might make sense.

Imagine a machine learning application which does some costly data transformation before starting a training job. This might require a spike at the beginning and then a lot less memory afterwards. Or imagine a web application, which seldomly gets large uploads which need to get loaded into memory. Or you have many short lived, restartable, tasks with unpredictable memory usage and you want to even out the memory usage over e.g. 100 tasks.

Practical Recommendations

I recommend the following

Regularly check your monitoring, or if you don’t have monitoring use “kubectl top pod” to get an idea of the real values
Set your CPU request to 70% of the permanently used CPU (CPU requests should be always slightly too low)
(Almost) Never set your CPU limit under 200m. This will lead to CPU starvation of your process during busy time on the node. I have seen applications being slowed to being unusable by too low requests (under 200m)
Set your CPU limit to the maximum amount of CPU cores available (this gives nicer metrics in monitoring. If you don’t care about metrics you can always set it to 99, no matter how many cores are available. Anyway you never get more CPU than what’s physical available.)
Set your Memory request to 130% of the permanently used Memory (memory requests should be always 30% too high)
Set your Memory limit to the same value like the memory request

Concrete Examples

Imagine we have an 8 Core 64GB Kubernetes Node.

Business Application (Java), needs 3GB of heap.:
- CPU request: 200m**,** CPU limit: 8000m
- Memory request: 3*1,3 = 3,9 -> let’s give it 4GB, Memory limit: same as request -> 4GB
Machine Learning Model Training, needs 15GB of memory as seen in monitoring statistics and permanently uses 2500m CPU during training:
- CPUrequest: 2500m * 0,7 = 1750m**,** CPU limit: 8000m
- Memory request: 15*1,3 = 19,5 -> let’s give it 20GB, Memory limit: same as request -> 20GB

With higher memory volumes (e.g. the 20GB of the machinelearning application) you can get away with giving it less oversizing (e.g. 115% instead of 130%). Just watch your monitoring and adjust as needed.

One pod can use all the excess CPU by its own

These pods have memory limits set higher than their requests and are open season for the OOM