-
Bug
-
Resolution: Obsolete
-
Major
-
None
-
None
-
False
-
False
-
NEW
-
NEW
-
Undefined
-
-
0
Based on https://github.com/prometheus-operator/kube-prometheus/issues/294, fsSpaceFillingUpCriticalThreshold was adjusted as it was in conflict with the Kubernetes Garbage Collection default values (nicely documented in https://www.openshift.com/blog/image-garbage-collection-in-openshift and https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/).
But with the default value for imageGCLowThresholdPercent, we are still able to trigger NodeFilesystemSpaceFillingUp on warning level as this is starting when 40% space is left.
Hence even with Garbage Collection running, the alert may continue to fire, even though necessary clean-up is done.
This is why fsSpaceFillingUpWarningThreshold should be adjusted to a reasonable value and if going to 20% is not possible, it would be required to talk to kubelet Engineering group to understand whether Garbage Collection default values need to be adjusted.