Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-14995

Ingress operator performs spurious updates in response to API's defaulting of router deployment's router container's ports' hostPort field when using HostNetwork

XMLWordPrintable

    • Low
    • No
    • 3
    • Sprint 237, Sprint 238, Sprint 239, Sprint 240
    • 4
    • Rejected
    • False
    • Hide

      None

      Show
      None
    • Hide
      * Previously, the Ingress Operator would specify the `spec.template.spec.hostNetwork: true` on a router deployment without specifying the `spec.template.spec.containers[*].ports[*].hostPort`. This caused the API server to set a default value for each port's `hostPort` filed, which the Ingress Operator would then detect as an external update and attempt to revert it. Now, the Ingress Operator no longer incorrectly performs these updates resolving the issue. (link:https://issues.redhat.com/browse/OCPBUGS-14995[*OCPBUGS-14995*])
      Show
      * Previously, the Ingress Operator would specify the `spec.template.spec.hostNetwork: true` on a router deployment without specifying the `spec.template.spec.containers[*].ports[*].hostPort`. This caused the API server to set a default value for each port's `hostPort` filed, which the Ingress Operator would then detect as an external update and attempt to revert it. Now, the Ingress Operator no longer incorrectly performs these updates resolving the issue. (link: https://issues.redhat.com/browse/OCPBUGS-14995 [* OCPBUGS-14995 *])
    • Bug Fix
    • Done

      Description of problem

      When the ingress operator creates or updates a router deployment that specifies spec.template.spec.hostNetwork: true, the operator does not set spec.template.spec.containers[*].ports[*].hostPort. As a result, the API sets each port's hostPort field to the port's containerPort field value. The operator detects this as an external update and attempts to revert it. The operator should not update the deployment in response to API defaulting.

      Version-Release number of selected component (if applicable)

      I observed this in CI for OCP 4.14 and was able to reproduce the issue on OCP 4.11.37. The problematic code was added in https://github.com/openshift/cluster-ingress-operator/pull/694/commits/af653f9fa7368cf124e11b7ea4666bc40e601165 in OCP 4.11 to implement NE-674.

      How reproducible

      Easily.

      Steps to Reproduce

      1. Create an IngressController that specifies the "HostNetwork" endpoint publishing strategy type:

      oc create -f - <<EOF
      apiVersion: operator.openshift.io/v1
      kind: IngressController
      metadata:
        name: example-hostnetwork
        namespace: openshift-ingress-operator
      spec:
        domain: example.xyz
        endpointPublishingStrategy:
          type: HostNetwork
      EOF
      

      2. Check the ingress operator's logs:

      oc -n openshift-ingress-operator logs -c ingress-operator deployments/ingress-operator
      

      Actual results

      The ingress operator logs "updated router deployment" multiple times for the "example-hostnetwork" IngressController, such as the following:

      2023-06-15T02:11:47.229Z        INFO    operator.ingress_controller     ingress/deployment.go:131       updated router deployment       {"namespace": "openshift-ingress", "name": "router-example-hostnetwork", "diff": "  &v1.Deployment{\n  \tTypeMeta:   {},\n  \tObjectMeta: {Name: \"router-example-hostnetwork\", Namespace: \"openshift-ingress\", UID: \"d7c51022-460e-4962-8521-e00255f649c3\", ResourceVersion: \"3356177\", ...},\n  \tSpec: v1.DeploymentSpec{\n  \t\tReplicas: &2,\n  \t\tSelector: &{MatchLabels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"example-hostnetwork\"}},\n  \t\tTemplate: v1.PodTemplateSpec{\n  \t\t\tObjectMeta: {Labels: {\"ingresscontroller.operator.openshift.io/deployment-ingresscontroller\": \"example-hostnetwork\", \"ingresscontroller.operator.openshift.io/hash\": \"b7c697fd\"}, Annotations: {\"target.workload.openshift.io/management\": `{\"effect\": \"PreferredDuringScheduling\"}`, \"unsupported.do-not-use.openshift.io/override-liveness-grace-period-seconds\": \"10\"}},\n  \t\t\tSpec: v1.PodSpec{\n  \t\t\t\tVolumes: []v1.Volume{\n  \t\t\t\t\t{Name: \"default-certificate\", VolumeSource: {Secret: &{SecretName: \"router-certs-example-hostnetwork\", DefaultMode: &420}}},\n  \t\t\t\t\t{\n  \t\t\t\t\t\tName: \"service-ca-bundle\",\n  \t\t\t\t\t\tVolumeSource: v1.VolumeSource{\n  \t\t\t\t\t\t\t... // 16 identical fields\n  \t\t\t\t\t\t\tFC:        nil,\n  \t\t\t\t\t\t\tAzureFile: nil,\n  \t\t\t\t\t\t\tConfigMap: &v1.ConfigMapVolumeSource{\n  \t\t\t\t\t\t\t\tLocalObjectReference: {Name: \"service-ca-bundle\"},\n  \t\t\t\t\t\t\t\tItems:                {{Key: \"service-ca.crt\", Path: \"service-ca.crt\"}},\n- \t\t\t\t\t\t\t\tDefaultMode:          &420,\n+ \t\t\t\t\t\t\t\tDefaultMode:          nil,\n  \t\t\t\t\t\t\t\tOptional:             &false,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tVsphereVolume: nil,\n  \t\t\t\t\t\t\tQuobyte:       nil,\n  \t\t\t\t\t\t\t... // 8 identical fields\n  \t\t\t\t\t\t},\n  \t\t\t\t\t},\n  \t\t\t\t\t{\n  \t\t\t\t\t\tName: \"stats-auth\",\n  \t\t\t\t\t\tVolumeSource: v1.VolumeSource{\n  \t\t\t\t\t\t\t... // 3 identical fields\n  \t\t\t\t\t\t\tAWSElasticBlockStore: nil,\n  \t\t\t\t\t\t\tGitRepo:              nil,\n  \t\t\t\t\t\t\tSecret: &v1.SecretVolumeSource{\n  \t\t\t\t\t\t\t\tSecretName:  \"router-stats-example-hostnetwork\",\n  \t\t\t\t\t\t\t\tItems:       nil,\n- \t\t\t\t\t\t\t\tDefaultMode: &420,\n+ \t\t\t\t\t\t\t\tDefaultMode: nil,\n  \t\t\t\t\t\t\t\tOptional:    nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tNFS:   nil,\n  \t\t\t\t\t\t\tISCSI: nil,\n  \t\t\t\t\t\t\t... // 21 identical fields\n  \t\t\t\t\t\t},\n  \t\t\t\t\t},\n  \t\t\t\t\t{\n  \t\t\t\t\t\tName: \"metrics-certs\",\n  \t\t\t\t\t\tVolumeSource: v1.VolumeSource{\n  \t\t\t\t\t\t\t... // 3 identical fields\n  \t\t\t\t\t\t\tAWSElasticBlockStore: nil,\n  \t\t\t\t\t\t\tGitRepo:              nil,\n  \t\t\t\t\t\t\tSecret: &v1.SecretVolumeSource{\n  \t\t\t\t\t\t\t\tSecretName:  \"router-metrics-certs-example-hostnetwork\",\n  \t\t\t\t\t\t\t\tItems:       nil,\n- \t\t\t\t\t\t\t\tDefaultMode: &420,\n+ \t\t\t\t\t\t\t\tDefaultMode: nil,\n  \t\t\t\t\t\t\t\tOptional:    nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tNFS:   nil,\n  \t\t\t\t\t\t\tISCSI: nil,\n  \t\t\t\t\t\t\t... // 21 identical fields\n  \t\t\t\t\t\t},\n  \t\t\t\t\t},\n  \t\t\t\t},\n  \t\t\t\tInitContainers: nil,\n  \t\t\t\tContainers: []v1.Container{\n  \t\t\t\t\t{\n  \t\t\t\t\t\t... // 3 identical fields\n  \t\t\t\t\t\tArgs:       nil,\n  \t\t\t\t\t\tWorkingDir: \"\",\n  \t\t\t\t\t\tPorts: []v1.ContainerPort{\n  \t\t\t\t\t\t\t{\n  \t\t\t\t\t\t\t\tName:          \"http\",\n- \t\t\t\t\t\t\t\tHostPort:      80,\n+ \t\t\t\t\t\t\t\tHostPort:      0,\n  \t\t\t\t\t\t\t\tContainerPort: 80,\n  \t\t\t\t\t\t\t\tProtocol:      \"TCP\",\n  \t\t\t\t\t\t\t\tHostIP:        \"\",\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t{\n  \t\t\t\t\t\t\t\tName:          \"https\",\n- \t\t\t\t\t\t\t\tHostPort:      443,\n+ \t\t\t\t\t\t\t\tHostPort:      0,\n  \t\t\t\t\t\t\t\tContainerPort: 443,\n  \t\t\t\t\t\t\t\tProtocol:      \"TCP\",\n  \t\t\t\t\t\t\t\tHostIP:        \"\",\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t{\n  \t\t\t\t\t\t\t\tName:          \"metrics\",\n- \t\t\t\t\t\t\t\tHostPort:      1936,\n+ \t\t\t\t\t\t\t\tHostPort:      0,\n  \t\t\t\t\t\t\t\tContainerPort: 1936,\n  \t\t\t\t\t\t\t\tProtocol:      \"TCP\",\n  \t\t\t\t\t\t\t\tHostIP:        \"\",\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tEnvFrom:       nil,\n  \t\t\t\t\t\tEnv:           {{Name: \"DEFAULT_CERTIFICATE_DIR\", Value: \"/etc/pki/tls/private\"}, {Name: \"DEFAULT_DESTINATION_CA_PATH\", Value: \"/var/run/configmaps/service-ca/service-ca.crt\"}, {Name: \"RELOAD_INTERVAL\", Value: \"5s\"}, {Name: \"ROUTER_ALLOW_WILDCARD_ROUTES\", Value: \"false\"}, ...},\n  \t\t\t\t\t\tResources:     {Requests: {s\"cpu\": {i: {...}, s: \"100m\", Format: \"DecimalSI\"}, s\"memory\": {i: {...}, Format: \"BinarySI\"}}},\n  \t\t\t\t\t\tVolumeMounts:  {{Name: \"default-certificate\", ReadOnly: true, MountPath: \"/etc/pki/tls/private\"}, {Name: \"service-ca-bundle\", ReadOnly: true, MountPath: \"/var/run/configmaps/service-ca\"}, {Name: \"stats-auth\", ReadOnly: true, MountPath: \"/var/lib/haproxy/conf/metrics-auth\"}, {Name: \"metrics-certs\", ReadOnly: true, MountPath: \"/etc/pki/tls/metrics-certs\"}},\n  \t\t\t\t\t\tVolumeDevices: nil,\n  \t\t\t\t\t\tLivenessProbe: &v1.Probe{\n  \t\t\t\t\t\t\tProbeHandler: v1.ProbeHandler{\n  \t\t\t\t\t\t\t\tExec: nil,\n  \t\t\t\t\t\t\t\tHTTPGet: &v1.HTTPGetAction{\n  \t\t\t\t\t\t\t\t\tPath:        \"/healthz\",\n  \t\t\t\t\t\t\t\t\tPort:        {IntVal: 1936},\n  \t\t\t\t\t\t\t\t\tHost:        \"localhost\",\n- \t\t\t\t\t\t\t\t\tScheme:      \"HTTP\",\n+ \t\t\t\t\t\t\t\t\tScheme:      \"\",\n  \t\t\t\t\t\t\t\t\tHTTPHeaders: nil,\n  \t\t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t\tTCPSocket: nil,\n  \t\t\t\t\t\t\t\tGRPC:      nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tInitialDelaySeconds:           0,\n  \t\t\t\t\t\t\tTimeoutSeconds:                1,\n- \t\t\t\t\t\t\tPeriodSeconds:                 10,\n+ \t\t\t\t\t\t\tPeriodSeconds:                 0,\n- \t\t\t\t\t\t\tSuccessThreshold:              1,\n+ \t\t\t\t\t\t\tSuccessThreshold:              0,\n- \t\t\t\t\t\t\tFailureThreshold:              3,\n+ \t\t\t\t\t\t\tFailureThreshold:              0,\n  \t\t\t\t\t\t\tTerminationGracePeriodSeconds: nil,\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tReadinessProbe: &v1.Probe{\n  \t\t\t\t\t\t\tProbeHandler: v1.ProbeHandler{\n  \t\t\t\t\t\t\t\tExec: nil,\n  \t\t\t\t\t\t\t\tHTTPGet: &v1.HTTPGetAction{\n  \t\t\t\t\t\t\t\t\tPath:        \"/healthz/ready\",\n  \t\t\t\t\t\t\t\t\tPort:        {IntVal: 1936},\n  \t\t\t\t\t\t\t\t\tHost:        \"localhost\",\n- \t\t\t\t\t\t\t\t\tScheme:      \"HTTP\",\n+ \t\t\t\t\t\t\t\t\tScheme:      \"\",\n  \t\t\t\t\t\t\t\t\tHTTPHeaders: nil,\n  \t\t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t\tTCPSocket: nil,\n  \t\t\t\t\t\t\t\tGRPC:      nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tInitialDelaySeconds:           0,\n  \t\t\t\t\t\t\tTimeoutSeconds:                1,\n- \t\t\t\t\t\t\tPeriodSeconds:                 10,\n+ \t\t\t\t\t\t\tPeriodSeconds:                 0,\n- \t\t\t\t\t\t\tSuccessThreshold:              1,\n+ \t\t\t\t\t\t\tSuccessThreshold:       
            0,\n- \t\t\t\t\t\t\tFailureThreshold:              3,\n+ \t\t\t\t\t\t\tFailureThreshold:              0,\n  \t\t\t\t\t\t\tTerminationGracePeriodSeconds: nil,\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tStartupProbe: &v1.Probe{\n  \t\t\t\t\t\t\tProbeHandler: v1.ProbeHandler{\n  \t\t\t\t\t\t\t\tExec: nil,\n  \t\t\t\t\t\t\t\tHTTPGet: &v1.HTTPGetAction{\n  \t\t\t\t\t\t\t\t\tPath:        \"/healthz/ready\",\n  \t\t\t\t\t\t\t\t\tPort:        {IntVal: 1936},\n  \t\t\t\t\t\t\t\t\tHost:        \"localhost\",\n- \t\t\t\t\t\t\t\t\tScheme:      \"HTTP\",\n+ \t\t\t\t\t\t\t\t\tScheme:      \"\",\n  \t\t\t\t\t\t\t\t\tHTTPHeaders: nil,\n  \t\t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\t\tTCPSocket: nil,\n  \t\t\t\t\t\t\t\tGRPC:      nil,\n  \t\t\t\t\t\t\t},\n  \t\t\t\t\t\t\tInitialDelaySeconds:           0,\n  \t\t\t\t\t\t\tTimeoutSeconds:                1,\n  \t\t\t\t\t\t\tPeriodSeconds:                 1,\n- \t\t\t\t\t\t\tSuccessThreshold:              1,\n+ \t\t\t\t\t\t\tSuccessThreshold:              0,\n  \t\t\t\t\t\t\tFailureThreshold:              120,\n  \t\t\t\t\t\t\tTerminationGracePeriodSeconds: nil,\n  \t\t\t\t\t\t},\n  \t\t\t\t\t\tLifecycle:              nil,\n  \t\t\t\t\t\tTerminationMessagePath: \"/dev/termination-log\",\n  \t\t\t\t\t\t... // 6 identical fields\n  \t\t\t\t\t},\n  \t\t\t\t},\n  \t\t\t\tEphemeralContainers: nil,\n  \t\t\t\tRestartPolicy:       \"Always\",\n  \t\t\t\t... // 31 identical fields\n  \t\t\t},\n  \t\t},\n  \t\tStrategy:        {Type: \"RollingUpdate\", RollingUpdate: &{MaxUnavailable: &{Type: 1, StrVal: \"25%\"}, MaxSurge: &{}}},\n  \t\tMinReadySeconds: 30,\n  \t\t... // 3 identical fields\n  \t},\n  \tStatus: {ObservedGeneration: 1, Replicas: 2, UpdatedReplicas:
      2, UnavailableReplicas: 2, ...},\n  }\n"}
      

      Note the following in the diff:

                                                      Ports: []v1.ContainerPort{                                                                                                                                                                                                                                                                                                                                                               
                                                              {                                                                                                                                                                                                                                                                                                                                                                                
                                                                      Name:          \"http\",                                                                                                                                                                                                                                                                                                                                                 
      -                                                               HostPort:      80,                                                                                                                                                                                                                                                                                                                                                       
      +                                                               HostPort:      0,                                                                                                                                                                                                                                                                                                                                                        
                                                                      ContainerPort: 80,                                                                                                                                                                                                                                                                                                                                                       
                                                                      Protocol:      \"TCP\",                                                                                                                                                                                                                                                                                                                                                  
                                                                      HostIP:        \"\",                                                                                                                                                                                                                                                                                                                                                     
                                                              },                                                                                                                                                                                                                                                                                                                                                                               
                                                              {
                                                                      Name:          \"https\",
      -                                                               HostPort:      443,
      +                                                               HostPort:      0,
                                                                      ContainerPort: 443,
                                                                      Protocol:      \"TCP\",
                                                                      HostIP:        \"\",
                                                              },
                                                              {
                                                                      Name:          \"metrics\",
      -                                                               HostPort:      1936,
      +                                                               HostPort:      0,
                                                                      ContainerPort: 1936,
                                                                      Protocol:      \"TCP\",
                                                                      HostIP:        \"\",
                                                              },
                                                      },
      

      Expected results

      The operator should ignore updates by the API that only set default values. The operator should not perform these unnecessary updates to the router deployment.

            mmasters1@redhat.com Miciah Masters
            mmasters1@redhat.com Miciah Masters
            Shudi Li Shudi Li
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

              Created:
              Updated:
              Resolved: