Uploaded image for project: 'OpenShift Bugs'
  1. OpenShift Bugs
  2. OCPBUGS-23422

oc-mirror will hit 401 code after hang a while

XMLWordPrintable

    • Important
    • No
    • CLID Sprint 247
    • 1
    • False
    • Hide

      None

      Show
      None

      Description of problem:

      $ oc-mirror --config=./imageset.yaml file://<local_directory>
      
      This commands at the begining the response code is 200 okay , when the command hang for a while , then it throws below response
      ~~~
      unauthorized: Access to the requested resource is not authorized
      ~~~
      
      - This mirror attempts fail with single as well as multiple operators.
      - The issue arises when a network device between the RedHat.io CDN and the server running oc-mirror issues a connection reset, which is not handled properly by the oc-mirror tool.
      

      Version-Release number of selected component (if applicable):

       

      How reproducible:

       

      Steps to Reproduce:

      $ oc-mirror version
      Logging to .oc-mirror.log
      WARNING: This version information is deprecated and will be replaced with the output from --short. Use --output=yaml|json to get the full version.
      Client Version: version.Info{Major:"", Minor:"", GitVersion:"4.14.0-202310201027.p0.g68cf97e.assembly.stream-68cf97e", GitCommit:"68cf97ec715ad2d78fb2bac411a118709c191719", GitTreeState:"clean", BuildDate:"2023-10-20T23:48:14Z", GoVersion:"go1.20.10 X:strictfipsruntime", Compiler:"gc", Platform:"linux/amd64"}
      __________________________________________________________________
      
      ]$ cat imageset.yaml 
      kind: ImageSetConfiguration
      apiVersion: mirror.openshift.io/v1alpha2
      archiveSize: 1
      storageConfig:
        local:
          path: /var/www/html/pub/openshift
      mirror:
        platform:
          channels:
          - name: stable-4.12                                             
            type: ocp
            minVersion: '4.12.33'
            maxVersion: '4.12.33'
          graph: true
        operators:
        - catalog: registry.redhat.io/redhat/certified-operator-index:v4.12 
          packages:
          - name: gitlab-operator-kubernetes 
            channels:
            - name: stable
          - name: gitlab-runner-operator
            channels:
            - name: stable
          - name: dell-csm-operator-certified
            channels:
            - name: stable
          - name: splunk-operator
            channels:
            - name: stable
        - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.12
          packages:
          - name: advanced-cluster-management                                  
            channels:
            - name: release-2.8             
          - name: compliance-operator
            channels:
            - name: stable
          - name: ansible-automation-platform-operator                                
            channels:
            - name:  stable-2.4-cluster-scoped
          - name: container-security-operator                                  
            channels:
            - name: stable-3.9
          - name: file-integrity-operator                                  
            channels:
            - name: stable 
          - name: kubernetes-nmstate-operator                                  
            channels:
            - name: stable 
          - name: kubevirt-hyperconverged                                
            channels:
            - name: stable
          - name: local-storage-operator                                
            channels:
            - name: stable 
          - name: mtv-operator                                 
            channels:
            - name: release-v2.5 
          - name: odf-operator                                
            channels:
            - name: stable-4.12
          - name: openshift-gitops-operator
            channels:
            - name: latest
          - name: openshift-pipelines-operator-rh
            channels:
            - name: latest
          - name: quay-bridge-operator
            channels:
            - name: stable-3.9
          - name: quay-operator
            channels:
            - name: stable-3.9
          - name: rhacs-operator
            channels:
            - name: stable
          - name: rhsso-operator
            channels:
            - name: stable
          - name: multicluster-engine
            channels:
            - name: stable-2.3
        additionalImages:
        - name: registry.redhat.io/ubi8/ubi:latest                        
        - name: registry.redhat.io/rhel8/support-tools:latest
        - name: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
        - name: registry.k8s.io/sig-storage/csi-resizer:v1.8.0
        - name: registry.k8s.io/sig-storage/csi-attacher:v4.3.0
        - name: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
        - name: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.2
        - name: docker.io/dellemc/csi-metadata-retriever:v1.4.0
        helm: {}
      __________________________________________________________________
      
      ]$ cat -n oc-mirror.log
       
      114  level=debug msg=fetch response received digest=sha256:7323084df042bd67b2bbdb22f8518b3a1c890091c74cf229082ba83763516e50 mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip response.header.accept-ranges=bytes response.header.age=85688 response.header.connection=keep-alive response.header.content-length=79171494 response.header.content-type=binary/octet-stream response.header.date=Tue, 14 Nov 2023 12:28:16 GMT response.header.etag="316809a4a659d5fa75e0819576e2641f-1" response.header.last-modified=Tue, 14 Nov 2023 12:20:06 GMT response.header.server=AmazonS3 response.header.via=1.1 03ffca0f67e3596b9a0c92342fe91598.cloudfront.net (CloudFront) response.header.x-amz-cf-id=KZQxuqXp8ZnZxxUCvR-p0OUv_R6jAs-E9tICHxzF0IrA1hECf3z9EQ== response.header.x-amz-cf-pop=FRA56-P2 response.header.x-amz-replication-status=COMPLETED response.header.x-amz-server-side-encryption=AES256 response.header.x-amz-version-id=RY4w7Lwa_8gGyISLVNGacM90Oi6i7c5E response.header.x-cache=Hit from cloudfront response.status=200 OK size=79171494 url=https://registry.redhat.io/v2/redhat/certified-operator-index/blobs/sha256:7323084df042bd67b2bbdb22f8518b3a1c890091c74cf229082ba83763516e50
       
      
      8040  uploading: file://gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator sha256:425eec93a9873db227da81b4bf8eaa70072d79f01816ff5248cd43c75aeab3ca 17.14MiB
      8041  uploading: file://gitlab-org/gl-openshift/gitlab-runner-operator/gitlab-runner-operator sha256:ddef8910f97171c51224af310fbecbca2a4c48dc8f55abf902b381a64f5803c2 39.81MiB
      8042  error: unable to push registry.redhat.io/rhel8/postgresql-12: failed to retrieve blob sha256:54fc5ed135c58d0816d33fb576e52e3b10934262c4ae7e40ebfcb4f33e25ba6f: unauthorized: Access to the requested resource is not authorized
      8043  error: unable to push registry.redhat.io/rhel8/postgresql-12: failed to retrieve blob sha256:1513534557a903224a0b9766117b731d1ef118a1da58d647e9825a888f160748: unauthorized: Access to the requested resource is not authorized
      <<snip>>
      
      8368  error: unable to open source layer sha256:88d1a334b20b9cc4b0318b053e5fa5183244951aa184158ce0e9bb0a558b32dc to copy to file://openshift4/ose-kube-rbac-proxy: unauthorized: Access to the requested resource is not authorized
      8369  error: unable to open source layer sha256:d274f94bea4dbc172918e470e054d0ae98ab8804d1b39308f75d030227925591 to copy to file://openshift4/ose-kube-rbac-proxy: unauthorized: Access to the requested resource is not authorized
      8370  error: unable to open source layer sha256:03bf2f9ff79ce68fdf647999d3c96dd98a59121fae75dd2c1dcce34e3e159eeb to copy to file://openshift4/ose-kube-rbac-proxy: Get "https://registry.redhat.io/v2/openshift4/ose-kube-rbac-proxy/blobs/sha256:03bf2f9ff79ce68fdf647999d3c96dd98a59121fae75dd2c1dcce34e3e159eeb": read tcp xx.xxx.xx.xx:60990->xx.xxx.xx.xx:443: read: connection reset by peer
      8371  error: unable to open source layer sha256:b161e1ed868cfcfdd85462339ac69bb780f20ce7f98f0fcddb69208fc62155f6 to copy to file://openshift4/ose-kube-rbac-proxy: Get "https://registry.redhat.io/v2/openshift4/ose-kube-rbac-proxy/blobs/sha256:b161e1ed868cfcfdd85462339ac69bb780f20ce7f98f0fcddb69208fc62155f6": read tcp xx.xxx.xx.xx:32772->xx.xxx.xx.xx:443: read: connection reset by peer
      <<snip>>
      
      8579  error: unable to open source layer sha256:d506b5c4a8b832553099e1af0c816d4371fdc856efaebdb689dd52e49a35709f to copy to file://migration-toolkit-virtualization/mtv-console-plugin-rhel9: unauthorized: Access to the requested resource is not authorized
      8580  error: unable to open source layer sha256:c90ba8fd17e77bdfa4c399f026a5606cefc557d926df7cfa90eda5096b9d111c to copy to file://migration-toolkit-virtualization/mtv-console-plugin-rhel9: unauthorized: Access to the requested resource is not authorized
      8581  info: Mirroring completed in 7m18.4s (6.659MB/s)
      8582  error: one or more errors occurred while uploading images
      
      

      Actual results:

      oc-mirror doesn't handle network connection properly and throws
      ~~~
      unauthorized: Access to the requested resource is not authorized
      ~~~ 

      Expected results:

      oc-mirror should handle network connections properly.

      Additional info:

      Similar bug https://issues.redhat.com/browse/OCPBUGS-20137 was raised earlier and fixed in oc-mirror 4.14 version.

            luzuccar@redhat.com Luigi Mario Zuccarelli
            rhn-support-duge Dushyant Uge
            ying zhou ying zhou
            Dushyant Uge
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: