Uploaded image for project: 'Red Hat OpenShift AI Engineering'
  1. Red Hat OpenShift AI Engineering
  2. RHOAIENG-6709

[2.9.z] Jupyter notebook creation might fail when different environment variables specified

XMLWordPrintable

    • 1
    • False
    • Hide

      None

      Show
      None
    • False
    • No
    • c.f. release notes text on RHOAIENG-7070
    • Bug Fix
    • No
    • Hide
      1. Open Dashboard and go to Applications -> Enabled -> Jupyter
      2. Select any notebook and start it
      3. Then stop it
      4. Adjust section of Environment Variables by adding a custom variable for you, e.g.: custom variable named "env1" with value "value1"
      5. Start the notebook
      6. See the error message
      Show
      Open Dashboard and go to Applications -> Enabled -> Jupyter Select any notebook and start it Then stop it Adjust section of Environment Variables by adding a custom variable for you, e.g.: custom variable named "env1" with value "value1" Start the notebook See the error message
    • Dashboard - General-5
    • Important
    • Testable

      When a Jupyter notebook (workbench via Jupyter app) is created, then after it is stopped and started again with a different set of environment variables configured via Dashboard, it's not possible to create a pod for it anymore:

      Danger alert:FailedCreate
create Pod jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-0 in StatefulSet jupyter-nb-htpasswd-2dcluster-2dadmin-2duser failed error: Pod "jupyter-nb-htpasswd-2dcluster-2dadmin-2duser-0" is invalid: spec.containers[0].env[2].valueFrom: Invalid value: "": may not be specified when `value` is not empty

      Though, the added environment variable seem to be reflected:

      Relevant Notebook CR excerpt
                env:
            - name: NOTEBOOK_ARGS
              value: |-
                {redacted}
            - name: JUPYTER_IMAGE
              value: >-
                image-registry.openshift-image-registry.svc:5000/redhat-ods-applications/minimal-gpu:2024.1
            - name: env1
              value: '8888'
              valueFrom:
                configMapKeyRef:
                  key: env1
                  name: >-
                    jupyterhub-singleuser-profile-htpasswd-2dcluster-2dadmin-2duser-envs
            - name: PIP_CERT
              value: /etc/pki/tls/custom-certs/ca-bundle.crt
            - name: REQUESTS_CA_BUNDLE
              value: /etc/pki/tls/custom-certs/ca-bundle.crt
            - name: SSL_CERT_FILE
              value: /etc/pki/tls/custom-certs/ca-bundle.crt
            - name: PIPELINES_SSL_SA_CERTS
              value: /etc/pki/tls/custom-certs/ca-bundle.crt
      Referenced config map (except managed fields)
      kind: ConfigMap
apiVersion: v1
metadata:
  name: jupyterhub-singleuser-profile-htpasswd-2dcluster-2dadmin-2duser-envs
  namespace: rhods-notebooks
  uid: 05f70d39-e087-4414-aff1-4000865e826c
  resourceVersion: '5319391'
  creationTimestamp: '2024-05-02T08:17:27Z'
  labels:
    opendatahub.io/dashboard: 'true'
data:
  env1: value1

      A workaround is to delete the relevant Notebook CR and then we can start a workbench via Jupyter app with a different set of environment variables again.

        1. Screenshot from 2024-05-02 20-36-18.png
          117 kB
          Jan Stourac

            aballantyne Andrew Ballantyne
            jstourac@redhat.com Jan Stourac
            Conor O'Malley
            Jan Stourac Jan Stourac
            RHOAI Dashboard
            Votes:
            1 Vote for this issue
            Watchers:
            9 Start watching this issue

              Created:
              Updated:
              Resolved: