Uploaded image for project: 'Red Hat OpenShift AI Engineering'
  1. Red Hat OpenShift AI Engineering
  2. RHOAIENG-5067

Model server metrics broken with 400 Bad Request

XMLWordPrintable

    • 1
    • False
    • Hide

      None

      Show
      None
    • False
    • Release Notes
    • Yes
    • Hide
      Model server metrics page does not load for a model server based on the ModelMesh component

      Data Science Project names containing capital letters or spaces can cause issues on the model server metrics page for model servers based on the ModelMesh component. The metrics page might not receive data correctly, resulting in a `400 Bad Request` error and preventing the page from loading.

      Workaround:
      In OpenShift, change the display names of your Data Science Projects to meet Kubernetes resource name standards (lowercase alphanumeric characters and hyphens only).
      Show
      Model server metrics page does not load for a model server based on the ModelMesh component Data Science Project names containing capital letters or spaces can cause issues on the model server metrics page for model servers based on the ModelMesh component. The metrics page might not receive data correctly, resulting in a `400 Bad Request` error and preventing the page from loading. Workaround: In OpenShift, change the display names of your Data Science Projects to meet Kubernetes resource name standards (lowercase alphanumeric characters and hyphens only).
    • Known Issue
    • Done
    • No
    • 1
    • Dashboard - General-3
    • Important
    • Testable

      While individual model metrics seem to work, model server metrics are non-functional.  The user only sees a spinning request.  In the network requests, I can see failed requests to 

      POST https://rhods-dashboard-<...>/api/prometheus/serving

      with query json data:

      {
  "query": "namespace=Text+to+Image&query=round%28sum%28increase%28modelmesh_api_request_milliseconds_count%7Bnamespace%3D%22text-to-image%22%2Cpod%3D%7E%22modelmesh-serving-custom-model-server-.*%22%7D%5B300s%5D%29%29%29&start=1711975518.69&end=1712061918.69&step=288"
}

      and the response 

       

      {
    "statusCode": 400,
    "code": "400",
    "error": "Bad Request",
    "message": "Prometheus call error"
}

        1. Screenshot from 2024-04-02 11-27-54.png
          84 kB
          Chris Chase
        2. Screenshot from 2024-04-02 11-21-17.png
          201 kB
          Chris Chase
        3. Screenshot from 2024-04-02 09-05-20.png
          252 kB
          Chris Chase
        4. image-2024-04-10-12-15-01-405.png
          34 kB
          Conor O'Malley
        5. image-2024-04-10-11-02-18-973.png
          61 kB
          Conor O'Malley
        6. image-2024-04-10-10-58-46-255.png
          50 kB
          Conor O'Malley

            mikejturley Mike Turley
            cchase@redhat.com Chris Chase
            Conor O'Malley
            RHOAI Dashboard
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: