Description of problem:

During the upgrade of RHODS from version 1.30 to version 1.31, the data science pipelines operator controller pod tries to modify an immutable field in the data science pipelines operator yaml. This operation fails, and is logged as an error in the rhods operator pod logs:

time="2023-08-08T09:30:10Z" level=warning msg="Encountered error applying application data-science-pipelines-operator:  (kubeflow.error): Code 500 with message: Apply.Run : Deployment.apps \"data-science-pipelines-operator-controller-manager\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/name\":\"data-science-pipelines-operator\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable"``` 

Prerequisites (if any, like setup, operators/versions):

RHODS 1.30 installed, RHODS 1.31 build to upgrade to

Steps to Reproduce

1. Install RHODS 1.30
2. Try to upgrade to RHODS 1.31
3. Check RHODS operator pod logs for error message related to a failed oc apply

Actual results:

dspo spec is not updated as expected

Expected results:

dspo spec is updated and matches what would be present in a fresh install scenario

Reproducibility (Always/Intermittent/Only Once):

always

Build Details:

RHODS 1.31 RC build

Workaround:

Additional info: