In OpenShift, build pods automatically have the cluster-wide TLS certificate authority bundle mounted into /etc/pki/ca-trust/extracted/pem. This CA bundle can be modified by cluster admins to include corporate/self-signed CAs.

It appears that maven does not use this TLS CA bundle, causing builds to fail.

Reported in https://bugzilla.redhat.com/show_bug.cgi?id=1826183

Original report:

Description of problem:
We're running an S2I build in OCP, and trying to pull Maven dependencies from an internal repository. The repository is exposed over HTTPS, using a certificate signed by an internal PKI. The internal PKI is listed as a trusted CA in the cluster-wide proxy settings.

However we're receiving the following error during an S2I build based on Maven:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild (PKIXValidator.java:450)
at sun.security.validator.PKIXValidator.engineValidate (PKIXValidator.java:317)
at sun.security.validator.Validator.validate (Validator.java:262)
at sun.security.ssl.X509TrustManagerImpl.validate (X509TrustManagerImpl.java:330)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted (X509TrustManagerImpl.java:237)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted (X509TrustManagerImpl.java:132)

Below is the relevant section of our BuildConfig, showing the referenced S2I container and the MAVEN_MIRROR_URL:

strategy:
type: Source
sourceStrategy:
from:
kind: ImageStreamTag
namespace: jee-images
name: 'openjdk-8-rhel8:latest'
env:

• name: MAVEN_MIRROR_URL
  value: >-
  https://artifactory-prod.tcc.etn.com/artifactory/javawebteam-maven-virtual/

Version-Release number of selected component (if applicable):

How reproducible:
Everytime

Steps to Reproduce:
1. # oc new-project maven
2. # oc import-image --from=registry.access.redhat.com/redhat-openjdk-18/openjdk18-openshift redhat-openjdk18-openshift --confirm
3. # oc new-build redhat-openjdk18-openshift~https://github.com/ayush-garg-github/maven-hello-world-jfrog.git --env='MAVEN_MIRROR_URL=https://ayush.jfrog.rhcee.support/artifactory/libs-release/'

Actual results:
~~~
...
[ERROR] Plugin org.apache.maven.plugins:maven-resources-plugin:2.6 or one of its dependencies could not be resolved: Failed to read artifact descriptor for org.apache.maven.plugins:maven-resources-plugin:jar:2.6: Could not transfer artifact org.apache.maven.plugins:maven-resources-plugin:pom:2.6 from/to mirror.default (https://ayush.jfrog.xxx.xxx/artifactory/libs-release/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target -> [Help 1]
...
Caused by: org.eclipse.aether.resolution.ArtifactResolutionException: Could not transfer artifact org.apache.maven.plugins:maven-resources-plugin:pom:2.6 from/to mirror.default (https://ayush.jfrog.xxx.xxx/artifactory/libs-release/): sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
...
~~~

Expected results:
Build successful

Additional info:
All the above commands for reproducing the issue will work in any of the clusters as the git repo and artifactory are public and also they are mine. The CA certificate which signed the "https://ayush.jfrog.rhcee.support" certificate is:

----BEGIN CERTIFICATE----
MIICwDCCAaigAwIBAgIBATANBgkqhkiG9w0BAQsFADARMQ8wDQYDVQQDEwZyb290
Q0EwHhcNMTkxMTA0MTczOTM4WhcNMjQxMTAyMTczOTM5WjARMQ8wDQYDVQQDEwZy
b290Q0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDYfopx/49CLh5f
AvGLxk5MAQYJ7pKOtVf/bFyI4WzvxXfM4cRZUNNKKuCg1nmn8jbjOXZXIqLG6HiP
YRtBgDF55aKrZPuxsmb1RKDYpk8/crXkogLZLwmPgNoAwWC23l5I3T2XCGSO5xmo
rFi2T/YoKOEzNMtbwM3hrM9z54BnefHtdQHBMXRHAOAuQGpl09SUUt7rB1wFRUev
uIzDQeyS42XVkR5e0JhQcg2fCCc5TrLqk+Mq3Y0h8+9tGgF2z1VNmLbGYwOxIT2I
/2k/kjygGGumsCGgf/y77aEr6rr1mQx3xQJm6ivQ24EPXKWUaTI13eaC/v2Ee7Rs
rnMw7a+PAgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/
MA0GCSqGSIb3DQEBCwUAA4IBAQDXTvt89RDWOd4fI2yxnGjP+qvMz/FL4ySGxh9i
aKnNlwO/VPcfSfT8kaUyt4do9SAMgZSlBhDNVCTRjGMarE/WaFSDpCBgZWzcYgOy
I6ucZ7OBGfq63pCoNxCWExV07jLknQPbaDwB/eD23fjyMU/DJbxHb1pJxZNBSa8X
lcslmH+NnfL+3ubq+SdSSGVtnmrl+wtvMeeRv3/1nzwOtMRAvg8NYAHae+XwTfTo
3EcIQ/V4HAhTctegES3/944k8Qo0tTfasea8MtN2I6l8Ij3mhB2fJZCAg+RBifqO
ZdxL33rn/Qro0GrMWebCZz5l93+2mWJt6hYDL6PLqsWvVYgW
----END CERTIFICATE----

I have tried these all workarounds as well but the build still fails.

-----------------------------------------------------------------------------------------------
1. Leverage the proxy API by configuring the user-ca-bundle ConfigMap in the openshift-config namespace.

CA certificates that should be trusted by all platform components can be defined at runtime. These certificates should be added to a ConfigMap called user-ca-bundle in the openshift-config project.

Add CA certificates to a file

Create a ConfigMap called user-ca-bundle in the openshift-config project using the contents of the file created previously

oc create configmap -n openshift-config user-ca-bundle --from-file=ca-bundle.crt=<file_location>

Patch the cluster proxy object with the name of the ConfigMap

oc patch proxies.config.openshift.io/cluster --type=merge -p '{"spec":{"trustedCA":

}}'

Additional information for providing CA certificates at installation and runtime can be found in the Configuring a custom PKI

2. Provide a new MachineConfig that will inject the certificate into the proper location for CRI-O to pick it up similar to the following

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: 50-examplecorp-ca-cert
spec:
config:
ignition:
version: 2.2.0
storage:
files:

• contents:
  source: data:text/plain;charset=utf-8;base64,LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVORENDQXh5Z0F3SUJBZ0lKQU51bkkwRDY2MmNuTUEwR0NTcUdTSWIzRFFFQkN3VUFNSUdsTVFzd0NRWUQKV1FRR0V3SlZVekVYTUJVR0ExVUVDQXdPVG05eWRHZ2dRMkZ5YjJ4cGJtRXhFREFPQmdOVkJBY01CMUpoYkdWcApBMmd4RmpBVUJnTlZCQW9NRFZKbFpDQklZWFFzSUVsdVl5NHhFekFSQmdOVkJBc01DbEpsWkNCSVlYUWdTVlF4Ckh6QVpCZ05WQkFNTUVsSmxaQ0JJWVhRZ1NWUWdVbTl2ZENCRFFURWhNQjhHQ1NxR1NJYjNEUUVKQVJZU2FXNW0KWGpDQnBURUxNQWtHQTFVRUJoTUNWVk14RnpBVkJnTlZCQWdNRGs1dmNuUm9JRU5oY205c2FXNWhNUkF3RGdZRApXUVFIREFkU1lXeGxhV2RvTVJZd0ZBWURWUVFLREExU1pXUWdTR0YwTENCSmJtTXVNUk13RVFZRFZRUUxEQXBTCkFXUWdTR0YwSUVsVU1Sc3dHUVlEVlFRRERCSlNaV1FnU0dGMElFbFVJRkp2YjNRZ1EwRXhJVEFmQmdrcWhraUcKMHcwQkNRRVdFbWx1Wm05elpXTkFjbVZrYUdGMExtTnZiVENDQVNJd0RRWUpLb1pJaHZjTkFRRUJCUUFEZ2dFUApCRENDQVFvQ2dnRUJBTFF0OU9KUWg2R0M1TFQxZzgwcU5oMHU1MEJRNHNaL3laOGFFVHh0KzVsblBWWDZNSEt6CmQvaTdsRHFUZlRjZkxMMm55VUJkMmZRRGsxQjBmeHJza2hHSUlaM2lmUDFQczRsdFRrdjhoUlNvYjNWdE5xU28KSHhrS2Z2RDJQS2pUUHhEUFdZeXJ1eTlpckxaaW9NZmZpM2kvZ0N1dDBaV3RBeU8zTVZINXFXRi9lbkt3Z1BFUwpZOXBvK1RkQ3ZSQi9SVU9iQmFNNzYxRWNyTFNNMUdxSE51ZVNmcW5obzNBakxRNmRCblBXbG82MzhabTFWZWJLCkNFTHloa0xXTVNGa0t3RG1uZTBqUTAyWTRnMDc1dkNLdkNzQ0F3RUFBYU5qTUdFd0hRWURWUjBPQkJZRUZIN1IKNXlDK1VlaElJUGV1TDhacXczUHpiZ2NaTUI4R0ExVWRJd1FZTUJhQUZIN1I0eUMrVWVoSUlQZXVMOFpxdzNQegpjZ2NaTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3RGdZRFZSMFBBUUgvQkFRREFnR0dNQTBHQ1NxR1NJYjNEUUVCCkR3VUFBNElCQVFCRE52RDJWbTlzQTVBOUFsT0pSOCtlbjVYejloWGN4SkI1cGh4Y1pROGpGb0cwNFZzaHZkMGUKTUVuVXJNY2ZGZ0laNG5qTUtUUUNNNFpGVVBBaWV5THg0ZjUySHVEb3BwM2U1SnlJTWZXK0tGY05JcEt3Q3NhawpwU29LdElVT3NVSks3cUJWWnhjckl5ZVFWMnFjWU9lWmh0UzV3QnFJd09BaEZ3bENFVDdaZTU4UUhtUzQ4c2xqCjVlVGtSaml2QWxFeHJGektjbGpDNGF4S1Fsbk92VkF6eitHbTMyVTB4UEJGNEJ5ZVBWeENKVUh3MVRzeVRtZWwKU3hORXA3eUhvWGN3bitmWG5hK3Q1SldoMWd4VVp0eTMKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
  filesystem: root
  mode: 0644
  path: /etc/pki/ca-trust/source/anchors/examplecorp-ca.crt
  -----------------------------------------------------------------------------------------------

There is an issue with the Machine Config Operator (MCO) supporting Day 2 proxy support, which describes when an existing non-proxied cluster is reconfigured to use a proxy. The MCO should apply newly configured proxy CA certificates in a ConfigMap to the RHCOS trust bundle; this is not working. As a workaround, you must manually add the proxy CA certificate to your trust bundle and then update the trust bundle:

Either use the Machine Config to copy the certificates as mentioned by Jaspreet below.

OR

$ cp /opt/registry/certs/<my_root_ca>.crt /etc/pki/ca-trust/source/anchors/
$ update-ca-trust extract
$ oc adm drain <node>
$ systemctl reboot
-----------------------------------------------------------------------------------------------