-
Sub-task
-
Resolution: Done
-
Critical
-
None
-
None
-
None
-
False
-
False
-
OCPSTRAT-343 - Onboarding New Providers/Platforms (Phase 2)
{}USER STORY:{}
As a Partner running OPCT, I would like to make sure what tests failed are directly related with my certification environment, getting more precise information to immediate fix it without contacting Red Hat / before submitting the results.
{}DESCRIPTION:{}
The OPCT should provide more accurate feedback to the partner when he is reviewing the results. There are a big effort to isolate those items, but we are looking for data sources to do it in an automated fashion.
The Sippy is a tool used to extract insights from CI Jobs/tests. We can consume the API to query current failures directly from the CLI, reporting flakes from the failed tests.
This is a PoC querying the Sippy API, checking the flake count and percentage for an specific test from all jobs:
$ cat ~/opct/partner/partner-20221014/processed/tests_*_provider_failures-3-filter2_baseline.txt | ./ocp-ci-query-flakes Flakes Perc TestName 1 0.139% [sig-api-machinery] CustomResourcePublishOpenAPI [Privileged:ClusterAdmin] works for multiple CRDs of same group and version but different kinds [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] 2 0.277% [sig-api-machinery] ResourceQuota should create a ResourceQuota and capture the life of a secret. [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] 101 10.632% [sig-arch][bz-DNS][Late] Alerts alert/KubePodNotReady should not be at or above pending in ns/openshift-dns [Suite:openshift/conformance/parallel] 67 7.053% [sig-arch][bz-Routing][Late] Alerts alert/KubePodNotReady should not be at or above pending in ns/openshift-ingress [Suite:openshift/conformance/parallel] 2 0.388% [sig-imageregistry] Image registry should redirect on blob pull [Suite:openshift/conformance/parallel] 0 0.000% [sig-network-edge][Feature:Idling] Unidling should handle many TCP connections by possibly dropping those over a certain bound [Serial] [Skipped:Network/OVNKubernetes] [Suite:openshift/conformance/serial] 32 4.885% [sig-network][Feature:EgressFirewall] egressFirewall should have no impact outside its namespace [Suite:openshift/conformance/parallel] 11 2.423% [sig-network][Feature:EgressFirewall] when using openshift-sdn should ensure egressnetworkpolicy is created [Suite:openshift/conformance/parallel] 0 0.000% [sig-storage] CSI Volumes [Driver: csi-hostpath] [Testpattern: Dynamic PV (default fs)] provisioning should provision storage with pvc data source [Suite:openshift/conformance/parallel] [Suite:k8s]
The source is available here: https://github.com/mtulio/mtulio.labs/commit/264a0bdf0f442a5df4afe0975e75aa3032fc42d0
Goal:
- check if we can use Sippy API to query the CI data for failed jobs.