Loading...

XML

Word

Printable

Type: Bug
Resolution: Done
Priority: Major
Fix Version/s: Logging 5.5.6
Affects Version/s: Logging 5.5.4
Component/s: Log Collection
Labels:

Blocked:
False
Blocked Reason:
None
Ready:
False
Backlogs:

Logging
Docs QE Status:
NEW
QE Status:
VERIFIED
Release Note Text:
Before this update, records written to Elasticsearch would fail if multiple label keys had the same prefix and some keys included included dots. With this update, underscores replace dots in label keys, resolving the issue.
Risk Impact Level:
Medium
Steps to Reproduce:
Hide

1) Create a new project

$ oc create project adri

2) Label the namespace

$ oc label namespace adri "app=config-server" $ oc label namespace adri "app.kubernetes.io/instance=config-server-uat" $ oc label namespace adri "app.test=test"

3) Deploy an application

$ oc new-app rails-postgresql-example

4) Check collector logs

2022-11-18 11:25:08 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil tag="kubernetes.var.log.pods.adri_rails-postgresql-example-1-build_bc8495af-acd6-4da3-bede-3d4b47ff99fb.sti-build.0.log" time=2022-11-18 11:25:08.027204286 +0000 record={"@timestamp"=>"2022-11-18T11:25:08.027204286+00:00", "message"=>"Push successful", "docker"=>{"container_id"=>"f822a2fada8223e39e4a6d26dca41ec21e52c10ce39a3146ec9a6e7b68b860ae"}, "kubernetes"=>{"container_name"=>"sti-build", "namespace_name"=>"adri", "pod_name"=>"rails-postgresql-example-1-build", "container_image"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e", "container_image_id"=>"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e", "pod_id"=>"bc8495af-acd6-4da3-bede-3d4b47ff99fb", "pod_ip"=>"10.129.2.80", "host"=>"worker-2.adricluster.lab.psi.pnq2.redhat.com", "master_url"=>"https://kubernetes.default.svc", "namespace_id"=>"b2c972e1-05ea-412d-a2e9-8db4bb800c94", "namespace_labels"=>{"app"=>"config-server", "app.kubernetes.io/instance"=>"config-server-uat", "app.test"=>"test", "kubernetes.io/metadata.name"=>"adri", "pod-security.kubernetes.io/audit"=>"restricted", "pod-security.kubernetes.io/audit-version"=>"v1.24", "pod-security.kubernetes.io/warn"=>"restricted", "pod-security.kubernetes.io/warn-version"=>"v1.24"}, "flat_labels"=>["openshift.io/build.name=rails-postgresql-example-1"]}, "level"=>"unknown", "hostname"=>"worker-2.adricluster.lab.psi.pnq2.redhat.com", "pipeline_metadata"=>{"collector"=>{"ipaddr4"=>"10.74.215.95", "inputname"=>"fluent-plugin-systemd", "name"=>"fluentd", "received_at"=>"2022-11-18T11:25:08.028075+00:00", "version"=>"1.14.6 1.6.0"}}, "openshift"=>{"sequence"=>2598}, "viaq_msg_id"=>"NGY0YzNlMmMtNWNjZS00ODZlLWJkNWItYWQ3MzE0OWNkYmRl", "log_type"=>"application", "viaq_index_name"=>"app-write"}
Show
1) Create a new project $ oc create project adri 2) Label the namespace $ oc label namespace adri "app=config-server" $ oc label namespace adri "app.kubernetes.io/instance=config-server-uat" $ oc label namespace adri "app.test=test" 3) Deploy an application $ oc new -app rails-postgresql-example 4) Check collector logs 2022-11-18 11:25:08 +0000 [warn]: dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error= "400 - Rejected by Elasticsearch" location=nil tag= "kubernetes. var .log.pods.adri_rails-postgresql-example-1-build_bc8495af-acd6-4da3-bede-3d4b47ff99fb.sti-build.0.log" time=2022-11-18 11:25:08.027204286 +0000 record={ "@timestamp" => "2022-11-18T11:25:08.027204286+00:00" , "message" => "Push successful" , "docker" =>{ "container_id" => "f822a2fada8223e39e4a6d26dca41ec21e52c10ce39a3146ec9a6e7b68b860ae" }, "kubernetes" =>{ "container_name" => "sti-build" , "namespace_name" => "adri" , "pod_name" => "rails-postgresql-example-1-build" , "container_image" => "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e" , "container_image_id" => "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c043c4d68753120a32044f476a8e1977101139cbab3694761ef523da5be8fb0e" , "pod_id" => "bc8495af-acd6-4da3-bede-3d4b47ff99fb" , "pod_ip" => "10.129.2.80" , "host" => "worker-2.adricluster.lab.psi.pnq2.redhat.com" , "master_url" => "https: //kubernetes. default .svc" , "namespace_id" => "b2c972e1-05ea-412d-a2e9-8db4bb800c94" , "namespace_labels" =>{ "app" => "config-server" , "app.kubernetes.io/instance" => "config-server-uat" , "app.test" => "test" , "kubernetes.io/metadata.name" => "adri" , "pod-security.kubernetes.io/audit" => "restricted" , "pod-security.kubernetes.io/audit-version" => "v1.24" , "pod-security.kubernetes.io/warn" => "restricted" , "pod-security.kubernetes.io/warn-version" => "v1.24" }, "flat_labels" =>[ "openshift.io/build.name=rails-postgresql-example-1" ]}, "level" => "unknown" , "hostname" => "worker-2.adricluster.lab.psi.pnq2.redhat.com" , "pipeline_metadata" =>{ "collector" =>{ "ipaddr4" => "10.74.215.95" , "inputname" => "fluent-plugin-systemd" , "name" => "fluentd" , "received_at" => "2022-11-18T11:25:08.028075+00:00" , "version" => "1.14.6 1.6.0" }}, "openshift" =>{ "sequence" =>2598}, "viaq_msg_id" => "NGY0YzNlMmMtNWNjZS00ODZlLWJkNWItYWQ3MzE0OWNkYmRl" , "log_type" => "application" , "viaq_index_name" => "app-write" }
Intelligence Requested:
Market:

Sprint:
Log Collection - Sprint 228, Log Collection - Sprint 229, Log Collection - Sprint 230

SFDC Cases Links:
SFDC Cases Counter:

Description of problem:

-RHOL VERSION 5.5.4

ElasticsearchError error="400 - Rejected by Elasticsearch" appeared in collector pods for application logs when an application namespace has some labels, the main point is that it is not happening with all labels, it seems to be specific with some labels.

The behavior of this issue is the same to--> https://issues.redhat.com/browse/LOG-2972