Uploaded image for project: 'OpenShift Logging'
  1. OpenShift Logging
  2. LOG-3527

Collector to act as rsyslog server

XMLWordPrintable

    • Icon: Epic Epic
    • Resolution: Done-Errata
    • Icon: Normal Normal
    • Logging 5.9.0
    • None
    • Log Collection
    • None
    • rsyslog-server
    • False
    • None
    • False
    • Green
    • NEW
    • To Do
    • OBSDA-552 - Vector Collector to act as rsyslog server
    • OBSDA-552Vector Collector to act as rsyslog server
    • NEW
    • 0% To Do, 0% In Progress, 100% Done
    • This feature introduces a new type of remote log receiver, the Syslog Receiver. This can be configured to expose a port over a network where external systems can send Syslog logs using compatible tools such as rsyslog.
    • Feature
    • Log Collection - Sprint 245

      Goals

      Collector can be configured to listen for connections as an rsyslog server.

      Logs from inbound connection can be normalized, filtered and forwarded using existingn features of the collector.

      Non Goals

      Once we can consume rsyslog logs, there may be need for new input filters or normalization features motivated by the structure of rsyslog records. This is not part of this Epic but is likely as future work.

      Motivation

      Requests to collect logs from outside the cluster using a common framework and log store.

      In particular the OpenStack project has this need.

      It is common for non-cluster applications to use syslog-based libraries for logging.

      Rsyslog allows  us to collect such logs with minimal intrusion into the source applications.

      Alternatives

      Require external log sources to connect to LokiStack directly.

      Don't provide a common log stores for cluster and non-cluster logs.

      Acceptance Criteria

      • Rsyslog logs received by the collector can be stored in Loki
      • Rsyslog logs received by the collector can be forwarded to other output types.

      Risk and Assumptions

      Documentation Considerations

      Open Questions

      Distributing load from external sources

      Currently  we run a collector per cluster node, which partitiions the total work of collecting from the cluster.

      Data from outside the cluster isn't naturally partitioned this way, consider load-balancing and/or starting additional collector instances to handle external traffic.

      Rsyslog round trip

      We can already forward to rsyslog, question: if we receive from rsyslog and forward to rsyslog, can we preserve the log data exactly? Not clear if this is a requirement, in fact it may be desirable to add a JSON envelope on the inbound side and preserve it on the outbound side so messages from separate sources can be separated.

      Additional Notes

            rhn-engineering-jlarriba Juan Larriba
            rhn-engineering-aconway Alan Conway
            Anping Li Anping Li
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

              Created:
              Updated:
              Resolved: