Skip to content

Receiving Syslog Events

LogZilla’s Syslog-ng Configuration

LogZilla supports customization of its syslog-ng configuration. Although it is available, it is not recommended due to its complexity and the complications it may introduce to the LogZilla installation.

There are two ways to customize the syslog-ng configuration:

Editing /etc/logzilla/syslog-ng/config.yaml

This is a yaml file, and it is used to generate main config for the syslog-ng container.

It contains some generic settings, and list of sources and destinations. Many of the values are automatically generated by LogZilla (often based on the logzilla config settings) and should not be modified, but there are some parts that can be customized.

The only generic setting that can be customized is extra_log_rules. This contains a string, that will be put in the log section of the main syslog config, between list of sources and list of destinations. This can be used to add some extra filters or rewrites - in conjunction with adding extra files (defining these filter or rewrites) in /etc/logzilla/syslog-ng/conf.d directory.

Destinations

Only two types of destination are supported, first is logzilla, which is dedicated for sending events to LogZilla, and probably you should not change or add anything of this type.

The second type is file, which can be used for writing events to a file, and you can use it for your own purposes, in particular with some templates and filters defined in /etc/logzilla/syslog-ng/conf.d directory.

If you just want to dump all events to a file, you can use configuration options as described in Debugging Event Reception.

For each destination you should define following fields: - name - name of the destination, this is used in the main config and should be unique for each destination - enabled - boolean value, if set to True then this destination will be used, otherwise it will be ignored - type - type of the destination, in our case file - path - path to the file, remember this is path inside container, so usually you'd want to use something in /var/log/logzilla directory as it is mounted from the host - template - template defining how to format each event, usually this requires defining your own template in some file in the /etc/logzilla/syslog-ng/conf.d - remember that when defining own templates, they should be named with t_logzilla_ prefix, which is automatically added to the name defined in this field.

You can use also one of the predefined templates: - json - formats event as JSON object - debug_tsv - dump only timestamp, name of the source and original message in tab-separated format - pci_tsv - dump only original message

Sources

This can be a bit more useful for customization, as you can define your own ports and dedicate it to particular sources. With a special directive you can assign some tags to events coming from this source, and then use this tag to optimize parsing of these events in rules and apps.

Standard ports are provided by default configuration which should not be changed - these are:

  • bsd - tcp on port 514 (or other defined by config SYSLOG_BSD_TCP_PORT), for BSD-style syslog messages
  • bsd_udp - udp on port 514 (or other defined by config SYSLOG_BSD_UDP_PORT), for BSD-style syslog messages using UDP
  • rfc5424 - tcp on port 601 (or other defined by config SYSLOG_RFC5424_PORT), for RFC5424-style syslog messages
  • json - tcp on port 515 (or other defind by config SYSLOG_JSON_PORT), for sending raw JSON messages (newline separated) with a TCP connection.
  • tls - tcp on port 6514 (or other defined by config SYSLOG_TLS_PORT), it's the same as rfc5424, but with TLS encryption
  • raw - tcp on port 516 (or other defined by config SYSLOG_RAW_PORT), for sources not complying to the syslog standard, here no parsing is performed and the raw message is sent to LogZilla as is.
  • raw_udp - udp on port 516 (or other defined by config SYSLOG_RAW_UDP_PORT), same as raw - without parsing, message is sent to LogZilla as is.

User can define any extra ports by adding new entries to the sources array, each definition should contain following fields: - name - name of the source, this is used in the main config and should be unique for each destination - enabled - boolean value, if set to True then this source will be used, otherwise it will be ignored - type - type of the source, two values are supported, network and syslog, see syslog-ng documentation for more details on these - port - port number to listen on - transport - transport protocol, tcp or udp (tcp is default), also tls can be specified for TLS-encrypted TCP connection - tls_cert_file - path to the certificate file, this is used only when transport is set to tls - tls_key_file - path to the key file, this is used only when transport is set to tls - flags - list of flags which are passed to syslog-ng, see syslog-ng documentation for more details - program_override - here you can specify name that will be set to the program field of the event - extra_fields - this is dictionary (object) with key-value pairs that will be added to the event's extra_fields dictionary - source_tag - this is a string, which can be used to specify a tag that will be added to all events (in the extra_fields._source_tag) that are received from this source. This can be used to optimize parsing of these events in some apps - as for the version 6.32 this is used in the vmware app.

Defining source_tag in sources

Starting from version 6.32, you can optimize parsing of these events by enabling dedicated source for some apps. If any of the LUA rules defines SOURCE_FILTER variable in its body, and there's a source with source_tag set to that value, then this rule will be applied only to events from this source.

For example - if you enable SYSLOG_VMWARE_PORT by setting it to any port number greater then 0, it will be automatically added to the list of sources with a source_tag set to vmware. Then only events from this port will be parsed by vmware app.

Please note, that filtering works only if there's a source with source_tag with a corresponding value - so if you install vmware app, but you don't have a dedicated source for vmware events, then all events will be parsed by the app (which can results in significant performance degradation).

If you need some custom dedicated sources (e.g. for your custom rules/apps), then remember to add your source tag to the DEDICATED_SOURCES configuration option (with logzilla config DEDICATED_SOURCES command line), so the parser will know that only events with this tag should be parsed by the app. For the VMWare app this setting is extended automatically with "vmware" if you enable SYSLOG_VMWARE_PORT.

Adding extra files in /etc/logzilla/syslog-ng/conf.d directory

For more complex cases you can add any *.conf files in this directory, and they will be included in the main config. This can be used to add some extra syslog-ng sources, destinations, filters, or rewrite rules. To accomplish this:

  1. Create a xxx.conf file (where xxx is the desired name) in the /etc/logzilla/syslog-ng/conf.d directory. (More than one of these files can be created, as desired, and they can all take effect.)
  2. Add configuration directives appropriate for source, destination, filter, or rewrite rule to the new xxx.conf file. These should follow standard syslog-ng syntax (more information can be found at syslog-ng Open Source Edition 3.22 - Administration Guide).
  3. Important: Custom log entries should not be created or configured. It is required that the log section be modified only by LogZilla, or LogZilla may cease receiving events.

If log customization is desired, such as adding new filters or rewrites, then see below for detailed instructions.

For the basic cases, like adding new destinations or sources, adding a file in conf.d is enough. All sources and destinations defined in these files will be implicitly added to the main config. If this is all you need, then restart syslog-ng as described below.

For some advanced cases, like when you want to add some extra filters, then /etc/logzilla/syslog-ng/config.yaml should be modified. In particular, if extra syslog-ng configuration directives are needed, they should be added to the extra_log_rules entry in this file.

Custom Configuration Example

In this example, a special source reading from an MQTT broker will be added. In addition, these log messages will be filtered such that the only log messages handled are those from host 1.2.3.4.

First, create the file /etc/logzilla/syslog-ng/conf.d/mqtt.conf with the following content:

source s_mqtt {
    mqtt(
        address("tcp://my-mqtt-server:4444")
        topic("test/abc")
    );
};

filter f_host_1234 {
    host("1.2.3.4");
};

As we want to also add some extra filters, we need to modify the yaml configuration file /etc/logzilla/syslog-ng/config.yaml.

Find the extra_log_rules setting (it’s an empty string by default) and update it:

extra_log_rules: "filter(f_host_1234);"

Restarting syslog-ng after changes

After any changes are made to the syslog-ng configuration, LogZilla’s syslog-ng module must be restarted. This can be accomplished via logzilla restart -c syslog. If proper operation is not observed or for more information, the syslog-ng operation logs can