Receiving Syslog Events

LogZilla's Syslog-ng Configuration

LogZilla supports customization of its syslog-ng configuration. Though it is available, because it is complex to configure and it complicates the LogZilla installation it is not recommended.

To use a custom syslog-ng configuration the /etc/logzilla/syslog-ng/config.yaml needs to be modified as appropriate for the desired configuration. Be aware that LogZilla may make changes to this file on its own when certain logzilla config options are changed.

Typically, customization is desired to create new syslog-ng sources, destinations, filters, or rewrite rules, To accomplish this:

Create a xxx.conf file (where xxx is the desired name) in the /etc/logzilla/syslog-ng/conf.d directory. (More than one of these files can be created, as desired, and they can all take effect.)

Add configuration directives appropriate for source, destination, filter, or rewrite rule to the new xxx.conf file. These should follow standard syslog-ng syntax (more information can be found at syslog-ng Open Source Edition 3.22 - Administration Guide

Important: Custom log entries should not be created or configured. It is required that the log section be modified only by LogZilla, or LogZilla may cease receiving events.

If log customization is desired, such as adding new sources or destinations, first a configuration file as described above (xxx.conf in the example) should contain the syslog-ng directives to define the source or destination.

Then /etc/logzilla/syslog-ng/config.yaml should be modified. This is a YAML text file. There is a section for sources: and a section for destinations:. Each of those lists the basic sources and destinations that LogZilla uses. Any custom sources or destinations would be added to these sections alongside the existing, default ones.

For example /etc/logzilla/syslog-ng/config.yaml by default may look like the following (excerpt):

destinations:
- buffer_dir: /var/lib/logzilla-maintenance/syslog-ng/
  disk_buf_size: 268435456
  enabled: true
  host: 127.0.0.1
  mem_buf_length: 10000
  mem_buf_size: 4194304
  name: logzilla
  port: 32412
  reliable: false
  type: logzilla
  workers: 1
- enabled: false
  name: debug_tsv
  path: /var/log/logzilla/syslog/debug.log
  template: debug_tsv
  type: file

This portion of the file is defining the destinations that LogZilla's syslog-ng will use. A few lines will need to be added to this section, corresponding to any destinations defined in the xxx.conf custom file mentioned above.

Those lines should go at the end of the current list of destinations but before the next YAML entry. The entry for the new destination would be "inside" the YAML entry for destinations: as above, so it would precede the section:

extra_log_rules: ''
flow_control: true
sources:
- enabled: true

This example will assume that a new destination d_example_destination was created in the xxx.conf file. Then the new YAML destination entry should look like the following:

- name: d_example_destination
  type: user_defined
  enabled: True

In that example d_example_destination would be replaced by the name given to the new syslog-ng destination in xxx.conf. If there is more than one new destination, those three lines would be repeated for each.

This would make the middle of the /etc/logzilla/syslog-ng/config.yaml file look like the following:

- enabled: false
  name: debug_json
  path: /var/log/logzilla/syslog/debug-json.log
  template: json
  type: file
- enabled: false
  name: pci_compliant
  path: /var/log/logzilla/pci-compliant/$R_YEAR-$R_MONTH/$R_YEAR-$R_MONTH-$R_DAY.log
  template: pci_tsv
  type: file
- name: d_example_destination
  type: user_defined
  enabled: True
extra_log_rules: ''
flow_control: true
sources:
- enabled: true
  flags:
  - syslog-protocol

This same process would be used for any new custom sources, except that the new entry would go in the YAML sources: data element. As an example, for a new source s_example_source the new YAML entry would go in the sources: section, and look like:

- name: s_example_source
  type: user_defined
  enabled: True

If extra syslog-ng configuration directives are needed, besides custom sources and destinations, they should be added to the extra_log_rules entry in the /etc/logzilla/syslog-ng/config.yaml file. Additional log rules placed there will be used by LogZilla.

Custom Configuration Example

In this example, a special source reading from MQTT broker will be added. In addition, these log messages will be filtered such that the only log messages handled are those from host 1.2.3.4.

First, the file /etc/logzilla/syslog-ng/conf.d/mqtt.conf should be created, with the following content:

    source s_mqtt {
        mqtt(
            address("tcp://my-mqtt-server:4444")
            topic("test/abc")
        );
    };

    filter f_host_1234 {
        host("1.2.3.4");
    };

Next, the yaml configuration file /etc/logzilla/syslog-ng/config.yaml, should be modified.

First, the new source should be added to the list of sources:

    sources:
    # ...
    # all existing entries in this section should remain here
    # ...
      - name: s_mqtt
        type: user_defined
        enabled: True

Second, extra_log_rules should be updated:

    extra_log_rules: "filter(f_host_1234);"

After any changes are made to the syslog-ng configuration, LogZilla's syslog-ng module must be restarted. This can be accomplished via logzilla restart -c syslog. If proper operation is not observed or for more information, the syslog-ng operation logs can be examined via docker logs lz_syslog.