Skip to content

AWS CloudWatch and Kinesis Setup

This section details how to set up LogZilla and AWS so that AWS CloudWatch logs are sent to LogZilla for processing.

LogZilla Setup

Auth Token

If you have not already generated an auth token for AWS Firehose to use in connection with LogZilla, ssh to your logzilla server and execute logzilla authtoken create (use sudo if you are not logged in as root).

Sample output:

root@1206r [~]: # logzilla authtoken create
No user specified (missing -U option). I'll create key for admin
b2d8c210f54ed85511f1867cb6cc4faa8ae85bff42c3dd26
The last string is the one you will need to copy and keep somewhere safe.

AWS Setup

Log into your AWS account and go to the AWS Services drop-down menu at the top left and search for kinesis, then select Kinesis Data Firehose.

Select Kinesis Data Firehose

Next, select Create delivery stream near the top right.

Select Create Delivery Stream

Set the source as Direct PUT and destination as HTTP Endpoint, then click Create Delivery Stream

Select Create Delivery Stream Source and Destination

Next, set a Delivery Stream Name such as logzilla

Enter Create Delivery Stream Name

For Destination Settings, set the HTTP endpoint name, HTTP endpoint URL, Access key, and enable GZIP.

The Access key is the token generated by the logzilla authtoken create command at the top of this document. Note: if this token value needs to be changed after initial configuration, the LogZilla http_receiver docker container must be restarted. This can be done by restarting LogZilla altogether (logzilla restart) or can be selectively accomplished via restarting just the http_receiver container without restarting LogZilla, by doing: 

logzilla restart -c http_receiver

Enter Desination Settings

Under Backup Settings, either select a current S3 bucket that your company uses, or create a new one.

Enter Backup Settings

Click Create Delivery Stream at the bottom of the form.

Click Create Delivery Stream

Check your LogZilla server for events.

Troubleshooting

If you do not have any incoming events from AWS, verify your settings in AWS for the correct URL and settings.

Verify using cURL

To verify that your LogZilla server is able to receive events, use the following command:

  • Be sure to replace the X-Amz-Firehose-Access-Key below with the token generated by the logzilla authtoken create command at the top of this document.

The following curl command will send a test event in gzip format to your LogZilla server. The event should show up in LogZilla as Curl test for LogZilla firehose reception.

url="http://logzilla.company.com/incoming"
apikey="b2d8c210f54ed85511f1867cb6cc4faa8ae85bff42c3dd26"
base64="base64"
[[ $OSTYPE == "linux-gnu" ]] && base64="base64 -w 0"

curl -X POST $url -H 'Content-Type: application/json' -H "X-Amz-Firehose-Access-Key: $apikey" -d '{"requestId": "xyz", "records": [{"data": "'$(echo "Curl test for LogZilla firehose reception" |gzip|$base64)'\n"}]}'

After event generation from curl, search your LogZilla instance for a program name of kinesis:

Check LogZilla Programs for Kinesis

Your search results will appear similar to:

LogZilla Kinesis Search Results

Verify using tcpdump

You can also check reception from AWS to LogZilla using the instructions in the Debugging Event Reception section.