I think, it’s a frequent question how to increase the performance of syslog-ng. In the current post, I try to introduce how to set up syslog-ng to reach the best performance, if it receives messages from the network via TCP and stores them into file. These settings will affect to syslog-ng working as a server.

Warnings:

• Tuning these settings may cause higher CPU and/or memory usage of syslog-ng
• It may increase the chance of possible losing messages if your computer crashes or other fatal error happens.
• My measurements affects to syslog-ng PE 4.0.1. I used TCP source and file destination in my configuration. Perhaps, these results and conclusions are not true for other syslog-ng versions (including OSE) or for other source and destination types.
• Do not tune syslog-ng, if you don’t know how these settings work, or you don’t understand the impact of tuning options perfectly. In other cases, to tune syslog-ng can cause worse results or unexpected behavior.

First of all, let’s talk about the syslog-ng tuning options affecting performance:

source options:

• log_fetch_limit : The maximum number of messages fetched from a source during a single poll loop
• log_iw_size: The size of the initial window, this value is used during flow control

destination options:

• flush_lines: Specifies how many lines are flushed to a destination at a time. Syslog-ng waits for this number of lines to accumulate and sends them off in a single batch. Setting this number high increases throughput as fully filled frames are sent to the network, but also increases message latency
• flush_timeout: specifies the time syslog-ng waits for lines to accumulate in its output buffer
• log_fifo_size: The number of entries in the output buffer (output fifo)

log_path option:

• flags(flow_control): Enables flow-control to the log path, meaning that syslog-ng will stop reading messages from the sources of this log statement if the destinations are not able to process the messages at the required speed. If disabled, syslog-ng will drop messages if the destination queues are full. If enabled, syslog-ng will only drop messages if the destination queues/window sizes are improperly sized
• For more details: http://www.balabit.com/sites/default/files/documents/syslog-ng-pe-v4.0-guide-admin-en.html/concepts_flow_control.html

Now, let’s see how these options change syslog-ng performance. I measured the performance of syslog-ng using the above options with several values to determine if they can change it.

Test environment

My test environment consisted two computers with the same hardware:

• CPU: 2 x Intel Xeon E5620 @ 2,4 GHz (quad-core)
• RAM: 18 GB
• HDD: Western Digital WD2502ABYS
• Network: GigaBit Ethernet
• Operating system: Ubuntu Lucid x64

syslog-ng version:

• I used syslog-ng PE 4.0.1 as server
• Note: In v4.0.1, a lot of syslog-ng performance fixes was improved, particularly in case of file destination
• Note: syslog-ng PE v4.0.1 is still a single-threaded application, so the benefits of multiple cores (processors) are not used (excepting if you are using logstore). We are planning to introduce multi-threaded syslog-ng in version 4.1.0.

Client settings:

• I used loggen with a high rate to generate messages, because it is a very useful tool for performance testing.
• Note: loggen of syslog-ng v4.0.1 can simulate more active and idle connections
• Basically, I tested syslog-ng with 200 bytes size of messages
• Every test last 600 seconds (10 min)

First of all, I measured the performance of syslog-ng using a basic configuration:

@version: 4.0
options {
};
source legacy_tcp {tcp(port(514) max-connections(100))
;};
destination d_file{file("/tmp/outputfile");};
log {
source(legacy_tcp);
destination(d_file);
flags(flow-control);
};

As you can see, it contains a simple TCP source, and the incoming messages are written into a simple text file. I used flow-control in my all measurements to avoid losing messages.

Here is the loggen output, if I used the above basic configuration:

syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 115228.48 msg/sec, count=69137275, time=600.001, (last) msg size=200, bandwidth=22492.60 kB/sec

In my case, syslog-ng could process 115000 msg/sec by default. It will be my initial number.

Tuning options

Source options

log_fetch_limit

Firstly, I tuned log_fetch_limit source option. Generally, increasing log_fetch_limit gives a better performance, because syslog-ng can read more lines from the source in a poll loop. So I put log_fetch_limit() into my default configuration, then changed its value.

log_fetch_limit(1):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 66769.30 msg/sec, count=40061765, time=600.002, (last) msg size=200, bandwidth=13033.37 kB/sec

log_fetch_limit(10) (default, I copied here the output of my initial measurement):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 115228.48 msg/sec, count=69137275, time=600.001, (last) msg size=200, bandwidth=22492.60 kB/sec

log_fetch_limit(100):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 140555.76 msg/sec, count=84333456, time=600.000, (last) msg size=200, bandwidth=27436.48 kB/sec

log_fetch_limit(1000):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 133018.41 msg/sec, count=79811302, time=600.001, (last) msg size=200, bandwidth=25965.19 kB/sec

Summary:

• Decreasing log_fetch_limit to a smaller number than the default made the results worse
• Increasing to 100 gave +20% performance compared to initial measurement
• Increasing to a larger number (1000) made the results worse
• The best value was 100

log_iw_size

log_iw_size(10), log_iw_size(100), log_iw_size(1000):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 115228.48 msg/sec, count=69137275, time=600.001, (last) msg size=200, bandwidth=22492.60 kB/sec

Summary:

• Based on my tests, increasing log_iw_size only, didn’t change the performance compared to basic measurement.

Destination options

flush_lines

flush_lines(1) (default):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 115228.48 msg/sec, count=69137275, time=600.001, (last) msg size=200, bandwidth=22492.60 kB/sec

flush_lines(10):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 142941.38 msg/sec, count=85764976, time=600.001, (last) msg size=200, bandwidth=27902.16 kB/sec

flush_lines(100):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 175138.63 msg/sec, count=105083475, time=600.001, (last) msg size=200, bandwidth=34187.06 kB/sec

flush_lines(1000):
well, it did not work, because flush_lines was larger than the default log_iw_size (100) ,so syslog-ng waited for reaching flush_timeout instead of flush_lines. Increasing flush_lines higher than log_iw_size, syslog-ng will process messages very slowly.
So that, I set log_iw_size(1000) instead of default 100:

syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 157789.94 msg/sec, count=94687378, time=600.085, (last) msg size=200, bandwidth=30800.60 kB/sec

Summary:

• Increasing flush_lines to a larger number made the performance definitely better
• If I used flush_lines(100), it made +50% performance improvement
• Using flush_lines(1000) with a higher log_iw_size value (1000) made the results a bit worse(+40%), so the best value was flush_lines(100)
• Keep in your mind, that syslog-ng will write out the messages when it reaches the value of flush_lines or the flush_timeout. When I used 1000 for flush_lines, technically, it meant that syslog-ng waited for 1000 incoming messages before writing out. It can cause a larger latency if you follow the output files.

log_fifo_size

log_fifo_size(1000)  (default):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 115228.48 msg/sec, count=69137275, time=600.001, (last) msg size=200, bandwidth=22492.60 kB/sec

log_fifo_size(10000):
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 114182.71 msg/sec, count=68509892, time=600.002, (last) msg size=200, bandwidth=22288.47 kB/sec

Summary:

• Increasing log_fifo_size to a large number(10000) did not change the results
• Important: you should know that increasing this value causes that larger number of messages will be stored in the memory in the output buffer while they are waiting to write. You should not increase it too high (excepting if you have a lot of sources), because it can cause possible message losing, if your computer crashes.

Summary of all results

Based on my tests, the following values were the best to reach the highest performance with syslog-ng in my test environment:

• log_iw_size(100)
• log_fetch_limit(100)
• flush_lines(100)
• log_fifo_size(1000)

So, I made a newest performance test with these combined values. Here is the result:

syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 183752.99 msg/sec, count=110252035, time=600.001, (last) msg size=200, bandwidth=35868.58 kB/sec

Summary

I think, it’s a very good speed, taking into consideration, that syslog-ng uses only one core. Using all of tuning values, I reached 183000 msg/sec instead of the default 115 000 msg/sec, that is +60% performance improvement. Also, it means that syslog-ng could process 123 GB messages/hour.

Finally, here is my tuned configuration:

@version: 4.0
options {
flush_lines(100);
log_fetch_limit(100);
log_iw_size(100);
log_fifo_size(1000);
};
source legacy_tcp {tcp(port(514) max-connections(100));};
destination d_file {file("/var/log/messages");};
log {
source(legacy_tcp);
destination(d_file);
flags(flow-control);
};

Notes:

• Don’t forget, that the performance of syslog-ng depends on your system environment, too. For example, if your client(s) can forward messages quickly, but the network connection of the computer running syslog-ng is too slow, you won’t manage to increase the performance of syslog-ng regardless of setting any tuning option.
• Before tuning syslog-ng, you should identify these bottlenecks within your system and fix them.

Testing with bigger log sizes

After that, I continued measuring performance with larger log sizes to see if my tuned configuration works in this cases as well.
I used the above tuned configuration, but I increased the log sizes of generated messages on the client.

log_size: 500 bytes
syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 500 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 151869.27 msg/sec, count=91121565, time=600.000, (last) msg size=500, bandwidth=74112.21 kB/sec

log_size: 1024 bytes

syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 1024 -I 600 --active-connections=1 --idle-connections=0 10.100.254.179 514
average rate = 91864.74 msg/sec, count=55119498, time=600.007, (last) msg size=1024, bandwidth=91811.83 kB/sec

Summary:

• As you can see, the maximum number of processed messages were a bit lower compared to 200 bytes of messages, but the amount traffic was definitely higher.
• Using 1024 bytes of messages, I could reach 90 MB/s transfer speed from the theoretical 125 MB of the Gigabit Ethernet network. It’s a very good value. Technically, it means that syslog-ng could process even 310 GB of message/hour or 7 TB/day.

Testing with more clients

Now, let’s see what happened if I had more than one client.

Firstly, I measured the performance with the default configuration using 100 clients:

syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=100 --idle-connections=0 10.100.254.179 514
average rate = 117718.26 msg/sec, count=70630964, time=600.000, (last) msg size=200, bandwidth=22978.60 kB/sec

Then I used the above tuned configuration in the following cases:

number of clients: 10

ssyslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=10 --idle-connections=0 10.100.254.179 514
average rate = 174034.85 msg/sec, count=104428178, time=600.041, (last) msg size=200, bandwidth=33971.60 kB/sec

number of clients: 100

syslog-ng-perf1:/# /opt/syslog-ng/bin/loggen -iS -r 10000000 -s 200 -I 600 --active-connections=100 --idle-connections=0 10.100.254.179 514
average rate = 153359.44 msg/sec, count=92016599, time=600.006, (last) msg size=200, bandwidth=29935.76 kB/sec

Summary:

• With my tuned configuration, I could reach +30% performance compared to default even in case of 100 clients.
• syslog-ng could process 17 000 msg/sec/connection in case of 10 clients or 1500 msg/sec/connection in case of 100 clients, if all clients send messages simultaneously

Well, I guess I managed to measure the important numbers relation with syslog-ng performance and tuning options in case of TCP source and file destination.
In the future, I am planning to continue this topic with other performance tips and tricks (e.g.: Increasing performance of SQL destination, configuration optimization or just a how to use loggen tool).