Troubleshooting and debugging syslog-ng

Sometimes, syslog-ng seems to be working wrong. For example, it doesn’t send logs, or it doesn’t start, or – in an extreme case – it crashes.

How can you decide what the problem is? Is it a real syslog-ng bug or not?
I will tell you some methods to resolve these problems.

Before beginning to debug syslog-ng, make sure that your config contains what you really want. It’s the most important, because usually there are a lot of problems with wrong filters or flags (tip: always check “final” flag in your config). If you are satisfied with your config, you can continue debugging.

Using the built-in debugging parameters of syslog-ng

So, let’s start.
First of all, syslog-ng has a lot of parameters for debugging:

~$ sbin/syslog-ng –help-all
…
-F, –foreground Do not go into the background after initialization
-v, –verbose Be a bit more verbose
-d, –debug Enable debug messages
-t, –trace Enable trace messages
-e, –stderr Log messages to stderr
…

You can use these options to understand what syslog-ng does.
Here is my syslog-ng’s output after starting it:

~$ syslog-ng -Fevd
No server license found, running in client mode;
Running application hooks; hook=’1′
Running application hooks; hook=’3′
syslog-ng starting up; version=’3.0.3a’, cfg-fingerprint=’2641b80231fac047f4a9b9d65cd4c4cd4de839e8′, cfg-nonce-ndx=’0′, cfg-signature=’b70463cc06878b4f63937aa33f1e38a525e026c5′
Syslog connection accepted; fd=’8′, client=’AF_UNIX(anonymous)’, local=’AF_UNIX(/dev/log)’
Incoming log entry; line=’Dec 21 17:09:58 pzolee: something’
Filter rule evaluation begins; filter_rule=’f_match’
Filter node evaluation result; filter_result=’not-match’
Filter rule evaluation result; filter_result=’not-match’, filter_rule=’f_match’

You can see some information about syslog-ng (e.g., version, fingerprint of the config), the source of the message (in this example: /dev/log) and the full content of the incoming message.

If syslog-ng doesn’t send the logs, you need to check the incoming logs. If there is no incoming log, the problem will be on your source side. Probably, your source file is wrong (e.g., wrong name, wrong path), or the application, generating the log messages, doesn’t use the given source of syslog-ng.
You have to check your filters as well, the message may not to match your rule and will be dropped.

The first time the message matches your log path, you will also see the destination:

Initializing destination file writer; template=’/var/log/messages’, filename=’/var/log/messages’

A typical problem if you are using a file destination and everything seems to be correct, but the destination file is empty:
The file was removed and syslog-ng was not reloaded or restarted.

Debugging with strace

It can occur, that syslog-ng doesn’t do what you want and you cannot find the reason for this problem in debug mode. So, what can you do in this case?
I suggest using strace.

For example:
syslog-ng won’t resolve IP addresses, although “use_dns(yes)” is configured.

~$ strace -s 256 -f syslog-ng
…
9835 open(“/etc/resolv.conf”, O_RDONLY) = -1 ENOENT (No such file or directory)
9835 uname({sys=”Linux”, node=”thor”, …}) = 0
9835 stat(“/etc/resolv.conf”, 0x7fff612518a0) = -1 ENOENT (No such file or directory)
…

As you can see, resolv.conf is missing and that’s the reason for the behavior.
For more information about strace, see the man page of strace.

Creating and using core dump

If syslog-ng crashes, a core file can be created. To do this, you need to start syslog-ng with the “–enable-core” option or set “ulimit -c unlimited”.
You can set them in the syslog-ng init script or when starting syslog-ng manually from the command line.

For example:

~$ syslog-ng -F –enable-core

Occasionally, syslog-ng hangs in an unknown state and doesn’t drop a core file.
In this case, you need to determine the PID of syslog-ng and send a SEGV signal to syslog-ng.

If you have a core file and syslog-ng was compiled with the “–enable-debug” option, you can debug it with gdb. To make sure that it was turned on, issue the following command:

~$ syslog-ng -V
…
Enable-Debug: on
…

Now you can use gdb (just for experts). For example:

~$ gdb syslog-ng core
…
Core was generated by `/home/pzolee/uj/product/syslog-ng/ose/3.0.5/src/syslog-ng -F –enable-core’.
Program terminated with signal 11, Segmentation fault.
[New process 26709]
#0 0x00007fd65a0c245f in poll () from /lib/libc.so.6
gdb) bt full
#0 0x00007fd65a0c245f in poll () from /lib/libc.so.6
No symbol table info available.
#1 0x00007fd65af8e77f in ?? () from /usr/lib/libglib-2.0.so.0
No symbol table info available.
#2 0x00007fd65af8ea7c in g_main_context_iteration () from /usr/lib/libglib-2.0.so.0
No symbol table info available.
#3 0x00000000004086b1 in main_loop_run (cfg=0x7fff27810900) at main.c:151
iters = 0
stats_timer_id = 5
…

For more details about gdb, see the gdb documentation (http://www.gnu.org/software/gdb/documentation/)

Other useful tools

You can also use other useful programs for testing syslog-ng.

To send logs to “/dev/log”, use the logger. This program exists on the most platforms.
For example:

~$ logger “message part”

To send logs to syslog-ng over the network, use the loggen program, which comes with syslog-ng 3.0. This is a very effective tool for testing different protocols (e.g.: UDP, TCP, SYSLOG).
For example:

pzolee@thor:~$ loggen -r 10 -i -s 300 -I 2 10.30.0.32 9999
average rate = 14.00 msg/sec, count=28, time=2.000, msg size=300, bandwidth=4.10 kB/sec

To watch logs on the other side without a second syslog-ng, use the netcat program.
For example:

root@thor:/home/pzolee# nc -lp 9999
123 1 2009-12-23T09:16:35+01:00 thor.balabit syslog-ng 6149 – [meta sequenceId="1"] syslog-ng starting up; version=’3.0.5′

I think, you know almost everything about troubleshooting syslog-ng now. So, if you have problems with syslog-ng, just read again this blog and follow the instructions.