JPGraph Error

I asked a colleague of mine to set up a PHP-Syslog-NG (http://code.google.com/p/php-syslog-ng/) as the central logging system of all our devices and servers. He did this on a FreeBSD 7.0 and everything started working fine but the "Graph" section which works with JPGraph couldn't draw any Graphs and it came up with this message: "JpGraph Error Font file "/usr/share/fonts/truetype/msttcorefonts/verdana.ttf" is not readable or does not exist..
After some digging into the codes and configs I got over the problem following the bellow steps:

1. Install TrueType font on FreeBSD (XfStt).
2. Fetch the "verdana.ttf" font and place it in the directory that XfStt created for TrueType fonts.
3. Change the default "TTF_DIR" parameter.

Step 1:
An easy way to use TrueType fonts in BSD is to install XfStt which is available through ports collection at "/usr/ports/x11-servers/Xfstt/".
After the installation a directory will be created for TrueType fonts at "/usr/local/lib/X11/fonts/TrueType/".
Step 2:
downloading verdana.ttf and placing it in "/usr/local/lib/X11/fonts/TrueType/".
I fetched my copy from "http://www.afosteo.org/Download/Fonts/"
Step 3:
The final step is to point JPgraph TTF_DIR parameter inside the jpg-config.inc configuration file to to proper location.
jpg-config.inc is located at "/usr/svr/php-syslog-ng/html/includes/jpgraph"

TCPDUMP Tutorial

TCPDUMP is a wonderful command line tool which helps analyzing and troubleshooting network traffic on a Linux host.

The tcpdump options I use the most are:

• -n : Don't resolve hostnames.

• -nn : Don't resolve hostnames or port names.

• -v, -vv, -vvv : Increase the amount of packet information you get back.

The followings are the most tcpdump expressions I use:

Display any traffic souring and destining a specific host:

• tcpdump host "Host Address"

Display any traffic sourcing a specific host:

• tcpdump src "Host-address"

Display any traffic destining a specific host:

• tcpdump dst "Host-address"

Display any ICMP traffic:

• tcpdump icmp

Display traffic sourced or destined a specific network:

• tcpdump net "net-address"

Display any traffic sourcing or destining a specific port:

• tcpdump port "port-number"

Display any traffic sourcing a specific port:

• tcpdump src port "port-number"

Display any traffic destining a specific port:

• tcpdump dst port "port-number"

It is also possible to use "AND", "OR", and "Excpet":

You can learn more about tcpdump options and expressions with great examples at this location: http://dmiessler.com/study/tcpdump/

Dig DNS Lookup in Windows!

Every System Administrator dealing with DNS administration knows the power of "dig" command-line tool in Linux/Unix environment. But there are times when an administrator needs to monitor and troubleshoot DNS from a Windows station and she would then realizes the deprivation!
The good news is that many Linux/Unix tools have been ported to windows (Check my post about Windows IPFW) and one of them is the "dig" utility.

The windows version of "dig" can be downloaded and installed from here: http://members.shaw.ca/nicholas.fong/dig/


For those who are new to "dig" the following link helps:
Dig Howto: http://www.madboa.com/geek/dig/

Perl and Regular Expressions

I have been drawn into an exciting area called "Perl scripting!". It is a great fun and until I started studying Perl I didn't know how much I was lost in system and network administration!
Currently, what I mostly need to do with Perl is text processing.
In my first project I needed to telnet into a Fortigate firewall, send a couple of commands, fetch the output, process and reorder the output which was the main purpose of the job and store it in a file and trigger an action if specific pattern observed in the output.
All these required complex text processing which could not have been accomplished without "Regular Expressions".
The following are the references I found very useful for me to get a grasp on "Regular Expressions" in Perl.

Regular expressions in Perl
Perl regular expressions tutorial
Steve Litt's Perls of Wisdom: Perl Regular Expression

Application Layer Monitoring

It's been some time since I set up "ManageEngine Applications Manager" to monitor our services availability and health status. Like many popular monitoring applications it provides SNMP (Monitoring resources like CPU, Memory, Network Interfaces, Disks) and ICMP (Monitoring Layer 3 availability) monitoring but it goes beyond that by sending application specific probes to make sure different applications and services are available and healthy. A variety of application and services are supported like:

• POP3 and SMTP
• Web Server monitoring: This includes IIS and Apache with the ability to perform URL Monitoring
• Database Monitoring: DB2, MS-SQL, MySQL, Oracle
• Microsoft .NET / Tomcat / JBoss / WebSphere Monitoing

For the complete list click here.
The top reason I choose ManageEngine was the ability to monitor URLs. We are hosting around 200 web sites and it happens that a single URL goes down while the web server itself is healthy and no sign of service fault can be recognized.
Currently I am using MS-SQL, MySQL, Apache and IIS monitors and I really like the outputs, Reports and Graphs. The GUI provides quick overall view and quick access to monitors and reports.
Another great thing is the SLA management feature which lets us defining different SLA levels and assign them to monitor groups and report when a monitoring group has violated SLA Agreement.
It is really crucial to have a complete logging and monitoring solution functioning up to the application layer. There are a lot of application layer monitoring systems available; Commercial and Free. Many vendors are now including this level of monitoring into their network monitoring products.
To learn more on Different Monitoring tools just visit: http://www.monitortools.com/

A Complimentary to PHP-Syslog-NG

Its been a long time since I ran my php-syslog-ng server and I did a tough job to monitor, troubleshoot and make improvements to the logging system so that it makes more sense and acts more effectively. Since I was dealing with more than 150 devices sending their log messages down to my syslog-ng server a top task was to filter all unnecessary logs which is only possible with Syslog-ng and this great feature helped a lot to block thousands of logs.

Current statistics show around 400,000 logs per month which is great improvement against the over 5,000,000 logs from the very early months and I hope with stricter monitoring on redundant logs this number can still be reduced.

I am thinking of a Complimentary GUI to my PHP-Syslog-NG interface presenting analytical reports and I have already documented the requirements and asked a PHP developer in. I like it to provide a lot of different statistical reports and charts so that I can easily track and address issues.

How To Set Up A Linux Syslog Server

Any network administrator in charge of a few network devices would like to keep a record of the events on these devices like:

• who logged into devices (or tried to connect to network devices)
• when does he/she try to login
• what did she/he do after login
• what changes and events have been announced on the device (like interface status change)

And lots of other information that might give a clue about a policy violation or tracking a series of events that led to a network disruption incident.
What I am talking about is to have a simple SYSLOG server in place to collect all log messages to a central location.
If you need to setup a quick and easy syslog server just follow the link bellow. It is meant for Debian but will work on almost all Linux distributions :
Linux Syslog Server - How To Set Up A Debian Linux Syslog Server

Linux Traffic Control, DNS ALG issue , and Service Monitoring

I am busy with a couple of interesting stuff that keeps me away from posting here.
First,
I am working on a linux box which is an internet gateway and controls traffic using iptable and at same time it acts as the inter VLAN router for around 10 VLANs using 802.1Q. I am trying to run some sort of QoS to put traffic control on every VLAN for their internet usage. And with Linux this will be easily done with a tool called TC.
I have expressed that before but it is worth to mention again that with linux we will get tons of outstanding networking features that gives us full control on our network and what is running behind the scene on the wires! I am really impressed!!!
Second,
My firewall is not handling DNS ALG as expected so it has brought us some disturbance. Everything is fine with outside to inside regular DNS queries but when a DNS query for PTR record comes in, DNS ALG does not translate the IP address in the response payload while it does so when it comes to Forward queries. According to RFC 2694 this should work unless our Firewall is not RFC compliant. I am documenting the issue regarding RFC to send it over to our Firewall Vendor.
Third,
I was studying available service monitoring tools to monitor our IDC services and servers and nearly reached to the conclusion to implement "ManageEngine Applications Monitror" and I already have a pilot server in place. I will post on it later.

MySQL table crashed again on Syslog server!

A couple of days ago we were having a downtime so I had to shutdown all my servers including Syslog server which is a php-syslog-ng server, as mentioned in my previous posts. But after it came back online no logs were collected anymore and by closer examination I found out that the "logs" table in MySQL has been marked as crashed and automatic repair didn't help either(Through mysqld log files and "check table [tablename]" through mysql client).
I used "myisamchk" tool which is used to check and repair MyISAM tables as follow:
[root@logcollector]# myisamchk -rf logs.MYI (tried the quick mode first but not helpful)
It took eight hours to check and fix the indexes!
This was the second time that my syslog-ng server went down for about 2 days because of a failure in MySQL table. The first time a sudden and unexpected shutdown brought about the table crash but this time was different. I am doubting that mysqld couldn't shutdown its databases safely and that might be because there are more than 150 devices sending their messages to this server. Checking my past rotated syslog databases I see that each month I have a database of 15 to 20 GB in size. I am thinking of some way to split my database into different parts to make it easier to handle.

Syslog Message Header Format

Finally I fixed my problem with huawei syslog messages! Actually I found a work around to it on my Syslog-NG server. There is a bad_hostname("regexp") directive in syslog-ng configuration file which can be used to ignore wrong hostnames extracted from syslog header messages if specified, and the cool thing is that you can use regular expresions to specify a range of hostnames matching your criteria. Once a bad_hostname matched syslog-ng will look into the IP header instead for the source address and places this address as the hostname.

But to realize what exactly the problem was I decided to take a close look into RFC 3164. Section 4.1.2 says:

• The HEADER contains two fields called the TIMESTAMP and the HOSTNAME.
• The TIMESTAMP field is the local time and is in the format of "Mmm dd hh:mm:ss" (without the quote marks)
• A single space character MUST follow the TIMESTAMP field
• The HOSTNAME field will contain only the hostname, the IPv4 address, or the IPv6 address of the originator of the message. The preferred value is the hostname
• Example: <34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8

It has firmly defined how a syslog message format should be and to see how Huawei is complying with this format I have captured traffic from huawei routers and switches with tcpdump:
>Oct 28 16:15:03 2006 SHZ-C2H-Router IFNET/5/UPDOWN:S[|syslog]

OK! instead of the HOSTNAME stgraight after the TIMESTAMP I see "2006" which is the YYYY part of the current date on huawei devices!
Anyway, as soon as the bad_hostname directive is there no need to bother for this kind of problem!

Add Syslog Support to Windows Servers

Windows does not support syslog but with a free add-on this ability can be added to all servers. SyslogAgent runs as a service on Windows Servers and it can convert all logs including logs in event viewer and service log files like those IIS generates for FTP, WWW and SMTP to syslog format and sends it to a central syslog server.
SyslogAgent is under GNU license, so it is free:
Datagram Free SyslogAgent

Huawei Devices and Strange Syslog Message Format!

My php-syslog-ng is up and running and I am in the process of configuring all my nodes to send their syslog messages to syslog-ng server but there seems to be some problems on some devices. No problem with my Linux servers and my Huawei firewalls but Huawei switches and routers are apparently using the wrong syslog message format. I can see all my switches and routers as a host named "2006" which is the YYYY part of the date in my php-syslog-ng web interface! and the hostname appears in the message string which is different from what RFC 3164 says about the syslog packet header. It seems that Huawei managed to have its own syslog message format on some of their devices. I am looking for some way to change this behavior.

Setting up Syslog-ng in Fedora Core 5

I am busy with setting up a php-syslog-ng server on Fedora Core 5 these days but things are really getting harder than I imagined! Having the syslog-ng service itself up and running is simple and straight forward but adding the opportunity to monitor the logs through a web interface; which is what I really need has made my life difficult!
The difficulty is where different details from Apache, MySQL and syslog-ng configuration file must be adjusted.
I am documenting the process as I go on and will put it here once completed.