Let Big Brother keep tabs on the health of your servers

August 12, 2003
Thomas Nooning CCNA, CCDA

Monitoring servers for basic health and problems with specific services is crucial for administrators. Getting notified that a system is pegged at 100 percent CPU utilization, for example, can give you some time to troubleshoot prior to a crash. Although there are numerous programs out there designed for monitoring and notification, many can be quite expensive. For Linux, Big Brother can definitely get the job done.

Big Brother allows you to monitor for standard services such as SMTP, POP3, DNS, and FTP, as well as system data, such as disk space and load. Big Brother is a highly configurable and extensible Web-based monitoring tool. The primary version is available for use on Linux/UNIX servers, and this is the version we will focus on here. There is also a version for Windows servers, which basically provides a subset of the Linux/UNIX features.

Licensing for Big Brother
Big Brother is available under the terms of its own "Better than Free" license. In a nutshell, if you're using Big Brother to monitor your own internal servers, you do not need to purchase a commercial license. However, some commercial enterprises do need to purchase licenses, such as IT consultants who use Big Brother to monitor clients' systems and ISPs that offer Big Brother to customers for monitoring their colocated servers.

The basics
Big Brother monitors configured system and network objects in real time and displays information on a Web page. Monitored devices appear in a list on the left-hand side; and what is being monitored, displayed on the right. (Click here for an example.)

Big Brother uses color coding—green, yellow, and red—so that you can quickly determine whether there is a problem. Green, of course, means that all is well and nothing is above configured thresholds. Yellow signifies a warning level. This is user-configurable and will depend on what is being monitored. If a local df returns a value of 91 percent usage (with 90 percent being the warning threshold), that particular host will be in a yellow warning state. Once it meets or exceeds the panic state, it will turn red. The background of the main Web page also takes on the same color as the most severe issue. This allows you to know right away if there is a problem.

You will also be able to click through on individual hosts or monitored objects to get more detailed information. The length of time the current status has been in effect is presented, as well as a link for historical data. The user interface is straightforward and easy to navigate.

Before we get too far ahead of ourselves, let's look at how to install Big Brother on one of your servers.

Installing Big Brother
Big Brother is available for download here. Once you have downloaded the package, you can prepare your system for installation. It is recommended that you create a new system account for the specific purpose of running Big Brother. This is generally a good idea and is preferred over running as root. To demonstrate, let's create a user "bb" and then install the package with the following commands:
adduser bb
tar xvfz bb-1.9c.tar.gz
cd bb-1.9c
cd install
./bbconfig <OS NAME>

Options for <OS NAME> include bsdi, sco3, sco, freebsd, solaris, hpux9, hpux, linux, sunos, netbsd, osf, ultrix, irix, unixware, redhat, aix, dynix, debian, dgux, caldera, and mandrake. You do not actually need to input an option here at all. Big Brother will attempt to auto-detect the OS and configure accordingly. But if you run into any installation issues, you may want to go back and manually set this option. File paths and library locations can differ from distribution to distribution and can cause the installation to fail.

You should already have an idea of how you plan to configure Big Brother when you run the bbconfig script. You should know what base directory you want the program to reside in. Will you be using fully-qualified domain names (FQDNs) in your monitoring? If everything is local, you may just plan on using the basic hostname (e.g., dns1, smtp-1, server1210, etc.). Otherwise, specify that FQDN and hostnames will be expanded to include the full name, such as dns1.example.net. You will also need to know which host or hosts will act as the BBDISPLAY (receives incoming messages and displays them on the Web page) and which will be the BBPAGER (processes page requests). Know also what e-mail address you want to use for notification and what the URL for Big Brother will be.

Once the initial configuration has been completed, you will need to execute the following commands to get Big Brother compiled:
cd bb-1.9c /src
make
make install

Big Brother should now be installed and ready for configuration.

Configuring Big Brother
After the initial installation of Big Brother, you will need to specify which hosts you want to monitor and configure additional settings that will control the monitoring. In the bb-1.9c/etc directory, you will first need to modify the bb-hosts file. The bb-hosts file will contain the IP addresses and hostnames of monitored devices, as well as any associated directives. The format of the file is:
<IP-ADDR> <HOSTNAME> # <DIRECTIVES>
Here are a few example entries:
192.168.1.5 web1.example.net # BBDISPLAY http://www.example.net/
192.168.1.25 smtp-1.example.net # BBPAGER ftp smtp pop3
192.168.2.53 ns2.example.net # dns noping

In the first line above, we see the IP address 192.168.1.5 mapped to web1.example.net. The directives specify that this machine will be the BBDISPLAY. This informs Big Brother that a Web server is running here to display monitoring information. The http:// that follows tells the program to monitor that specific URL. For each host, you can monitor http:// and https://. Multiple URLs can be entered with a space between each or with a pipe (|) between them, such as:
http://www.example.net/bb/|http://www.example.net/test/

The next line specifies smtp-1.example.net to be a BBPAGER, meaning this server will send page notifications. Next we see what services are to be monitored on the host—in this case, FTP, SMTP, and POP3. The machine will be polled on the corresponding ports at a set interval (default is five minutes).

The third line is similar, except this box will be monitored for DNS, and it will not be tested for ICMP reachability. Without an explicit "noping" directive, all hosts will be ping tested. This is useful if the host has ICMP disabled and you do not want it to constantly be in alarm.

Next in the configuration process will be the bbdef.sh file. This file stores settings for how Big Brother will react to different situations, as well as thresholds for the WARNING and PANIC alarms. For instance, the DFWARN and DFPANIC can be modified from their default 90% and 95% settings. This will depend on the host in question and the size of hard drives being monitored. You can always keep the default settings and tweak them as necessary. A few pages in the middle of the night and you might decide a higher PANIC level is required.

You'll need to check two more files before the program can get up and running: bbwarnsetup.cfg and bbwarnrules.cfg. These files allow you to configure a variety of settings governing how notification is handled. This includes paging intervals, formatting procedures, and options for excluding specific hosts or types of alarms. The default settings will probably work for most installations, but it's a good idea to at least review the available options.

You should already be running a Web server (such as Apache) to have Big Brother data viewable as a Web page. You can create a symbolic link under your Document Root directory to point to the Big Brother files like this:
ln –s /path_to_Big_Brother /Document_Root/bb

Prior to running Big Brother, you'll want to check ownership on the program files. If you created a user such as bb, chown the necessary files accordingly and make sure they will be viewable as Web pages. Big Brother is then ready to be started with the following command:
./runbb.sh start

Big Brother should be accessible via your installed URL—for example, http://www.example.net/bb. You can now click through your hosts and get comfortable with the interface.

Summary
Monitoring production (and even test or development) servers is an important part of any administrator's job. Being alerted when an important service has ceased to function decreases the likelihood of an extended outage. Tracking persistent problems can also be facilitated through the use of the historical data that Big Brother automatically collects.

As a bonus, Big Brother was created to be extensible, and there are add-on modules for monitoring databases and creating statistics, among other options. Click here for a list of many of the available modules.