SAR - System Activity Reporter

Basic SAR Setup (from HP-UX sys admin handbook and tooltips, p503):

sar -o /tmp/sar.data 60 300 		# run sar every 60 sec for 300 count, 
	-o store info in file (bin)

sar -u -f /var/adm/sa/saXX		# read data from file (Solaris, XX = date number)
sar -u -f /tmp/sar.data			# read data from file (HP-UX)
	-u display cpu info (similar to iostat and vmstat)
	-b buffer cache activity, imp for oracle
	-d disk activity
	-q avg queue length (if run queue > num of cpu, will have to wait).
	-w swap info


Solaris starts sadc in /etc/rc2.d/S21perf , a deamon to collect sar info.

------------

Setup SAR data collection for HP-UX (should also work for other platform):

http://www.sarcheck.com/sarhowto.htm	(Actually SarCheck.com, but cost money!)

mkdir  /var/adm/sa, 
then setup root crontab:

#collect sar data  	# every 20 min 8-5, hourly outside normal work 
0 * * * * /usr/lbin/sa/sa1
20,40 8-17 * * 1-5 /usr/lbin/sa/sa1
#reduce the sar data	# generate pre-formated report focus for business hrs
5 18 * * * /usr/lbin/sa/sa2 -s 8:00 -e 18:01 -i 1200 -A

# sample for SF + Minsk work hours
0 * * * * /usr/lbin/sa/sa1
15,30,45 0-8,10-19,23 * * 1-5 /usr/lbin/sa/sa1
05 21 * * * /usr/lbin/sa/sa2 -i 3600 -A

# sa1 is data collection to /var/adm/sa/saXX
# sa2 really produce condense version of report to /var/adm/sa/sarXX (sar vs sa)

# filenames are reused every month.
# use sar -A -f /var/adm/sa/saXX to get more detail report than std summary.


-------------
AIX has preset entries in crontab for 'adm'.  Check to ensure script exsit.
sar logs are stored in /var/adm/sa
-------------

Minsk is PT + 10 hours (ahead).

SF  vs  Minsk work hours
---- 	------
 8am	6pm
10am	8pm
 6pm	4am

10pm	 8am
midnite	10am
8am	 6pm

SE Toolkit

setup env:

export PATH=$PATH:/opt/RICHPse/bin
export SEPATH=/opt/RICHPse/examples:/opt/RICHPse/toptool

interactive tools:

se zoom.se		# gui, summary status for all components.  Main Window. 
se live_test.se		# text version of zoom.se
se multimeter.se	# gui, cpu, cache, vm and locks meter

se toptool.se		# gui, just like top
se xload.se		# gui, just like xload, show hostname :)
se infotool.se		# gui, menu to lot of sys info (cpu, net, disk, etc)
se xit			# gui  wrap on text disk stat dump (xiostat.se)

se -DWIDE pea.se 10	# text, dump top like info to stdout every 10 sec
se disks.se		# text, dump lot of disk usage info

se webtune.se		# display current, min and max values for perf params

se virtual_adrain.se &	# text, dump warning to stdout if perf problem found 
			# run cli in background, non permanent, only output to
			# login screen; process end, all cleared.

-------------------------

# install:
# pkgrm RICHPse
# gunzip RICHPse.tar.gz
# tar xf RICHPse.tar
# pkgadd -d . RICHPse
# edit /opt/RICHPse/etc/se_defines, enable "disk nfs"

# alt, can just copy to network drive, and set PATH and SEPATH
# at least for the interactive tools above

# always run monitor:
/opt/RICHPse/etc/init.d/vader start     # init.d script to start vader
se /opt/RICHPse/examples/vader.se       # the "Virtual Adrian Daemon", 
                                        # start on host to be monitored

se /opt/RICHPse/examples/darth.se -h remotehost # gui, start on client.
	# This gui is the front end of the bg monitor

#!/bin/sh

# setoolkit-install.sh
# quick script to setup  and start se toolkit

cd /mnt/sa/share/software/SEtoolkit

pkgadd -d . RICHPse.331


(cd /opt/RICHPse/etc; tar cf - *.d) | (cd /etc ; tar xvf - )

# /etc/init.d/mon_cm start
/etc/init.d/monlog start
/etc/init.d/percol start
/etc/init.d/va_monitor start
/etc/init.d/vader start

SAR

ganglia

Network Tracing

tcpdump

tcpdump parameters
-n: ip number, do no resolve hostname
-e: ethernet (?)
-i: interface
-s 16000		: set capture frame size to 16k 
-w [FILE]		: write output to file (capture use, more info than redirect output)
host IP-or-NAME		: capture info only related to the specified host

operators accepted:
&&	= and
||	= or
!	= not

eg cmd of tcpdump [expression]  :

tcpdump host 10.0.71.165
tcpdump src  10.0.71.165
tcpdump 'dst net 128.3'
tcpdump 'src or dst port ftp-data'   
tcpdump 'ether host 0:d0:b7:a9:c9:5a'

Sample trace output

showmount -e 192.168.209.30 # VIP
tcpdump -n host 172.24.51.182  # misconfigured NAT
18:49:41.964873 eth0 < 172.24.51.182 > tin-linux.zambeel.com: icmp: 172.24.51.182 udp port sunrpc unreachable [tos 0xc0] 
18:56:24.677264 eth0 < 172.24.51.182 > 10.0.15.11: icmp: 172.24.51.182 udp port sunrpc unreachable [tos 0xc0] 
18:56:24.679401 eth0 < 172.24.51.182 > 10.0.15.11: icmp: 172.24.51.182 udp port sunrpc unreachable [tos 0xc0] 
timestamp     src-if ?   source ip     destination prtl  err message

tcpdump -n port sunrpc
18:54:31.055821 eth0 > 10.0.15.11.1388 > 192.168.209.30.sunrpc: udp 56
              src-if ? source  ip.port ? dest        ip.port  : protocol + port


   [z-00D0B7A873CE] # tcpdump -e port sunrpc
18:15:55.628675 eth2 < 0:e0:52:d:7e:18 0:0:0:0:0:1 ip 74: 10.0.15.11.2499 > 172.24.51.182.sunrpc: S 4260207884:4260207884(0) win 32120  (DF)
time            if   ? src mac         dst-mac(host)      src ip.port            dest ip.port    TCP SYN and other protocol info
18:15:55.628696 eth2 > 0:0:0:0:0:0 0:2:e3:0:3b:9d ip 54: 172.24.51.182.sunrpc > 10.0.15.11.2499: R 0:0(0) ack 4260207885 win 0
time            if   ? src mac         dst-mac(host)      src ip.port            dest ip.port    TCP SYN and other protocol info

Here is an example of messed up translation.
Note that source & dest mac-address is rewritten on each router hop.


   [z-00D0B7A871DF] # tcpdump -n | egrep '10\.0\.15\.11|192\.168'
19:02:43.964206 eth2 > 172.24.51.12.telnet >   10.0.15.11.2411:   P 2646085534:2646085754(220) ack 2623622447 win 32120 {nop,nop,timestamp 2624922 80719743} (DF)
19:02:43.982115 eth2 < 10.0.15.11.2411     > 172.24.51.12.telnet: . 1:1(0) ack 220 win 31856 {nop,nop,timestamp 80720053 2624922} (DF)
19:02:45.277592 eth2 B 172.24.51.1.route   > 172.24.51.255.route: rip-resp 25: {192.168.13.0/255.255.255.0}(2) {192.168.14.0/255.255.255.0}(2) {192.168.15.0/255.255.255.0}(2) {192.168.16.0/255.255.255.0}(2) {192.168.17.0/255.255.255.0}(2)[|rip]

snoop

snoop host [IP]			# traffic with a given host (as src or dst)
snoop -r port 25		# all traffic in port 25 (smtp), 
				# do not resolve ip to dns names
-s 	= sniplet length (def is whole packet)
	= 80 ip hdr only, 120 = nfs header only

-V	= layer info
-v	= more verbose than -V, lot of info.


from cli :

Usage:  snoop
        [ -a ]                  # Listen to packets on audio
        [ -d device ]           # settable to le?, ie?, bf?, tr?
        [ -s snaplen ]          # Truncate packets
        [ -c count ]            # Quit after count packets
        [ -P ]                  # Turn OFF promiscuous mode
        [ -D ]                  # Report dropped packets
        [ -S ]                  # Report packet size
        [ -i file ]             # Read previously captured packets
        [ -o file ]             # Capture packets in file
        [ -n file ]             # Load addr-to-name table from file
        [ -N ]                  # Create addr-to-name table
        [ -t  r|a|d ]           # Time: Relative, Absolute or Delta
        [ -v ]                  # Verbose packet display
        [ -V ]                  # Show all summary lines
        [ -p first[,last] ]     # Select packet(s) to display
        [ -x offset[,length] ]  # Hex dump from offset for length
        [ -C ]                  # Print packet filter code

Sample snoop

Capture traffic on NIC hme0 specific to a host, capture up 8K of the packet, 
and dump result to an output file:
snoop -d hme0 -s 8192 -o /tmp/snoop.out host 10.215.55.211

Read input file back.  May wish to use ethereal to read this file for easier access.
snoop -i /tmp/snoop.out		


snoop -s 120 port 25 host 211.196.53.194

titaniumleg.com  mail server traffic monitor
snoop -r -D -P -s 1500 -c 100000 -o /export/tmp/smtp01.20030122.snoop port 25

snoop -n /dev/null  -D -P -s 1500 -c 100000 -o /export/tmp/smtp01.20030122.snoop port 25
snoop -D -s 9000 -c 100000 -o jumpstartclient.snoop host jumpstartclient
-r = do not resolve hostname  # not in sol  7 snoop
-D = display num of dropped packets
-P = non promiscuous mode capture   (don't  use in troubleshooting jumpstart problems).
-s snipplet length
-c count num of backets to capture
-o output file



###
### more explanations TBA
###

Ethereal

ethereal (GUI)
tethereal (CLI)

most flags work for both.



snoop-like behaviour (mostly for ethereal):
-l	: scroll capture 
-S 	: update as capture is in progress.
-k 	: start capture immediately  (disable iinteraction?)

--

-i [IF] : specify interface, eg eth0, hme0
-n 	: no dns resolution, use ip Number


-V 	: more verbose output, captured data diisplayed in tree mode instead of 1 line per packet.


-f 	: capture filter expression  (tcpdump nnotation needed), eg:

	>	tcp port 23 and host 10.0.0.5
	>   src net 10.0.15.0/24
	>   dst net 10.0.15.0 mask 255.255.255.0
	>> 	[src|dst] host 
	>>	ether [src|dst] host 00:E0:2B:DE:0E:00
	>> 	[tcp|udp] [src|dst] port 

	host 10.215.20.152 || host 10.215.2.21 || host 10.215.19.73



-------------------------------------------------------------

ethereal view filter expression 
[ work in GUI filter box when viewing, 
NOT as capture filter (which is tcpdump format ]

operatos:
           eq, ==    Equal
           ne, !=    Not equal
           gt, >     Greater than
           lt, <     Less Than
           ge, >=    Greater than or Equal to
           le, <=    Less than or Equal to

           and, &&   Logical AND
           or, ||    Logical OR
           not, !    Logical NOT

boolean: true (1) or false (0)

some commonly used filter fields:

           eth.src == aa-aa-aa-aa-aa-aa
           ip.dst eq www.mit.edu
           ip.src == 192.168.1.1
           ip.addr == 129.111.0.0/16
           eth.src == aa-aa-aa-aa-aa-aa
           eth.src[0:3] == 00:00:83			# filter by vendor by use of slide
           tcp.port == 80 and ip.src == 192.168.2.1
		   ip.addr is for both src or dest, these multiple ocurring field is a bit confusing for packet filtering.

for generic filter dealing with a specific host, but not necessary filtering by tcp/udp/icmp
ip.dst
ip.src
ip.addr

udp
udp.port
udp.dstport
udp.srcport

tcp
tcp.port
tcp.dstport
tcp.srcport
tcp.seq

icmp


bootp.dhcp==true		: frame is dhcp
bootp.hw.addr

smb.cmd==(unsigned 8 bit int)	: smb protocol command number
smb.cmd == 0x06  		: cmd is smb unlink
smb.status != 0x0000	: Error code, 4 bytes aka status, lot of items.
smb.errcls != 0x0		: error class, 1 byte represent the categories
              0x0       = Success
              0x1       = DOS Error
              0x2       = Server Error
              0x3 	= hardware error
              0x4	= not a smb cmd
			Note, netBench Fail code 32 maybe in Dos or Hrd.
smb.pid
smb.mid		(multiplex id)
smb.uid		(user id, maybe per process)
nfs.*
nfs.fh.version != 3		= not sure what this is, not nfs protocol version!
rpc.programversion != 3		= all packet that are rpc program nfs version 3.

lot of higher level protocol stuff available, including vlan on switches, etc.
see the man page on ethereal or tethereal (very long!)


GUI version, filter can just enter a protocol type.  eg: smb
That means smb protocol is present.  A protocol in the filter w/o any comparison operator means filter packets where such field is present in the packet.  
eg: smb.errcls  filter packet that contain smb error class.




Network trace capture with tcpdump or snoop, save to file for viewing with ethereal

tcpdump -i [interface] -s 1500 -w [some-file]
tcpdump -s 8192 -w netuse.tcpdump 'host 10.0.71.232 or host 10.0.71.15'
snoop -d hme0  -o /tmp/snoop.out host 10.215.55.211

editcap can be used to trim captured file, or convert between formats
(tcpdump, ethereal, snoop, ms netmon, etc).

Network Scanner

nmap

nmap: network scanner
nmapfe: w/ gui front end, supposed to need gtk, but worked anyway.

nmap -sT -O -PI -PT 172.27.31.0/24	# scan whole class C vlan 31, with os identification.  long output.

Intrusion Detection

tripwire

genereate site-key, host-key:
twadmin --generate-keys --site-keyfile ./site.key
twadmin --generate-keys --local-keyfile ./$HOSTNAME-local.key


compile config and policy file from text to binary format:

twadmin --create-cfgfile --cfgfile ./tw.cfg --site-keyfile ./site.key  /floppy/twcfg.txt
twadmin --create-polfile --cfgfile ./tw.cfg --site-keyfile ./site.key  /floppy/twpol.txt


tripwire -m i   	# or --init, to create initial DB of host config.

run tw periodically and monitor db changes, check that all binary and db have not been changed.

tripwire -m c		# or --check


twprint --print-report --twrfile  $TRIPWIRE-REPORT/host.date.twr
	# generate a human readable report from result of --check



Securing tripwire:
cd $TRIPWIRE-BIN		# Tripwire binaries, eg /usr/local/tripwire/bin
chmod 0500 siggen tripwire twadmin twprint
md5sum * > tripwire-bin-md5sum.txt	
cp tripwire-bin-md5sum.txt		# eject floppy when done!


cd $TRIPWIRE-CF		# Tripwire config files, eg /usr/local/tripwire/etc
chmod 0600 tw.cfg* tw.pol*

mv twcfg.txt* twpol.txt* /floppy	
	# move text config and policy file offline, eject floppy when done!


cd $TRIPWIRE-DB		# tripwire DB, eg /usr/local/tripwire/var/db

md5sum * > db-md5sum.txt
cp db-md5sum.txt /floppy		# eject floppy when done!

chmod -R u=rwX,go-rwx $TRIPWIRE		# eg /usr/local/tripwire



updating twpol.txt:

/home  -> $(Dynamic) ;

There maybe ref specific to given OS/Distro that may need to be updated acordingly.
eg /var/lost+found may not exist if it is not a dedicated partition.
/etc/mail/statistics is probably no longer used, etc

AIDE

snort

Network Testing

ttcp, speed performance test for tcp & udp

receiving comptuer:
java ttcp -r
java ttcp -r -l 4096 -n 100     # 4096 bytes buffer, 100 of them.
java ttcp -r -l 32768 -n 4096

Sending computer:
java ttcp -t 10.215.2.124


args: (try these in receiving computer)
-l 		= buffer size, 			def 8192, try 327768
-n 		= num of buffer to xfer, 	def 2048,, try  4096  ==> gives 128 MB xfer.

java version doesn't seems to suppport these:
-u		= udp test
-b 		= change system buffer size.
-v		= verbose, more stat
-d 		= dbg

----

various port avail.
linux rh come with a package
but seems rather old and no central org support.

http://www.netcordia.com/network-services.html