Cheshire Cat Computing

Steve,

Thanks for the great book. I used it to setup successfully some complicated monitoring, without previous experience. Great job.

On to the problem.. I am an old horse at windows trying to feel my way in the *nix world.

I installed mrtg and routers.cgi in a new Ubuntu 8.10 machine. All worked fine for a few weeks, as I played with your book and added more markers.

I then decided to install smokeping. The Synaptic package manager forced me to install mod_perl and speedy along with it, not that I could care less about accelerators in a test/learning machine.

After the smokeping install (that went fine, if relevant), routers.cgi started to show strange behavior, with some screens blank, and others only partially filled. I saw the note on your FAQ about that, and did the two steps you suggest to use with speedy (shebang line pointing to speedy, cache=y). Again, I don't care about speedy, I just want my old routers.cgi screens back.

I was still getting blank screens, so I moved routers2 back to the Perl call in the shebang line, and cache = no. Not sure if it matters, but in a desperate attempt to get rid of speedy.cgi, I altered the shebang line of smokeping as well, to call perl, not speedy. None of those helped. There is no other CGI software loaded in the PC other than those.

I ask for advice on what to do next. I am thinking of removing speedy (sudo apt-get remove), but am not sure about mod_perl. I am too green with the OS to know what to look for. Is there a way I can make router.cgi whole again, without removing smokeping.

Thanks for anyone who can clarify.

Fernando

I found out more about the problem, wanted to provide the extra details.

First, it is not mod_perl -- I removed mod_perl through apt-get remove; rebooted the computer. Still getting the blank screens. I cannot remove speed-cgi, or apt-get will take smokeping away too. Since speed-cgi is neither in the shebang anymore not cache is enabled in routers2.conf, I suspect it may be a different cause

The problem is random on showing up, but seems to be centered on the daily graph only. When routers.cgi shows only the blank screen, I noticed the graph files themselves are built OK, and are in the proper folder - but are not displayed on the browser.

Enabling debug - When debug was enabled, there was no output at all (including the debug output) on the daily graphs,if they were blank. There was debug output when the daily graphs were displayed partially. I pasted that debug output here, in the hope that it might clarify what is broken.

Thanks again for any help.

Fernando

**
<!-- rrdtool graph /var/www//graphs/hp-switch.cfg-192.168.1.254_2-d-x2.png --imgformat PNG --base 1000 --lazy --interlaced --force-rules-legend --slope-mode -e 1228347900 -s end-2000m -u 0.1 -w 800 -h 200 --x-grid HOUR:1:HOUR:24:HOUR:2:0:%k --title #2 -- HP-Switch --vertical-label traffic in Bps DEF:in=/var/www/mrtg/192.168.1.254_2.rrd:ds0:AVERAGE DEF:out=/var/www/mrtg/192.168.1.254_2.rrd:ds1:AVERAGE CDEF:fin=in,8,* CDEF:fout=out,8,* DEF:min=/var/www/mrtg/192.168.1.254_2.rrd:ds0:MAX DEF:mout=/var/www/mrtg/192.168.1.254_2.rrd:ds1:MAX CDEF:fmin=min,8,* CDEF:fmout=mout,8,* CDEF:pcin=fin,100,*,1000000000,/ CDEF:mpcin=fmin,100,*,1000000000,/ CDEF:pcout=fout,100,*,1000000000,/ CDEF:mpcout=fmout,100,*,1000000000,/ AREA:fin#00cc00:Incoming traffic LINE

If you are using mod_perl or speedycgi, then you should enable cache=yes in the routers2.conf in order to take advantage of this.

The error you're getting sounds as if routers2 is crashing out during the page generation. If this case, check the web server error log ( /etc/httpd/logs/error_log under redhat linux ) and see if there are messages in there.

Possible causes could be a conflict in the RRDs perl modules - installing smokeping MAY have installed a separate and different version which is conflicting with your previous version. In this case you'll get crashes in Perl when it loads the RRDs.so module. Signs of a crash would be that the screen is partially produced. The graph images are generated halfway through generating the screen so they may come or not.

Someone else reported this sort of issue, and it turned out to be an RRDs.so version problem causing a Perl crash.

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Thanks for your reply, Steve.

I followed your instructions and found errors reported in the apache2 error.log file, Since the same errors are repeated for a number of times, I edited so only one of each was pasted below. Some were quite repeated, for instance the error on line 3470 goes on for a few screens.

Thanks for the tip regarding an RRDs module conflict. I searched for the file RRDs.so in the disk, and found only one instance in location /usr/lib/perl5/auto/RRDs. I am not sure if the fact of having just one of these files will rule out a conflict or if there could still be a conflict.

To try to see what release of RRD I have, I typed <rrdtool> at a shell prompt. It came back as RRDtool 1.2.27. I do not know if that info will help rule out (or confirm!) a release conflict between that version of RRDs and routers.cgi 2.19, that I am running here.

Sorry, I am new to this. I feel as I should have more data ready. Let me know if you need other info and I will jump to get it.

Thanks,

Fernando


**
Below is the apache error log, heavily edited to avoid repetition of the same messages
**
*** attempt to put segment in horiz list twice
[Thu Dec 04 23:10:46 2008] [error] [client 192.168.1.7] (104)Connection reset by peer: ap_content_length_filter: apr_bucket_read() failed
Attempt to free unreferenced scalar: SV 0xa3d53d8, Perl interpreter: 0x9d4c008 at /usr/lib/cgi-bin/routers2.cgi line 3470.
Attempt to free unreferenced scalar: SV 0xa381418, Perl interpreter: 0x9cf8008 at /usr/lib/cgi-bin/routers2.cgi line 3470.
[Thu Dec 04 23:19:16 2008] [error] [client 192.168.1.7] (104)Connection reset by peer: ap_content_length_filter: apr_bucket_read() failed

This actually looks like a bug in your Perl interpreter (!). It is seg faulting due to trying to incorrectly free memory, but the script doesnt do this, it is the Perl garbage collector which takes care of it. So Perl is choosing to make this mistake itself... Perl itself doesnt even have a 'free' command as all variable freeing is done implicitly on function exit.

Can you give your exact perl version? IE, 'perl -V' and also 'uname -a'. This might not help but I can at least compare things.

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

Steve.

>>>This actually looks like a bug in your Perl interpreter (!)

I am glad you said that, I was going to try to update rrdtool (current 1.2.27) by myself without the crutches of synaptic (makefile, etc), something I have not done before and was not looking forward to, now that there is a lot to lose if I blow up the box. Have you also written a book on that .

I pasted below the output of perl -V and uname -a. Thanks for the big hand,

Fernando

**
user@user-desktop:/usr/lib/cgi-bin$ perl -V
Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
Platform:
osname=linux, osvers=2.6.24-15-server, archname=i486-linux-gnu-thread-multi
uname='linux palmer 2.6.24-15-server #1 smp tue apr 8 01:08:17 utc 2008 i686 gnulinux '
config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des'
hint=recommended, useposix=true, d_sigaction=define
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64',
optimize='-O2 -g',
cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -pipe -I/usr/local/include'
ccversion='', gccversion='4.3.1', gccosandvers=''
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld='cc', ldflags =' -L/usr/local/lib'
libpth=/usr/local/lib /lib /usr/lib /usr/lib64
libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=/lib/libc-2.8.90.so, so=so, useshrplib=true, libperl=libperl.so.5.10.0
gnulibc_version='2.8.90'
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib'


Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_ITHREADS
USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API
Built under linux
Compiled at Jul 24 2008 08:43:55
@INC:
/etc/perl
/usr/local/lib/perl/5.10.0
/usr/local/share/perl/5.10.0
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.10
/usr/share/perl/5.10
/usr/local/lib/site_perl
.
user@user-desktop:/usr/lib/cgi-bin$ uname -a
Linux user-desktop 2.6.27-9-generic #1 SMP Thu Nov 20 21:57:00 UTC 2008 i686 GNU/Linux
user@user-desktop:/usr/lib/cgi-bin$

Perl 5.10 eh? I've only tried this under Perl 5.8. I'll need to ask Tobi if he's tested the RRD Perl module under Perl 5.10...

I'm not really sure what I can do about testing this as I just dont have the hardware and setup to do much testing.

I'd suggest you try to install RRDTool 1.3.4, in case the problem is an incompatibility between the RRD module and perl 5.10 - its more likely to work in the latest branch as Tobi will be working on things there.

Installing RRDTool from source isnt too hard, but you need to have the requisite dev libraries and make sure you remove any old RRDs.pm and RRDs.so in your Perl path first.

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

>>>Perl 5.10 eh? <SNIP>
>>>I'd suggest you try to install RRDTool 1.3.4 <snip>

Indeed Sir, 5.10 it is. I wish I could remember if the release came in the Ubuntu 8.10 distro, or if it was updated later. The reason is router.cgi worked fine for a few weeks. I checked the logs, but found no reference to that.

I will try to install from rrdtool source. Another approach would be roll back to 5.8, - I did that in Windows at work. In Ubuntu, synaptic offers only 5.10 and if I am green and chicken to install rrdtool from source, Perl is likely not going to be easier

Thanks for the help, Steve. I will let you know what happened after the RRDTool update.

Fernando

Horray ! I was able to install rrdtool 1.3.4 from source, and run both mrtg and routers.cgi using the new rrdtool.

I got this error:
Dec 8 23:45:30 user-desktop kernel: [215383.363331] routers2.cgi[29420]: segfault at 322e313a ip 080b83e5 sp bfe4cf20 error 4 in perl[8048000+134000]

Also, the problem can be reproduced at will now.

It is always the daily chart that breaks.Inside the daily chart, *the stretch graph always works*.The other chart sizes in the daily chart may work or not. If the day does not have a lot of data, it seems to work more often than it breaks.

I downloaded 14all.cgi, as a test, and it worked OK with the new rrdtool-1.3.4-generated rrd files. With immense respect to the author of 14all, who gave generously his time, I would prefer if router.cgi were working.

Thanks for any pointers. If it helps, I can open the web page to the Internet, so the error can be seen in real time.

Fernando

**
Note:
I think I installed the new rrdtool right, but here are the steps. After d/l and compilation, I added these two lines to the mrtg config file:

PathAdd: /usr/local/rrdtool-1.3.4/bin/
LibAdd: /usr/local/rrdtool-1.3.4/lib/perl/5.10.0/i486-linux-gnu-thread-multi

ThenI moved the old RRDs.so from deep in the Perl lib to a non-path location. When I ran mrtg, it yelled this:
"WARN: found several copies of RRDs.pm in your path: /usr/local/rrdtool-1.3.4/lib/perl//5.10.0/i486-linux-gnu-thread-multi, /usr/lib/perl5 I will be using /usr/local/rrdtool-1.3.4/lib/perl//5.10.0/i486-linux-gnu-thread-multi. This could be a problem if this is an old copy and you think I would be using a newer one!"

Looks like you've compiled and installed RRDtool 1.3.4 correctly (and much thanks to you for going through all this to help track down the issue).

It looks like there's an incompatibility in the RRDs Perl library with Perl 5.10. I've reported it to Tobi but of course he needs more details (and he uses 5.8 as well...)

Since the stretch graphs work, and so do the (admittedly simpler) graphs generated by 14all, I think it is probably one of the more complex features used on occasion by Routers2 that is tickling the bug in the module and causing the Perl crash.

Obviously this is a real bugger to track down the cause of. From what you say it may be the autoscaling routines at fault but I'm no expert on the RRD internals, sadly. I'll pass the info on to Tobi in the hope that he can get more out of it...

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11