Friday, December 23, 2005

nntp upgrade

Tonight we rolled out the upgrade of our nntp server ( to colobus 2.1. Previously we were running a very old, somewhat hacked up version. Now we're running a new, only slightly hacked up version. (We had to make some changes for performance and quantity -- all sent back to the author of course.) You shouldn't notice any difference -- the speed seems about the same. The new version is backed by MySQL, which conveniently gives us easy access to the overview database (which contains threading information.) This will let us do cool things in the new web interface (which has been "Coming soon" for way too long.)

Thursday, December 22, 2005

Full Duplex

Tonight I went to the datacenter and changed our incoming internet pipe to full duplex mode. This will give us a few more usable megabits of bandwidth, making it much less likely that we'll max out our bandwidth. It also cuts off a few milleseconds of latency.

As always, I need to thank the wonderful support staff at InterNAP for making everything so easy! I called, and within 10 minutes we had the link forced to full duplex with no interruption of service.

Why were we in half duplex mode in the first place? Good question. Their side was set to autonegotiate as was ours. Maybe it's the media converter? Maybe its our switch? Setting our switch to full duplex (and leaving theirs on auto) only made things worse. But now both sides of the link are happy, I'm happy, and our users should be happy too!

Wednesday, December 14, 2005

Wednesday, December 7, 2005

RT / perlbug down today

So the issue from yesterday didn't get sorted out last night. I've been working on it today and the database is happy again. Just waiting for Robert to startup RT again. :-)

Tuesday, December 6, 2005

RT / perlbug issues

MySQL on our main database server (we were running an old-ish 4.1.x version) crashed and corrupted some (unrelated to the crash, I think) InnoDB tables.   We did run it with some vaguely aggressive settings, but it shouldn't lead to corruption (only losing a few updates).   Grrh!

Speaking of the database server, AMD are giving us at least one of their servers; we don't really know too much about it except that a package is supposed to arrive at my house.  One of our planned uses is a new database server which should speed up RT and several other services.

From:   Robert Spier
Subject: Perlbug Email Down, Web Interface Unstable
Date: December 6, 2005 9:52:20 PM PST

Dear p5p, p6i,

We're having some database issues.

In order to make my life easier when I try and recover from them, I've disabled incoming email to the RT instance.  (It will be queued up and
delivered later.)  You may also find the web interface to be unstable
as the database decides to disappear mid-stream.

I will be working on this tomorrow, and hope to have everything back

to normal within 24 hours.

(Details: it appears that somehow, we've made innodb unhappy.

Tomorrow, I will be doing a lot of dumps and restores to try and

straighten it out.)


update 1am PST: All seems well again. Unless it goes bonkers again during the night Robert is going to turn processing of the RT emails back on in the morning. - ask

Perl Foundation Weblog

Apologies if you already saw this on use.perl, my personal weblog or one of the soon bazillion other places it's been posted.

Yesterday we announced the Perl Foundation Weblog. Robert and I will post there occasionally too. I'm not sure yet if we'll just take some posts from here or if we'll split it up and leave this weblog more for just operational stuff and have news and status reports there.

Monday, November 28, 2005

CPAN / FUnet update

It's not something we (Robert, Graham & Ask) are in control of, but if you were wondering why hasn't been getting new module uploads quickly lately it's because of problems with the hardware running the master CPAN mirror.  It's making all of CPAN "not move" so to speak.

PAUSE is still running and taking uploads and no data is being lost anywhere, just a little delayed.

Last I heard it should get better sometime this week.

Sunday, October 16, 2005

New search engine

For a long time Search CPAN has used the WAIT module to provide searching of POD documents. Today we started testing the use of Swish-e. Try it and let us know what you think

Monday, September 12, 2005

power is back...

The power is back and the console server came up. Now for trying to sort out everything else, hopefully without having to drive there.

update: alright, I got everything beat into running again except for the mailing list server. A few of the other boxes (on really really really old hardware) had a hard time getting started too, but after an extra power cycle and a manual fsck I got them going. x6, the list server, isn't as responsive though. Grrrh.

On the radio they're talking about "45 minutes traffic" from Hollywood to downtown on the 101 (when there's no traffic it takes just 10-15 minutes) so I'm hesitating a little bit running down there to look at it, but I will have to go soon as I can't be holed up down there for too long tonight.

update 2, much too much later: The mailing list mails are flowing again... (and the console redirection on that server is working again; one of the harddrives in the RAID not so much. grrh).

Power outage in Los Angeles, down

Apparently there's a Really Big power outage here in Los Angeles.

At my house the power flicked enough for the UPS to click in and for things not on the UPS to reboot, but there's power just fine now so I didn't think anything of it.

Ask: Circuit not working. Can't breathe!

Internap NOC: Los Angeles? Big power outage there. Traffic signals not working, police on "full tactical alert" and stuff.

Ask: Uh-oh! Okay.

Ask: Generators?

Internap NOC: They said they came on but shut down due to overload.

Ask: Doh.

Our servers are in the same building but on a different floor, so it's entirely likely that they are affected too. Grrh. I'll have some fun getting everything running again when the power comes back down there (Robert is traveling today).

Thursday, September 8, 2005

Datacenter day - update

That went mostly well. We got two dead servers assembled into one that'll probably work, or maybe not. It was installing slower than slowly over the network. Robert set it up to redirect the BIOS to the serial port and we hooked it into the console server so he'll try installing it from home later. We also hooked the new switch up to the console. Cyclades++.

We got all the disks in the new RAID working. It'll be very very cool when we start actually using them. The hardware RAID-5 with the 120GB disks we are using now isn't performing well at all.

While taking disks out and checking the connectors Robert pulled the disk we thought was an extra extra hot spare for the existing RAID only to minutes later realize it had a swap partition on it. Ooops!

MySQL was apparently swapped a bit because it crashed with a really really long dump to the log and some corrupted MyISAM indexes. All the InnoDB tables came up fine though. When InnoDB first came out I was much more comfortable with MyISAM because how simple it was. I'm only going to use MyISAM in the future when the particular performance characteristics of it are needed.

Datacenter day

Robert and I are in the data center today (afternoon PST) replacing a bad disk or two and moving some equipment around.

Some of the services we run might be unavailable for a few moments once or twice, so don't be alarmed.

We have monitoring setup to alert if when things are not working, but if something is not working by the end of the day (~1am UTC), please let us know. :)

update: over here....

Sunday, September 4, 2005

Create your own "BackPAN"

If you didn't know, "backpan" is all the PAUSE uploads but with no deletions (PAUSE, by Andreas K├Ânig, is the Perl Authors Upload SErver which is the way the vast majority of content enters CPAN now).

Robert and I run a backpan at and Elaine is running an equivalent one at

We could use more though! It doesn't have to be public, but it'd be nice to know that we had a few more backups of old old CPAN distributions.

First mirror CPAN if you aren't already. Once a day is fine. Also once a day then run something like the following:

rsync --exclude CHECKSUMS -vrptgx ~/mirror/CPAN/authors/id/ ~/mirror/backpan/authors/id/

to copy new files into backpan. Notice there's no --delete argument to that rsync command.

Thursday, August 25, 2005

CPAN Ratings updates

Spent a couple hours this afternoon on CPAN Ratings and fixed and cleaned up a bunch of little things.

One of them was adding a link to the RSS feed. I didn't realize we forgot that when we added it. Oops! :-)

Of course I might have broken something too. Let me know if you see anything not working so well.

It's great that so many people have been "rating reviews" (marking them helpful or not). Next time I work on the site I'll make it do something useful with that data (besides just showing it next to each review).

Sunday, August 14, 2005

Link exchange requests

The webmaster email address gets link exchange requests from time to time. Today, we got a great one...

I have found your website by searching Google for "greenzap scam". I think our websites has a similar theme, so I have already added your link to my website.

Needless to say, we don't respond.

update: check out the comment this entry got. Hilarious. (I deleted their link from it though...) - ask

Monday, July 25, 2005

Thanks To Thalasar

Special thanks to Brian Despain of


for buying us some new hard drives.

We're using the drives as part of our extended disk array to help improve performance and do better backups. It's very cool having a terabyte of disk to play with.

Please visit to learn about what Brian is up to with his open-source company.

Wednesday, July 20, 2005

I fought glibc and won

This morning I woke to a few strange emails from Ask and a ticket that the Parrot TODO list was broken. The gist of it was: getprotobyname was causing our production mod_perl's to crash. (See entry from earlier today for a stacktrace.)

Obviously, not a good thing.

Question One: What changed? Neither Ask nor I could remember changing anything that *should* have affected this recently. The easy solution (revert) was out.

Step one: Reproduce and isolate. This turned out to be the easy part. I configured a bare-bones Combust with only a single website configured with only a single controller:

package Test::Control::Test;
use base 'Combust::Control';
use DBD::mysql;
use LWP::Simple;
sub handler ($$) {
my ($self, $r) = @_;
my $output = LWP::Simple::get("");
$self->send_output(\$output, 'text/html');

Without the use DBD::mysql, everything was fine. With it, KABOOM.

Step two: Debug.. By telling apache not to fork, it was easier to track things down. /pkg/apache1/bin/httpd -X -f /home/robert/minisite/apache/conf/httpd.conf GDB wasn't particularly helpful. It got me the stack trace, and told me it was happening during DSO symbol lookup. (Smells like glibc!). Hrm, might be a problem with something scribbling over memory. Lets try Valgrind. No luck, the older version of valgrind we have on the system bombs out.

Ask found this RedHat Bugzilla ticket. It is a similar problem, but didn't get resolved. It did lead me closer to the solution. As suggested in the ticket, I ran my apache with all of the dynamic loader debugging enabled: LD_DEBUG=all LD_DEBUG_OUTPUT=/tmp/some-file . The copious (16MB) output ended like this:

4702: symbol=_nss_files_parse_protoent; lookup in file=/pkg/packages/apache-1.3.33/libexec/
4702: symbol=_nss_files_parse_protoent; lookup in file=/pkg/packages/apache-1.3.33/libexec/
4702: symbol=_nss_files_parse_protoent; lookup in file=/lib/
4702: symbol=_nss_files_parse_protoent; lookup in file=/lib/

It should have looked something like this: (from earlier in some-file)

4702: symbol=strlen; lookup in file=/pkg/apache1/bin/httpd
4702: symbol=strlen; lookup in file=/lib/tls/
4702: symbol=strlen; lookup in file=/lib/
4702: symbol=strlen; lookup in file=/usr/lib/
4702: symbol=strlen; lookup in file=/lib/
4702: symbol=strlen; lookup in file=/lib/tls/
4702: binding file /pkg/packages/apache-1.3.33/libexec/ to /lib/tls/ normal symbol `strlen' [GLIBC_2.0]

So, we now know that the problem is the dynamic linker is having trouble binding the symbol _nss_files_parse_protoent (which lives in /lib/ and that has something to do with DBD::mysql. We also know that (the C portion of DBD::mysql) is linked against libnss_files. (See ldd output or information from the some-file.)

That struck me as odd, so I attempted rebuilding DBD::mysql to see why it was linking against the nss libraries. That's generally something that should be sucked in by libresolv. Definitely, a general purpose application shouldn't be linking against specific nss ("Name Service Switch") libraries.

Turns out, DBD::mysql was getting the information from mysql_config.

--libs [-L/usr/lib/mysql -lmysqlclient -lz -lcrypt -lnsl -lm -lc -lnss_files -lnss_dns -lresolv -lc -lnss_files -lnss_dns -lresolv]

Whoa! Duplication, redundancy, extra libraries, and explicit linking against libc. Definitely not something that most applications should do. I could understand that MySQL itself might need to do weird things - it's a complicated application - but things linking against it shouldn't have to.

Step three: Fix it.

--- mysql_config.old 2005-07-20 16:01:34.000000000 -0700
+++ mysql_config 2005-07-20 16:02:06.000000000 -0700
@@ -86,10 +86,10 @@
# Create options
libs="$ldflags -L$pkglibdir -lmysqlclient -lz -lcrypt -lnsl -lm "
-libs="$libs -lc -lnss_files -lnss_dns -lresolv -lc -lnss_files -lnss_dns -lresolv"
+libs="$libs -lc -lresolv"
libs=`echo "$libs" | sed -e 's; \+; ;g' | sed -e 's;^ *;;' | sed -e 's; *\$;;'`
-libs_r="$ldflags -L$pkglibdir -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -lpthread -lc -lnss_files -lnss_dns -lresolv -lc -lnss_files -lnss_dns -lresolv "
+libs_r="$ldflags -L$pkglibdir -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -lpthread -lc -lresolv "
libs_r=`echo "$libs_r" | sed -e 's; \+; ;g' | sed -e 's;^ *;;' | sed -e 's; *\$;;'`
cflags="-I$pkgincludedir -O2 -mcpu=i486 -fno-strength-reduce " #note: end space!

That's cheating. But it got the job done. After making that change, I rebuild DBD::mysql. By not explicitly linking
against -lnss_files, the internal magic of glibc could do the right thing and not blow up.

CPAN Ratings posting works again

Earlier this afternoon Robert got our DBD::mysql recompiled after some mysql_config hackery so it stopped making our httpd crash.

I've made another change to the site too now so you can post a "review" without giving the distribution a rating, effectively making it a comment.  Use that to comment on other peoples reviews, please. :-)

A dozen people have started marking reviews as helpful / not helpful.  It'll be interesting to see how that data set will grow over time.

Tuesday, July 19, 2005

CPAN Ratings broken #2

The slightly updated version of the CPAN Ratings site is up now.

It looks the same, but inside it got a decent house cleaning; and it's now using our new beta-ish authentication server instead of the old hack (we didn't call it the "ducttape API" for nothing!).

That's the good news.  The bad news is that the httpd is coredumping on "getprotobyname".

#0  0x00d5057b in do_lookup_versioned () from /lib/

#1  0x00d4f776 in _dl_lookup_versioned_symbol_internal () from /lib/

#2  0x00d53473 in fixup () from /lib/

#3  0x00d53330 in _dl_runtime_resolve () from /lib/

#4  0x006c76d4 in _nss_files_getprotobyname_r () from /lib/

#5  0x00203752 in getprotobyname_r@@GLIBC_2.1.2 () from /lib/tls/

#6  0x009945a9 in Perl_pp_gprotoent () from /pkg/packages/apache-1.3.33/libexec/

I hacked around it in a few places to make the site appear to work (and you can vote reviews as helpful or not helpful!), but you still can't add a new review.   I'd keep hacking on it, but it's 4.30 and have things to do wednesday so I can't stay up much longer... Zzzzzz...

<b>update</b>: it's working again. Uploads RSS 1.0 feed

Many people have not been ablt to find the RSS feed for CPAN uploads that search has been providing for quite sometime. The only link was in the FAQ. It was also only in RSS 0.91 and did not contain much information.

Well now there is a new feed at which is in the header of each page so discovery tools can find it. And there is a link on the recent upload page

Monday, July 18, 2005 broken is in a bit of a mess.  Posting new reviews has been broken for a while and I just super-broke it.   I was working on a few new features and while I had a separate branch for the code I forgot to do the same for the templates.  Oops.

Instead of reverting it back I'll just press on to get the new stuff out in the next few days.  Thank you for your patience. :-)

update: almost fixed.

Wednesday, July 6, 2005

Subversion, RT, CVS and mail (not anymore) down

Oops. Our ancient server that's hosting SVN, CVS, RT and does our incoming mail filtering crashed.

We have plans to get Subversion and CVS moved to a dedicated box, but we haven't gotten to it yet. (Likewise for Request Tracker).

There was no output on the terminal server, so I power cycled it and it's rebooting now. Slooowly.

/home: Clearing orphaned inode 1197164 (uid=27, gid=27, mode=0100600, size=0)

/home: clean, 189065/1359872 files, 2282241/2714977 blocks

/usr: recovering journal

/usr: clean, 74363/320640 files, 388588/640591 blocks

/var: recovering journal

/var: Clearing orphaned inode 160130 (uid=101, gid=102, mode=0100744, size=16775271)

/var: clean, 2854/192000 files, 269571/383551 blocks

<-------------reiserfsck, 2001------------->

reiserfsprogs 3.x.0j

/dev/hda1: recovering journal

/dev/hda1 has gone 268 days without being checked, check forced.

update 10:05 PST: it's a minute from being back up now, whee.

Friday, July 1, 2005

Search - help needed

Some weeks ago I made a simple page to search stuff on * The reason we haven't told anyone or even made a link to the page is that it's Really Ugly. See for example a search for DBI.

The template for the search is in this file, there's also a file with a few lines of extra CSS. Use your login to access those files. Patches more than welcome!

It's using the XML interface to CPAN Search for the CPAN search results and the Yahoo Search API (via Yahoo::Search). Oh, which reminds me – the search results page needs to mention it's using the yahoo search API.

Did I say that patches are welcome?

Saturday, June 25, 2005


Today (and yesterday), I spent some time trying to upgrade the Planet! software that runs Planet Perl and friends to the latest version.

I said "trying", because I'm declaring the upgrade a failure.

It appears to be because I'm testing by running two "update"s in a row, and it doesn't like rapid updates within the same minute or something, but it seems to randomly decide to only include certain people in the output. (And no, it's not always the same people.) That just isn't good enough. I could dive into the python code and fix it, but there are many other things on my todo list, and the old code works just fine, 99% of the time. It was also complaining bitterly about the RSS2.0 <items> block, so I had to tweak it for that.

Of course, instead of giving up, I got persistent, and it seems to be working ok now.

So... lets start over....

Tonight, I upgraded Planet Perl (and Parrot, and Six Perl) to the latest, greatest, Planet! Feed Aggregator.

While I was at it, I took the time to tweak the look of Planet Perl. (Thanks for the comments on the previous post.)I fixed up the CSS a bit, although it still needs more work. We're almost at the point where we can have alternate stylesheets. I did make one controversial change, which is to have it be full-width. The fixed-width we were using was a little too narrow, and picking another arbitrary width seemed silly.

Anyway, now I can leave this alone for a while.

Monday, June 20, 2005


I don't like how Planet Perl looks. There's too much content, its too vertical. And it doesn't help that PlanetPlanet is showing stories from last June.

Over the past few days I've played with trying to only show short summaries, alternate sidebars and fancy javascript controls. But I didn't find anything I liked.

In a few days I'll come back to this. Anyone got any new ideas?

Saturday, June 18, 2005

Planet Perl June Update

Welcome perlcast and Chip Salzenberg to Planet Perl.

I'll admit to not having listened to any of Josh McAdam's perlcasts yet, but they look cool. Chip has been writing up some very interesting things about Parrot recently.

Monday, June 6, 2005

More mail than you can shake a stick at

Today absorbed almost 30,000 spam and virus emails.

About 1/3 of that is viruses caught by clamav.

Plus at least another 12,000 invalid messages we never accepted. And I'm sure there's another huge bunch thats not getting logged properly

Then there's the huge portion of the internet that's outright blacklisted.

Without qpsmtpd, we'd be lost.

Wednesday, May 25, 2005


I rebooted a bunch of our servers earlier today to pick up new kernels. We have our own build of Perl, Apache and other software like that mounted via NFS and on a couple of the boxes the NFS directory didn't get mounted when it booted.

Also on one of the cpansearch boxes squid didn't start up on boot, and it wasn't in our monitoring setup so I only noticed just now because a user wrote in! Doh. (It worked half the time, depending on which server you got).

Our colobus server also didn't start up properly (or rather logging didn't), so the server has been flaky through the afternoon.

Anyway, it should all be better now. If not, send us mail.

Saturday, May 14, 2005


There's a barebones website for pgeodns now.

Robert moved our fancy DNS server out to a public repository, so it's availble from subversion if you are interested.

eww-tee-fff eight

Today some complaints came in about all the Perl Monger mailing lists switching to UTF-8. (As part of a fix for the encoding issues our Chinese group was having.)

So... I switched everything back to us-ascii, and created a special English (UTF-8) variant to use where necessary.

Thursday, May 12, 2005

EU mirror live

I flipped the switch, so is now using the EU mirror for European users.

(see the mirror post from yesterday)

Thanks again to Digital Craftsmen. We're also running one of our "geodns" thingies on their server now.

Wednesday, May 11, 2005

Subversive Ponies!

Ponie has moved from CVS to Subversion. You can now find it here. This leaves only a few small projects running off of our CVS server. mirror

Over the last week Graham and I have been setting up the box provided by Digital Craftsmen; our first mirror not hosted on the infrastructure. We are planning a mirror in Japan and another US mirror too (but haven't gotten the hardware for it yet).

It's not on the name yet, but (Very Alpha-ish) at If you are in Europe you should reach the new mirror, if you are anywhere else you should reach one of the hosted boxes (with the "hardware by bizrate" logo). The name will go away when we are done testing.

Please try it out and let me know if you are having trouble with it (email

Our geography-aware DNS server, pgeodns, is a bit naively assuming that all of Europe will have better access to the UK mirror than to the West Coast US mirror.

In the process of setting it up, I updated pgeodns a bit, so if I made a mistake or two it might impact the and names too, so please let me know if you are seeing any abnormalities with the DNS for those too.

pgeodns is in a private svn repository, but Robert says he'll move it to another one for me any day now. :-)

Tuesday, May 10, 2005

More disk!

More information and details later, but I wanted to share our new terabyte RAID with you:

[root@x4 root]# df -h /mnt/raid
Filesystem Size Used Avail Use% Mounted on
/dev/md0 1.4T 33M 1.3T 1% /mnt/raid

Monday, May 9, 2005

Data center day / new hardware

Tuesday afternoon is going to be datacenter day™ for Robert and I.

Our colocation facility asked us to get off the building UPS as the building apparently is running out of UPS power.  Since they are incredibly nice to us (they just gave us the neighbor rack so we have two racks there now!) we could hardly say no.  Last week we figured out what we need (I hope!) and ordered a APC SUA3000RM2U UPS that should have enough capacity.  The building has a generator that'll kick in very quickly so we only need a minimal run-time.  If it doesn't have enough capacity we'll get another one of those most likely.   The higher capacity ones gets a lot more expensive per watt as it quicky turns into some 208V+transformer mess.

Monday morning I heard the doorbell/barking dog duo and Vani came upstairs telling me there was a DHL guy with a heavy package that he was going to try getting down to the house.  I rushed and caught him going down the stairs so he could help me bring it up again and put it in the car.  The darn thing is 120lb!

Later in the day UPS showed up with a Dell 5324 gigabit switch (I mentioned that before) and Fedex showed up with more drives for the file/NFS/database/backup server (more on that later).

Tomorrow I have to go by the post office and pick up a 48-port 100Mbit switch I found on eBay and then we'll be off to the datacenter to setup the new rack.

Many or most of our services may be down briefly Tuesday afternoon (local time in California) as we shut down equipment to move it over to the UPS and install the new disks.

Sunday, May 8, 2005

Challenge Response for NNTP

A few weeks ago we enabled posting to mailing lists via Google Groups. In looking to expand that to the rest of the NNTP enabled world, we've settled on implementing a Challenge/Response system for verifying email addresses. (It really isn't fun when you reply on a mailing list and find out the email address you're replying to is fake.) Because we generally require posting from the subscribed address to the mailing lists, we know those are valid addresses.

So... Last night I attempted to implement TMDA. It is very easy to setup... but... I hit a roadblock. The envelope from/return path on NNTP gated postings is generally the gateway, not the original poster. So, TMDA had to be modified to use the From address instead. I had it mostly working after a few hours, alternately tweaking the scripts (something I didn't want to do -- makes it hard to upgrade) and writing procmail wrappers. Finally, I gave up, after TMDA made it clear that it wasn't happy if I sent a message from address A, and then sent the verification message from address B.

Next step is either to:

a) write my own Challenge/Response system just for this.

b) find another system that works

c) give up, letting the 3 or 4 people a wek who post via a NNTP server that is not send their emails directly to the mailing list.

Saturday, May 7, 2005


I upgraded mailman tonight to better support Chinese, as our new PM group was having issues with their mailing list archives.

While I was working on mailman, I also fixed some issues with periodic processes not being run. So, now you'll get reminders for your mailing list passwords, if you haven't turned it off in your account.

Wednesday, April 13, 2005

The console server rocks

Last fall we got a really cool terminal server from our generous friends at Cyclades. Oh, and super fancy powerstrips.

Do we use it a lot? Yes! Directly? No, rarely. Indirectly? All the time! In the past it was extra uncomfortable to change anything that could possibly be messed up in a way that'd bring the server down at a time where going to the datacenter would be inconvenient (say in the middle of the night or in the middle of the day with "real work" looming).

Tonight was one of those times where I used it directly. One of the servers had crashed with this message repeating over the "screen" (serial port).

rtc: lost some interrupts at 2048Hz.

rtc: lost some interrupts at 2048Hz.

rtc: lost some interrupts at 2048Hz.

rtc: lost some interrupts at 2048Hz.

rtc: lost some interrupts at 2048Hz.

What it means? I don't know. It's been doing it once or twice before over the last year. Ctrl-p and a "power cycle" command later the box was rebooting and back up. Whee. No datacenter visit for me. The people at IX2 (the datacenter) are great, but it's nicer not to go there too much; in particular at 3am. :-) (Yes, the datacenter people would have helped reboot it, but I'd rather not bother them and more importantly it's much better to be able to properly see what's going on before hitting the big button - virtual or otherwise).

Thanks Cyclades!

Sunday, April 10, 2005

Sunday, April 3, 2005

Post to the lists via Google Groups

In the past you have been able to read the mailing lists on Google Groups (and other news servers), but not post. You had to post by emailing the list or using

Robert got that sorted out yesterday, so we now experimentally (as in it might work it might not) support posting to the lists via Google Groups, but not yet other news servers.

design doc synchronization

Several people had noticed that the design documents at had gotten out of sync with reality.  This was a holdover from a CVS to SVN conversion of those design docs.

So... last night I hooked up some svn:externals magic, did a little reorganization, and the design docs will now be synchronized every night.

Saturday, April 2, 2005

You might already have seen this over here, but we are really happy to announce Jon Allen's wonderful new perldoc site that we are hosting.

Perl NOC Weblog

Whee! We've long needed a place for posting news and notices regarding the operation of the services we run over at We didn't want to run it with everything else because the whole point is to have a place that's always available.

Thanks to our friends at Six Apart we now have that place, whee. Thanks Ben and everyone else there!

We will post notices about scheduled maintenance, trouble with the servers and announcements of new features here now. (So no need to check our other sites anymore if that was all you went there for :-) ).

Be sure to subscribe to the rss feed!