The Tlog - a technology blog » Systems administration

Goodbye, Apache; hello, lighttpd!

Pedro Timóteo — Thu, 12 Jul 2007 22:10:48 +0000

Fed up with how Apache memory usage grows (and grows, and grows), I’ve changed all of my sites on my external server (where, for instance, this very blog is hosted) to lighttpd, a.k.a. Lighty.

I began by changing the most problematic site to Lighty (listening on port 81), and using Apache’s proxy module to redirect it there. After the results were promising, I went and changed each site at a time, dealing with the particular problems of each (I use lots of redirects, and the syntax is a bit different, and, furthermore, Lighty doesn’t support .htaccess files.

WordPress was relatively easy (just one line). MyBB would have been even easier (nothing to do), if not for the fact that I use an SEO mod which uses an .htaccess file for nicer URLs. But everything was easier than I expected.

After each site had been “moved”, it was just a matter of stopping Apache and moving Lighty to port 80.

Memory usage is way down, and so is swap file usage (basically, it’s not being used, and it was, before — a lot). Barring any future problems, I’m quite happy with this set-up, and would advise this change to anyone who’s never tried anything other than Apache on an Unix system. I’ll probably try doing the same thing on my home server, too.

Why I’m not a Sysadmin anymore

Pedro Timóteo — Mon, 23 Apr 2007 16:45:14 +0000

I have worked as a sysadmin (mostly Unix / Linux) for most of my professional life (not right now, though), and I’ve been meaning to write a few thoughts about it for a while.

My experience is that working as a sysadmin is, to me, interesting and fulfilling on a technical level, but ultimately disappointing and frustrating on a career and personal level. Why is that?

Let’s say you’re a good, competent sysadmin, and you’ve just joined a new company. During the first few days, you get acquainted with the company, the department, the sysadmin team (if any), the network, and the servers. Soon, any technical problems the company suffers from become apparent. Maybe a particular service is too slow, there have been security problems in the past, a server or application crashes often, there is some network congestion, that server’s logs tend to fill up the entire drive and need to be deleted from time to time, and so on. Or maybe you spot a need for something the company doesn’t have: a caching proxy server, an anti-spam / anti-virus email gateway, etc..

So, you get to work on those problems. Some software upgrades here, some tuning there, some cron entries here, some scripting there, some changes to the network, and so on. In months — maybe weeks, if the company is small — all the problems are mostly solved, and everything runs smoothly. Sure, you still have to reset users’ passwords (they keep ~~losing the Post-Its~~ forgetting them), keep software versions up to date (at least concerning bug fixes or newfound security holes), and, since you’re not dead and therefore haven’t stopped learning, maybe you later realize how a redesign or change of some particular server or software application can make things even better.

But, for the most part… most of your job is done. In the Unix world, with a decent knowledge of scripting and a good deal of experience, you can make your servers almost administer themselves, and you will be warned (by scripts) of potential problems in advance, so that they never actually happen. So… what now?

Now, you have a problem… especially if you’re an honest person. Because managers — and this has been my experience almost everywhere I’ve been — still tend to measure an employee’s work — and worth — by how busy he looks. Many people, then, simply pretend to be busy all the time (“change your email password? OK, I’ll get back to you next Monday.”), but such an attitude may be repulsive to you (it is to me). Explaining things to your manager doesn’t really work; even if he begins to understand, his own bosses won’t, and, if some head must roll, better yours than his…

So, after solving the company’s problems, and assuming you refuse to act busy when you’re not, what next? Well, you’ll get a reputation for laziness, for not “working” all the time, when everyone else does it (even if they’re just faking it). You’ll probably get assigned, in addition to your “proper” work, all the dumb, repetitive, non-sysadmin (and therefore non-scriptable) tasks — which, since you have free time, you probably can’t refuse, or at least feel you can’t. Any raise or promotion will certainly not go to you, but to your “hard-working” co-workers, who are always so “busy” and have so much “work” that they stay at work every day after 6, that they can never do a task “right now”, but only in a week’s time, and that, even their own results are much inferior to yours, it’s you who’re not “dependable”, “dedicated”, or “competent”.

Which is why I think it’s time for a career change. I currently work at home, in personal projects, but I’m probably going to look for a new job soon (for the extra money, and for learning something new “on the job”), and I’m thinking of programming, probably in PHP. I love the idea of creating something, instead of just making existing things work. And of (hopefully) being measured by results, not by how busy I look. I’m not a PHP “expert” (far from it), but I learn quickly, and I love to learn — even at 32 years old. Stagnation is always bad (though it seems that’s what most people seek in a job, especially in Portugal, oddly enough — learn a couple of skills, then do exactly that for the rest of your life), and, paraphrasing Duke Leto Atreides, a person needs new experiences… and new challenges.

Adventures with moonmoon and tidy

Pedro Timóteo — Tue, 06 Feb 2007 12:52:33 +0000

As I’ve mentioned here before, for Planet Atheism I’m using moonmoon, mostly because 1) everyone else uses planetplanet, and 2) it’s in PHP instead of Python, and I know a little PHP.

moonmoon is still on version 0.2, however, and, while it removes “dangerous” tags from feeds automatically, it doesn’t (yet?) deal with unclosed tags. As most of PA’s members are as far from being geeks as possible, they tend to use WYSIWYG editors, and aren’t really worried about “validating HTML”. So, from time to time, a post would make every other post after it show in bold or italic. Annoying, to say the least.

Yesterday, it was even worse: some posts “spilled over” to the sidebar. And it wasn’t just one post causing it, but two, from different blogs, at the same time!

Well, enough was enough.

“Fixing” moonmoon (or, more precisely, SimplePie) was out of the question; I simply don’t know enough PHP / XML parsing to do it. But I tried something else: I saved the generated HTML to a file on the site’s directory, and used tidy on it. Surprise: this new version was perfect! So, I got the idea of using tidy on the generated HTML every time.

Now, PHP has a tidy module, but in PHP5 I would have to compile PHP by hand. Ubuntu doesn’t have a package for that module, unfortunately, and I really didn’t want to make an exception from using apt packages on that server. So, I had to find another way.

My solution was to dump all the page into a buffer (using the ob_ functions in PHP), save it to a temporary file, use the system command to apply tidy on it, load the altered file, and show it to the browser. It’s probably not very efficient, but it works… better than I expected, too. It may be a crude solution, but I’m proud of it anyway.

Work: why a good sysadmin has a lot of free time

Pedro Timóteo — Mon, 12 Sep 2005 12:24:53 +0000

I’ve talked about this subject before (in the “keeping busy” entries), and I think I mentioned this in passing, but I believe that this is an important point, and deserves its own article.

My theory (which observation seems to validate) is that the better a system administrator is, the more free time he will eventually have.

Many people (including managers, team leaders, etc.), unfortunately, equate “free time” or “not working hard all the time” with “laziness”, and wrongly believe that a good worker is one who is working hard all the time – if he extends it to after work hours and weekends, even better.

Unfortunately, if they thought a couple of minutes about it, they might spot the huge, glaring error in such “logic”…

You know the “give a man a fish and you feed him for a day, teach him to fish and you feed him for a lifetime” saying? In systems administration, it’s similar. An average sysadmin fixes a problem. A good sysadmin not only fixes it, but he understands what happened, what caused it, and solves the underlying cause so that it doesn’t happen again in the future. Ideally, every problem only happens once.
Many (though not all) sysadmin jobs can be either partially or completely automated. Doing so requires skill and experience, something an average sysadmin may not have. But it saves a huge lot of effort and time. An average sysadmin works, a good one scripts, so that most of the work performs itself.
Systems administration is not a particularly “hard” work (though it’s a very specialized one), like, for instance, farming, in which effort and perseverance are the most important qualities. Instead, what makes a good sysadmin is: intelligence, quick learning, imagination, and, above all, experience. Those qualities tend to make a good sysadmin a very fast worker.
A sysadmin doesn’t just solve problems; he also researches, tests and implements new things. True. But even those don’t tend to take up all available time (except at particular situations, such as the final part of a project), because no “sane” company replaces their systems every month (unfortunately, there are a lot of “insane” companies out there, but that’s another story).
Unlike an average sysadmin, a good one has very efficient monitoring and warning systems. An average sysadmin tends to be “warned” of problems by the users – “this service is down”, “I can’t access my email account”, and so on. A good sysadmin, however, was warned about that abnormally growing log file a long time before it was even close to taking up all free space in its partition. And he fixed the cause of that growth long before it affected any services.
Managers tend to love a worker who works 7 days a week, stays in after hours, and so on. But what does that mean? Either that that worker is slow and inefficient, or that he’s doing the work of 3 persons or more. If the former, they should reconsider his value; if the latter, he’s the one who should reconsider his employment…
A good sysadmin is a master of efficiency and time saving. As I said, he scripts, he automates, he solves problems “once and for all”. He investigates and deploys his own tools for quick remote administration, which are much faster than going to the physical location of a server, for instance.

See the pattern?

Unfortunately, there’s a problem here: visibility. A good sysadmin is like an efficient police force. If citizens constantly see the police running after criminals, in spectacular car chases, they’ll probably be impressed, but a really good police force would mean that most crimes would be prevented in some way. Instead of a police which goes after criminals, there would be an apparent lack of crime. Sadly, that tends to make people start to believe that maybe we don’t need such a well-financed, well-staffed police force…

Here, it’s the same thing. Think of the crash from a rapidly growing log file I mentioned above. With an average sysadmin, it’s “I can’t access my mail!”, then he goes, deletes the file, restarts the service, and it’s working again. With a good sysadmin, nothing would have happened. Nobody would have noticed anything. So, they start to think that maybe that sysadmin doesn’t do a lot around here… maybe he isn’t pulling his weight.

That’s ridiculous, though – that one may have to do his job worse than he’s capable of, just so that people notice his work. It’s something I refuse to do. But maybe some “self-marketing” would be useful – like, when you solve a problem before it happens, you tell your superiors about it. It’s a bit absurd… but I think it’s better, and more honest, than the alternative (doing a bad job on purpose).