Fed up with how Apache memory usage grows (and grows, and grows), I’ve changed all of my sites on my external server (where, for instance, this very blog is hosted) to lighttpd, a.k.a. Lighty.

I began by changing the most problematic site to Lighty (listening on port 81), and using Apache’s proxy module to redirect it there. After the results were promising, I went and changed each site at a time, dealing with the particular problems of each (I use lots of redirects, and the syntax is a bit different, and, furthermore, Lighty doesn’t support .htaccess files.

WordPress was relatively easy (just one line). MyBB would have been even easier (nothing to do), if not for the fact that I use an SEO mod which uses an .htaccess file for nicer URLs. But everything was easier than I expected.

After each site had been “moved”, it was just a matter of stopping Apache and moving Lighty to port 80.

Memory usage is way down, and so is swap file usage (basically, it’s not being used, and it was, before — a lot). Barring any future problems, I’m quite happy with this set-up, and would advise this change to anyone who’s never tried anything other than Apache on an Unix system. I’ll probably try doing the same thing on my home server, too.

I have worked as a sysadmin (mostly Unix / Linux) for most of my professional life (not right now, though), and I’ve been meaning to write a few thoughts about it for a while.

My experience is that working as a sysadmin is, to me, interesting and fulfilling on a technical level, but ultimately disappointing and frustrating on a career and personal level. Why is that?

Let’s say you’re a good, competent sysadmin, and you’ve just joined a new company. During the first few days, you get acquainted with the company, the department, the sysadmin team (if any), the network, and the servers. Soon, any technical problems the company suffers from become apparent. Maybe a particular service is too slow, there have been security problems in the past, a server or application crashes often, there is some network congestion, that server’s logs tend to fill up the entire drive and need to be deleted from time to time, and so on. Or maybe you spot a need for something the company doesn’t have: a caching proxy server, an anti-spam / anti-virus email gateway, etc..

So, you get to work on those problems. Some software upgrades here, some tuning there, some cron entries here, some scripting there, some changes to the network, and so on. In months — maybe weeks, if the company is small — all the problems are mostly solved, and everything runs smoothly. Sure, you still have to reset users’ passwords (they keep losing the Post-Its forgetting them), keep software versions up to date (at least concerning bug fixes or newfound security holes), and, since you’re not dead and therefore haven’t stopped learning, maybe you later realize how a redesign or change of some particular server or software application can make things even better.

But, for the most part… most of your job is done. In the Unix world, with a decent knowledge of scripting and a good deal of experience, you can make your servers almost administer themselves, and you will be warned (by scripts) of potential problems in advance, so that they never actually happen. So… what now?

Now, you have a problem… especially if you’re an honest person. Because managers — and this has been my experience almost everywhere I’ve been — still tend to measure an employee’s work — and worth — by how busy he looks. Many people, then, simply pretend to be busy all the time (“change your email password? OK, I’ll get back to you next Monday.”), but such an attitude may be repulsive to you (it is to me). Explaining things to your manager doesn’t really work; even if he begins to understand, his own bosses won’t, and, if some head must roll, better yours than his…

So, after solving the company’s problems, and assuming you refuse to act busy when you’re not, what next? Well, you’ll get a reputation for laziness, for not “working” all the time, when everyone else does it (even if they’re just faking it). You’ll probably get assigned, in addition to your “proper” work, all the dumb, repetitive, non-sysadmin (and therefore non-scriptable) tasks — which, since you have free time, you probably can’t refuse, or at least feel you can’t. Any raise or promotion will certainly not go to you, but to your “hard-working” co-workers, who are always so “busy” and have so much “work” that they stay at work every day after 6, that they can never do a task “right now”, but only in a week’s time, and that, even their own results are much inferior to yours, it’s you who’re not “dependable”, “dedicated”, or “competent”.

Which is why I think it’s time for a career change. I currently work at home, in personal projects, but I’m probably going to look for a new job soon (for the extra money, and for learning something new “on the job”), and I’m thinking of programming, probably in PHP. I love the idea of creating something, instead of just making existing things work. And of (hopefully) being measured by results, not by how busy I look. I’m not a PHP “expert” (far from it), but I learn quickly, and I love to learn — even at 32 years old. Stagnation is always bad (though it seems that’s what most people seek in a job, especially in Portugal, oddly enough — learn a couple of skills, then do exactly that for the rest of your life), and, paraphrasing Duke Leto Atreides, a person needs new experiences… and new challenges.

As I’ve mentioned here before, for Planet Atheism I’m using moonmoon, mostly because 1) everyone else uses planetplanet, and 2) it’s in PHP instead of Python, and I know a little PHP.

moonmoon is still on version 0.2, however, and, while it removes “dangerous” tags from feeds automatically, it doesn’t (yet?) deal with unclosed tags. As most of PA’s members are as far from being geeks as possible, they tend to use WYSIWYG editors, and aren’t really worried about “validating HTML”. So, from time to time, a post would make every other post after it show in bold or italic. Annoying, to say the least.

Yesterday, it was even worse: some posts “spilled over” to the sidebar. And it wasn’t just one post causing it, but two, from different blogs, at the same time!

Well, enough was enough.

“Fixing” moonmoon (or, more precisely, SimplePie) was out of the question; I simply don’t know enough PHP / XML parsing to do it. But I tried something else: I saved the generated HTML to a file on the site’s directory, and used tidy on it. Surprise: this new version was perfect! So, I got the idea of using tidy on the generated HTML every time.

Now, PHP has a tidy module, but in PHP5 I would have to compile PHP by hand. Ubuntu doesn’t have a package for that module, unfortunately, and I really didn’t want to make an exception from using apt packages on that server. So, I had to find another way.

My solution was to dump all the page into a buffer (using the ob_ functions in PHP), save it to a temporary file, use the system command to apply tidy on it, load the altered file, and show it to the browser. It’s probably not very efficient, but it works… better than I expected, too. It may be a crude solution, but I’m proud of it anyway.

I’ve talked about this subject before (in the “keeping busy” entries), and I think I mentioned this in passing, but I believe that this is an important point, and deserves its own article.

My theory (which observation seems to validate) is that the better a system administrator is, the more free time he will eventually have.

Many people (including managers, team leaders, etc.), unfortunately, equate “free time” or “not working hard all the time” with “laziness”, and wrongly believe that a good worker is one who is working hard all the time – if he extends it to after work hours and weekends, even better.

Unfortunately, if they thought a couple of minutes about it, they might spot the huge, glaring error in such “logic”…

1. You know the “give a man a fish and you feed him for a day, teach him to fish and you feed him for a lifetime” saying? In systems administration, it’s similar. An average sysadmin fixes a problem. A good sysadmin not only fixes it, but he understands what happened, what caused it, and solves the underlying cause so that it doesn’t happen again in the future. Ideally, every problem only happens once.
2. Many (though not all) sysadmin jobs can be either partially or completely automated. Doing so requires skill and experience, something an average sysadmin may not have. But it saves a huge lot of effort and time. An average sysadmin works, a good one scripts, so that most of the work performs itself.
3. Systems administration is not a particularly “hard” work (though it’s a very specialized one), like, for instance, farming, in which effort and perseverance are the most important qualities. Instead, what makes a good sysadmin is: intelligence, quick learning, imagination, and, above all, experience. Those qualities tend to make a good sysadmin a very fast worker.
4. A sysadmin doesn’t just solve problems; he also researches, tests and implements new things. True. But even those don’t tend to take up all available time (except at particular situations, such as the final part of a project), because no “sane” company replaces their systems every month (unfortunately, there are a lot of “insane” companies out there, but that’s another story).
5. Unlike an average sysadmin, a good one has very efficient monitoring and warning systems. An average sysadmin tends to be “warned” of problems by the users – “this service is down”, “I can’t access my email account”, and so on. A good sysadmin, however, was warned about that abnormally growing log file a long time before it was even close to taking up all free space in its partition. And he fixed the cause of that growth long before it affected any services.
6. Managers tend to love a worker who works 7 days a week, stays in after hours, and so on. But what does that mean? Either that that worker is slow and inefficient, or that he’s doing the work of 3 persons or more. If the former, they should reconsider his value; if the latter, he’s the one who should reconsider his employment…
7. A good sysadmin is a master of efficiency and time saving. As I said, he scripts, he automates, he solves problems “once and for all”. He investigates and deploys his own tools for quick remote administration, which are much faster than going to the physical location of a server, for instance.

See the pattern?

Unfortunately, there’s a problem here: visibility. A good sysadmin is like an efficient police force. If citizens constantly see the police running after criminals, in spectacular car chases, they’ll probably be impressed, but a really good police force would mean that most crimes would be prevented in some way. Instead of a police which goes after criminals, there would be an apparent lack of crime. Sadly, that tends to make people start to believe that maybe we don’t need such a well-financed, well-staffed police force…

Here, it’s the same thing. Think of the crash from a rapidly growing log file I mentioned above. With an average sysadmin, it’s “I can’t access my mail!”, then he goes, deletes the file, restarts the service, and it’s working again. With a good sysadmin, nothing would have happened. Nobody would have noticed anything. So, they start to think that maybe that sysadmin doesn’t do a lot around here… maybe he isn’t pulling his weight.

That’s ridiculous, though – that one may have to do his job worse than he’s capable of, just so that people notice his work. It’s something I refuse to do. But maybe some “self-marketing” would be useful – like, when you solve a problem before it happens, you tell your superiors about it. It’s a bit absurd… but I think it’s better, and more honest, than the alternative (doing a bad job on purpose).

(NOTE: reading part 1 first may be a good idea. )

To explore the “acting busy” vs. “doing actual work” theme, I want to share (without the sordid details, of course) a situation I’ve been in.

A few years ago, I worked as a sysadmin in a company which had about 20-30 Linux servers, and about the same number of Windows (NT 4 and 2000, at the time) servers. There were two separate teams of sysadmins, one for each type of servers, though both teams had the same boss, and worked in the same room.

The two teams, however, had a very different philosophy of work…

First, except for the initial server installations, almost all the work of the Linux team was made remotely, using ssh. Except for coffee/lunch/toilet breaks, we were at our desks for most of the time. On the other hand, the Windows team did their work inside the datacenter, physically on the servers (at the time, remote desktop utilities were less used than they are today).

When the Linux team encountered a problem in a server, we would figure out what the problem was, what caused it, and how to prevent it from happening again in the future – most problems, indeed, only happened once. The Windows guys, however, were of the philosophy that the only thing that mattered was to quickly make the services available again – by rebooting. If it happened every week – or, indeed, every day, tough. They had, they believed, to be getting paid for something, right?

The result of this was that we Linux guys were, most of the time, calmly sitting on our desks, and had a lot of free time (which we sometimes used for research, for staying informed about our field of work, for trying out new solutions on test servers, and so on – but which we also used, sometimes, simply for resting, or for browsing non-work sites). The Windows guys, on the other hand, were often inside the datacenter, most or all of them at the same time, putting out the latest “fire” – which would likely repeat itself in a couple of days, since they never attacked the cause, only rebooted the servers. And when it happen, they made sure they looked very alarmed – and that everyone saw them (there were other non-sysadmin teams in the same room) running to the datacenter to solve the latest crash.

The Windows servers had a lot of downtime, but the Linux ones’ was almost nil.

Now… guess which team the bosses preferred. Which team was seen as “competent” and “hard working”. Which team got promotions, pay raises, and shiny new laptops.

Welcome to the first article in the freshly created “Work” category.

The first question: have you ever been in a situation where your boss attempts to direct your career – probably in a different direction than that you yourself have chosen?

Basically, suppose that there are several work-related things you like to do, that you really enjoy doing, and, so, you look for a job where you’ll be asked (and paid) to do exactly that. So far, so good. But… some time afterwards (maybe months, or even a couple of years), you increasingly feel that your boss is attempting to steer you away from the things you enjoy, and towards something you either hate, or simply doesn’t have anything to do with you.

You’re a reasonable person, so you’re aware that you won’t ever have a “perfect job” in which everything you do is something that challenges, stimulates or fascinates you – that there are some “menial”, or boring, or simply unpleasant jobs to be done, from time to time. That’s normal, and you were prepared from that from the beginning. But now, you’re finding out that those tasks are increasing in number and length, and that what you really love doing, what you were hired to do in the first place, is getting rarer and rarer.

If it was because the company’s requirements have changed – maybe they don’t really need anyone to do what you like doing anymore – the solution is obvious: move. (if you have kids to feed or something, move after you get hired to something better, of course.) But what if it’s not the company, but just your boss?

Talking to him is the obvious thing to do, right? But what if he refuses to understand? What if he treats you as if you’re a “spoiled” or arrogant worker who believes he must always do things he enjoys, and nothing else? What if he truly believes that he’s “helping” you, by making you learn new skills (even though they’re things you’ve already tried — and hated) – possibly even more “marketable” skills, like project management, or drawing Visio diagrams, or administration of some some software you dislike?

The solution again seems to involve “quitting”. But, have you been in a similar situation? Do you believe that it’s possible to make your boss appreciate your skills, your passion for your real work, and stop trying to steer you into a career you’d hate?

OpenSwan, ipsec.conf man page:

CONN PARAMETERS: MANUAL KEYING
The following parameters are relevant only to manual keying, and are ignored in automatic keying.

and, still in that section:

esp ESP encryption/authentication algorithm to be used for the connection, e.g. 3des-md5-96 (must be suitable as a value of ipsec_spi(8)'s --esp option); default is not to use ESP

Note that that option (“esp”) doesn’t appear in the AUTOMATIC KEYING options list. From the above, one would guess that it’s only for manual keying, and that for automatic keying that option is ignored – that, indeed, it’s not necessary.

Right? Unfortunately, that’s not true, from what I’ve seen.

Until I added:

esp=3des-sha1-96

to a particular automatic keyed connection, it simply wouldn’t work, because the default is to use md5 instead of sha1, and the other side used sha1.

Oh well… things like this end up making us sysadmins not trust documentation. Unless it’s OpenBSD, of course.