WireGuard is an exciting, new, extremely simple VPN system that uses state-of-the-art cryptography. Its Linux implementation runs in the kernel, which provides a significant performance boost compared to traditional userspace VPN implementations

The WireGuard kernel module is great, but sometimes you might not be able to install new kernel modules. One example scenario is on a VPS that uses OpenVZ or LXC. For these cases, we can use wireguard-go, a userspace implementation of WireGuard. This is the same implementation used on MacOS and Windows.

This post focuses on Debian, however the instructions should mostly work on other Linux distros too.

Install WireGuard Tools

We need to install the WireGuard tools (wg-quick). On Debian, you can run this as root:


echo "deb http://deb.debian.org/debian/ unstable main" > /etc/apt/sources.list.d/unstable.list
printf 'Package: *\nPin: release a=unstable\nPin-Priority: 90\n' > /etc/apt/preferences.d/limit-unstable
apt update
apt install wireguard-tools --no-install-recommends

(see the WireGuard site for instructions if you're not on Debian)

Install Go

Unfortunately, since wireguard-go is not packaged for Debian, we need to compile it ourselves. To compile it, we first need to install the latest version of the Go programming language (currently version 1.12.6):


cd /tmp
wget https://dl.google.com/go/go1.12.6.linux-amd64.tar.gz
tar zvxf go1.12.6.linux-amd64.tar.gz
sudo mv go /opt/go1.12.6
sudo ln -s /opt/go1.12.6/bin/go /usr/local/bin/go

Now, running go version should show the version number.

Compile wireguard-go

Now that we've got Go, we can download and compile wireguard-go. Download the code:


cd /usr/local/src
git clone https://git.zx2c4.com/wireguard-go

If you are on a system with limited RAM (such as a 64 MB or 128 MB "LowEndSpirit" VPS), you will need to do a small tweak to the wireguard-go code to make it use less RAM. Open device/queueconstants_default.go and replace this:


	MaxSegmentSize             = (1 << 16) - 1 // largest possible UDP datagram
	PreallocatedBuffersPerPool = 0 // Disable and allow for infinite memory growth

With these values (taken from device/queueconstants_ios.go):


	MaxSegmentSize             = 1700
	PreallocatedBuffersPerPool = 1024

This will make it use a fixed amount of RAM (~20 MB max), rather than allowing memory usage to grow infinitely.

Now we can compile it:


cd wireguard-go
make
# "Install" it
sudo cp wireguard-go /usr/local/bin

Running wireguard-go --version should work and show the version number.

Configuration

You'll need to configure /etc/wireguard/wg0.conf to contain the configuration for your peer. This post won't go into significant detail about this; please refer to another general WireGuard guide (like this one) for more details. Your wg0.conf should end up looking something like:


[Interface]
Address = 10.123.0.2
PrivateKey = 12345678912345678912345678912345678912345678
ListenPort = 51820

[Peer]
PublicKey = 987654321987654321987654321987654321987654321
AllowedIPs = 10.123.0.1/32
Endpoint = 198.51.100.1:51820

We need to modify the systemd unit to pass the WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD flag to wireguard-go, to allow it to run on Linux. Open /lib/systemd/system/wg-quick@.service, find:


Environment=WG_ENDPOINT_RESOLUTION_RETRIES=infinity

and add this line directly below:


Environment=WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD=1

Finally, enable and start the systemd service:


systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

Enabling the systemd service will connect the VPN on boot, and starting the systemd service will connect it right now.

You're Done

Now, everything should be working! You can check the status of wg-quick by running systemctl status wg-quick@wg0, which should return something like:


● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2019-07-01 06:30:30 UTC; 1 day 22h ago

Running wg will give you a list of all the peers, and some details about them:


interface: wg0
  public key: 987654321987654321987654321987654321987654321
  private key: (hidden)
  listening port: 38917

peer: 987654321987654321987654321987654321987654321
  endpoint: 198.51.100.1:51820
  allowed ips: 10.123.0.1/32
  latest handshake: 1 day, 22 hours, 59 minutes, 34 seconds ago
  transfer: 2.75 KiB received, 2.83 KiB sent

So I recently encountering a strange issue on two of my servers. I noticed that the load average was increasing approximately every 20 minutes:

Load average graph

I suspected a cronjob, but I don't have any cronjobs that run every 20 mins. Also, CPU usage doesn't actually increase during that period:

Low CPU usage graph

I did some digging and it took a long time to work out what was happening.

So What Is Load Average Anyway?

"Load average" is a term used to describe a measure of how "busy" a system is. Unix-like systems (including Linux) show a load average as three numbers, representing the system load over the previous one minute, five minutes, and fifteen minutes. These numbers represent the number of processes that are using the CPU right now, waiting to use the CPU, or waiting for disk I/O. The Wikipedia article has more details.

Linux updates the load average every 5 seconds. In fact, it actually updates every 5 seconds plus one "tick". The reason for this is to avoid coinciding with other five-second timers:

It turns out that there are a few other five-second timers in the kernel, and if the timers get in sync, the load-average can get artificially inflated by events that just happen to coincide. So just offset the load average calculation it by a timer tick.

From the Linux kernel code:

sched/loadavg.h:


#define LOAD_FREQ	(5*HZ+1) /* 5 sec intervals */

sched/loadavg.c


 * The global load average is an exponentially decaying average of nr_running +
 * nr_uninterruptible.
 *
 * Once every LOAD_FREQ:
 *
 *   nr_active = 0;
 *   for_each_possible_cpu(cpu)
 *	nr_active += cpu_of(cpu)->nr_running + cpu_of(cpu)->nr_uninterruptible;
 *
 *   avenrun[n] = avenrun[0] * exp_n + nr_active * (1 - exp_n)
HZ is the kernel timer frequency, which is defined when compiling the kernel. On my system, it's 250:

% grep "CONFIG_HZ=" /boot/config-$(uname -r)
CONFIG_HZ=250

This means that every 5.004 seconds (5 + 1/250), Linux calculates the load average. It checks how many processes are actively running plus how many processes are in uninterruptable wait (eg. waiting for disk IO) states, and uses that to compute the load average, smoothing it exponentially over time.

Say you have a process that starts a bunch of subprocesses every second. For example, Netdata collecting data from some apps. Normally, the process will be very fast and won't overlap with the load average check, so everything is fine. However, every 1251 seconds (5.004 * 250), the load average update interval will be an exact multiple of one second (that is, 1251 is the least common multiple of 5.004 and 1). 1251 seconds is 20.85 minutes, which is exactly the interval I was seeing the load average increase. My educated guess here is that every 20.85 minutes, Linux is checking the load average at the exact time that several processes are being started and are in the queue to run.

I confirmed this by disabling netdata and manually watching the load average:


while true; do uptime; sleep 5; done

After 1.5 hours, I did not see any similar spikes. The spikes only occur when Netdata is running.

It turns out other people have hit similar issues in the past, albeit with different intervals. The following posts were extremely helpful:

In the end, I'm not sure if I'd call this a bug, but perhaps netdata could implement some jitter so that it doesn't perform checks every one second exactly. I posted a GitHub issue so their developers can take a look.

Recently I moved all my sites onto a new server. I use Duplicity and Backupninja to perform weekly backups of my server. While configuring backups on the new server, I kept encountering a strange error:

Error: gpg: using "D5673F3E" as default secret key for signing
Error: gpg: signing failed: Inappropriate ioctl for device
Error: gpg: [stdin]: sign+encrypt failed: Inappropriate ioctl for device

It turns out this error is due to changes in GnuPG 2.1, which only recently landed in Debian Testing. The error occurs because GnuPG 2.1 by default ignores passphrases passed in via environment variables or stdin, and is trying to show a pinentry prompt. "Inappropriate ioctl for device" is thrown because the Backupninja script is not running through a TTY, so there's no way to actually render the prompt.

To solve the problem, you need to enable loopback pinentry mode. Add this to ~/.gnupg/gpg.conf:

use-agent
pinentry-mode loopback

And add this to ~/.gnupg/gpg-agent.conf, creating the file if it doesn't already exist:

allow-loopback-pinentry

Then restart the agent with echo RELOADAGENT | gpg-connect-agent and you should be good to go!

Yesterday, I had a hard drive die on me  :(. It was a 160 GB Seagate Barracuda, and was less than a year old. It started with strange error messages late last night. I went to the server, reset it (need to power it off - It wouldn't shutdown cleanly), and now it won't boot. When I try to boot the system, I can hear the drive head hitting against the side of the drive, which makes me think there is a problem internally with the drive...

I'm looking into recovery options, though I think it will be way too expensive for my needs. If it is too expensive, I'll just have to cut my losses and move on. A lot of my development work was on this server (it was my backup PC as well). Whilst I have local copies of stuff, a lot of it is slightly old. It's kinda depressing to have something like this happen, so I'm not really in a good mood at the moment  :(

Update: About two weeks after the hard drive "failed", I tried it again. It was working, but made a strange sound. I quickly copied all the data off of it, and disconnected it. I was able to get a copy of all the most important data. So, it's not so bad... This story has a happy ending :D

I checked my email inbox this morning, and guess what I found? The firewall (ConfigServer Security and Firewall) on a server I help run blocked a brute-force attack from Nokia:

Time: Tue May 1 02:28:18 2007
IP: 63.97.248.34 (machine34.nokia.com)
Failures: 5 (sshd)
Interval: 135 seconds
Blocked: Yes

Log entries:

May 1 02:28:08 blue sshd[9363]: Failed password for root from ::ffff:63.97.248.34 port 56057 ssh2
May 1 07:28:08 blue sshd[9364]: Failed password for root from ::ffff:63.97.248.34 port 56057 ssh2
May 1 02:28:11 blue sshd[9368]: Failed password for root from ::ffff:63.97.248.34 port 56436 ssh2
May 1 07:28:11 blue sshd[9369]: Failed password for root from ::ffff:63.97.248.34 port 56436 ssh2
May 1 02:28:13 blue sshd[9370]: Failed password for root from ::ffff:63.97.248.34 port 56591 ssh2

Just thought it was funny :P
(oh yeah, and I will report it to them!)

This tutorial will show you how to set up a serial console on a Linux system, and connect to it via a null modem cable. This is quite useful if your Linux server is in a headless configuration (no keyboard or monitor), as it allows you to easily get a console on the system if there are any problems with it (especially network problems, when SSH is not available). In the end, the GRUB menu will appear over the serial link, as will the bootup messages (output when booting the system). I'm using Debian Etch on the server and Ubuntu Edgy on my client, although this should work on any Linux distribution.

Read more ⇒

The other day, I was looking for an easy way to restore a MySQL dump (or backup, whatever you like to call it) in PHP. I've previously used a segment of the code from PHP MySQL Backup V 2.2 for this, but it didn't seem to support FULLTEXT indicies that well. So, I searched around, but couldn't find anything. I even asked on the PHP IRC channel, and they suggested to use shell_exec to call mysql (unfortunately, I've disabled shell_exec for security reasons). Looking closer, I noticed that this was actually quite easy to do.

Read more ⇒

In this tutorial, I'll show you how to install Linux-Vserver on Debian Testing (Etch), the easy way. This was the first tutorial I posted to HowtoForge.com, so please tell me if you like it or not. You may find it a bit verbose, as I try to explain things in enough detail so that everyone understands what I mean :-)

What is Linux-Vserver, you ask? It's simple. Basically, Linux-Vserver is an open-source system used to separate a single physical server into multiple virtual servers. From the Linux-Vserver website:

"Linux-VServer allows you to create virtual private servers and security contexts which operate like a normal Linux server, but allow many independent servers to be run simultaneously in one box at full speed. All services, such as ssh, mail, Web, and databases, can be started on such a VPS, without modification, just like on any real server. Each virtual server has its own user account database and root password and doesn't interfere with other virtual servers."

Read more ⇒

Update: Since I made this post in September 2006, things have changed. Compiz-Quinn has changed its name to Beryl, and up-to-date installation instructions are available for Ubuntu Edgy and Ubuntu Feisty

Well, I got my laptop on Friday (15th September). I must say, the Inspiron 6400 was definately a good choice. The only thing I don't really like about it is the reflectiveness of the TrueLife screen, but I suppose I'll get used to it (the colours are definately more vivid, and it has better contrast... The screen looks absolutely awesome when playing games!). Anyways, one of the first things I did was partition the drive, and install Ubuntu Linux :). Once I did this, the first thing I did was install i8kutils and Gkrellm (so that I could control the speed of the fan, to make sure it doesn't overheat), and then install the ATI Drivers for the X1400 (called 'FGLRX'). Everything was working excellently (see screenshot)
Desktop screenshot

After everything was done, I thought about what to install. I remember my friend telling me about something called 'Compiz', but I couldn't remember how to install it. After searching for a while, I found a nice tutorial on installing GLX/Compiz. It appeared to be quite easy to do, so I followed the instructions on the tutorial. And, it works perfectly! :D It comes with heaps of themes, and there are some similar to the upcoming Windows Vista. Take a look at this:
Compiz screenshot 1
Look at the title bar of the 'Home Directory' window. That's called a 'glass effect', and is quite nice...

Another thing implemented in Compiz is window transparency: You can make a window semi-transparent. For example, look at this screenshot of my home directory on top of Opera (with the Google site open). I've faded the window out a bit, so it's partially transparent:
Compiz screenshot 2 - transparency
Nice, huh? Yet another thing in Compiz is multiple desktops. Sure, Linux has support for multiple workplaces in-built (the selector on the bottom-right hand side of Gnome). However, Compiz improves on it vastly, making all the virtual desktops into one large cube:
Compiz screenshot 3 - Cube
I thought the cube was interesting, but something really interesting is if you run VMWare in fullscreen mode. You can make it appear as if you're running multiple operating systems at once... Take a look:
Compiz screenshot 4 - VMWare

In addition to all of this, Compiz has heaps of other effects which I couldn't take screenshots of (including wobbly windows - windows wobble as you drag them, windows 'zoom in' when you start a program, and 'zoom out' when you exit it, and a lot more)... If you'd like to get Compiz working, and have an ATI graphics card, first get the ATI Linux drivers (FGLRX) working (see http://wiki.cchtml.com/index.php/Ubuntu_Dapper_Installation_Guide for a tutorial. Make sure you follow the second bit of the tutorial, to install version 8.28.8). Then, follow the Compiz tutorial, as found at http://www.compiz.net/topic-389-1.html (I followed the second howto, as I personally think it's better to have Compiz as a seperate session... Have fun! :)

Anyways, I think this is probably the longest blog post I've written, ever :P

Well, I was working on my Linux testing server, and it stopped working for some reason... I tried to restart, but the computer wouldn't even boot (even from a CD). Upon unplugging all the IDE devices, and only leaving a CD drive in (a different one, in case it was faulty for whatever reason), it still wouldn't boot :(. I guess I won't be using that for a while...

Anyways, I'm buying a new server on eBay, so it isn't that bad (all I need to do is wait for it to get shipped to me).

Old server:

  • Celeron 300 MHz
  • 4.3GB Quantum Fireball lct and 4.3GB Seagate hard drives
  • 48X CD-ROM drive
  • 192MB membory

New server:

  • Pentium III 650 MHz
  • 8.4 GB hard drive
  • 48X CD-ROM drive
  • 256MB membory

Don't say that it isn't that powerful... Linux makes any computer look powerful :D. Besides, it's only a server for testing stuff.

EDIT: Computer arrived today, 30th June 2006. That company on eBay was quick!