In this post, I'll explain how to securely configure NFS on Debian, to mount a directory from one server on another machine. A good use case for this is if you have a storage VPS with a large amount of storage, and want to use this space from other servers.

Security

NFS is unencrypted by default. It can be encrypted if you use Kerberos, but I wouldn't recommend going through the pain of configuring Kerberos unless you're setting up a corporate network with dozens of users.

Because of this, I would recommend never exposing an NFS server directly to the internet. I'd also advise against exposing it on "internal" networks which are not isolated per customer, such as what HostHatch provides. On isolated private networks (like what BuyVM provides), it's fine to use NFS unecrypted.

To secure NFS connections over the internet or other untrusted network, I'd recommend using WireGuard. There are various guides on how to configure WireGuard (like this one) so I won't go into it in too much detail. Note that WireGuard does not have the concept of a "client" and "server" like classic VPN solutions like OpenVPN. Each node is a "peer", and the overall topology is up to you. For example, you can have a "mesh" VPN network where every machine can directly access every other machine, without a central server.

On Debian 11 (Bullseye, testing) you can simply use apt install wireguard to get WireGuard. On Debian 10 (Buster), you'll have to enable buster-backports then do apt -t buster-backports install wireguard.

Generate a private and public key on each system:

wg genkey | tee privatekey | wg pubkey > publickey

Then configure /etc/wireguard/wg0.conf on each system. The [Interface] section should have the private key for that particular system. The NFS server should have a [Peer] section for each system that is allowed to access the NFS server, and all the other systems should have a [Peer] section for the NFS server. It should look something like this:

[Interface]
Address = 10.123.0.2
PrivateKey = 12345678912345678912345678912345678912345678
ListenPort = 51820

[Peer]
PublicKey = 987654321987654321987654321987654321987654321
AllowedIPs = 10.123.0.1/32
Endpoint = 198.51.100.1:51820

where 10.123.0.1 and 10.123.0.2 can be any IPs of your choosing, as long as they're in the same subnet and in one of the IP ranges reserved for local networks (10.x.x.x is usually a good choice).

Enable and start the WireGuard service on each machine:

systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

Run wg to check that it's running. Make sure you can ping the NFS server from the other servers.

NFS Server

On the NFS server, install the nfs-kernel-server package:

apt install nfs-kernel-server

A best practice these days is to only enable NFSv4 unless you really need NFSv3. To only enable NFSv4, set the following variables in /etc/default/nfs-common:

NEED_STATD="no"
NEED_IDMAPD="yes"

And the following in /etc/default/nfs-kernel-server. Note that RPCNFSDOPTS is not present by default, and needs to be added.

RPCNFSDOPTS="-N 2 -N 3 -H 10.123.0.1"
RPCMOUNTDOPTS="--manage-gids -N 2 -N 3"

10.123.0.1 should be the IP address the NFS server will listen on (the WireGuard IP).

Additionally, rpcbind is not needed by NFSv4 but will be started as a prerequisite by nfs-server.service. This can be prevented by masking rpcbind.service and rpcbind.socket:

systemctl mask rpcbind.service
systemctl mask rpcbind.socket

Next, configure your NFS exports in /etc/exports. For example, this will export the /data/hello-world directory and only allow 10.123.0.2 to access it:

/data/hello-world 10.123.0.2(rw,sync,no_subtree_check)

Refer to the exports(5) man page for more details.

Finally, start the NFS server:

systemctl start nfs-server

NFS Client

On the NFS client, you need to install the nfs-common package:

apt install nfs-common

Now, you can use the mount command to mount the directory over NFS:

mkdir -p /mnt/data/
mount -t nfs4 -o vers=4.2,async 10.123.0.1:/data/hello-world /mnt/data/

Try write some files to /mnt/data, and it should work!

To automatically mount the directory on boot, modify /etc/fstab:

10.123.0.1:/data/hello-world /mnt/data nfs4 auto,vers=4.2

Optional: Caching

You can optionally cache data from the NFS server on the local disk by using a transparent read-through cache called CacheFS. The first time files are read via NFS, they will be cached locally. On subsequent reads, if the file has not been modified since the time it was cached, it will be read from the local cache rather than loading over the network. This can provide a significant performance benefit if the NFS server has slower disks and/or is physically distant from the clients.

To enable caching, first install cachefilesd:

apt install cachefilesd

Turn it on by editing /etc/default/cachefilesd, following the instructions in the file:

# You must uncomment the run=yes line below for cachefilesd to start.
# Before doing so, please read /usr/share/doc/cachefilesd/howto.txt.gz as
# extended user attributes need to be enabled on the cache filesystem.
RUN=yes

Modify your NFS mount in /etc/fstab to add the fsc (file system cache) attribute. For example:

10.123.0.1:/data/hello-world /mnt/data nfs4 auto,vers=4.2,fsc

Finally, start the service and remount your directory:

systemctl start cachefilesd
mount -o remount /mnt/data

To check that it's working, read some files from the mount, and you should see /var/cache/fscache/ growing in size:

du -sh /var/cache/fscache/
76K     /var/cache/fscache/

By default, the cache will keep filling up until the disk only has 7% space left. Once the disk drops below 7% free space. If the disk space drops below 3%, caching will be turned off entirely. You can change these thresholds by modifying /etc/cachefilesd.conf.

WireGuard is an exciting, new, extremely simple VPN system that uses state-of-the-art cryptography. Its Linux implementation runs in the kernel, which provides a significant performance boost compared to traditional userspace VPN implementations

The WireGuard kernel module is great, but sometimes you might not be able to install new kernel modules. One example scenario is on a VPS that uses OpenVZ or LXC. For these cases, we can use wireguard-go, a userspace implementation of WireGuard. This is the same implementation used on MacOS, Windows, and the WireGuard mobile apps. This implementation is slower than the kernel module, but still plenty fast.

This post focuses on Debian, however the instructions should mostly work on other Linux distros too.

Install WireGuard Tools

We need to install the WireGuard tools (wg-quick). On Debian, you can run this as root:

echo "deb http://deb.debian.org/debian/ unstable main" > /etc/apt/sources.list.d/unstable.list
printf 'Package: *\nPin: release a=unstable\nPin-Priority: 90\n' > /etc/apt/preferences.d/limit-unstable
apt update
apt install wireguard-tools --no-install-recommends

(see the WireGuard site for instructions if you're not on Debian)

Install Go

Unfortunately, since wireguard-go is not packaged for Debian, we need to compile it ourselves. To compile it, we first need to install the latest version of the Go programming language (currently version 1.13.4):

cd /tmp
wget https://dl.google.com/go/go1.13.4.linux-amd64.tar.gz
tar zvxf go1.13.4.linux-amd64.tar.gz
sudo mv go /opt/go1.13.4
sudo ln -s /opt/go1.13.4/bin/go /usr/local/bin/go

Now, running go version should show the version number.

Compile wireguard-go

Now that we've got Go, we can download and compile wireguard-go. Download the latest release version:

cd /usr/local/src
wget https://git.zx2c4.com/wireguard-go/snapshot/wireguard-go-0.0.20191012.tar.xz
tar xvf wireguard-go-0.0.20191012.tar.xz
cd wireguard-go-0.0.20191012

If you are on a system with limited RAM (such as a 256 MB or lower "LowEndSpirit" VPS), you will need to do a small tweak to the wireguard-go code to make it use less RAM. Open device/queueconstants_default.go and replace this:

MaxSegmentSize             = (1 << 16) - 1 // largest possible UDP datagram
PreallocatedBuffersPerPool = 0 // Disable and allow for infinite memory growth

With these values (taken from device/queueconstants_ios.go):

MaxSegmentSize             = 1700
PreallocatedBuffersPerPool = 1024

This will make it use a fixed amount of RAM (~20 MB max), rather than allowing memory usage to grow infinitely.

Now we can compile it:

make
# "Install" it
sudo cp wireguard-go /usr/local/bin

Running wireguard-go --version should work and show the version number.

If you have multiple VPSes that use the same OS version and architecture (eg. Debian 10, 64-bit), you can compile it on one of them and then just copy the wireguard-go binary to all the others.

Configuration

wg0.conf

You'll need to configure /etc/wireguard/wg0.conf to contain the configuration for your peer. This post won't go into significant detail about this; please refer to another general WireGuard guide (like this one) for more details. The basic jist is that you need to run:

wg genkey | tee privatekey | wg pubkey > publickey

to generate a public/private key pair for each peer, then configure the [Interface] with the private key for the peer, and a [Peer] section for each peer that can connect to it.

Your wg0.conf should end up looking something like:

[Interface]
Address = 10.123.0.2
PrivateKey = 12345678912345678912345678912345678912345678
ListenPort = 51820

[Peer]
PublicKey = 987654321987654321987654321987654321987654321
AllowedIPs = 10.123.0.1/32
Endpoint = 198.51.100.1:51820

systemd

We need to modify the systemd unit to pass the WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD flag to wireguard-go, to allow it to run on Linux. Open /lib/systemd/system/wg-quick@.service, find:

Environment=WG_ENDPOINT_RESOLUTION_RETRIES=infinity

and add this line directly below:

Environment=WG_I_PREFER_BUGGY_USERSPACE_TO_POLISHED_KMOD=1

Finally, enable and start the systemd service:

systemctl enable wg-quick@wg0
systemctl start wg-quick@wg0

Enabling the systemd service will connect the VPN on boot, and starting the systemd service will connect it right now.

You're Done

Now, everything should be working! You can check the status of wg-quick by running systemctl status wg-quick@wg0, which should return something like:

● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0
   Loaded: loaded (/lib/systemd/system/wg-quick@.service; enabled; vendor preset: enabled)
   Active: active (exited) since Mon 2019-07-01 06:30:30 UTC; 1 day 22h ago

Running wg will give you a list of all the peers, and some details about them:

interface: wg0
  public key: 987654321987654321987654321987654321987654321
  private key: (hidden)
  listening port: 38917

peer: 987654321987654321987654321987654321987654321
  endpoint: 198.51.100.1:51820
  allowed ips: 10.123.0.1/32
  latest handshake: 1 day, 22 hours, 59 minutes, 34 seconds ago
  transfer: 2.75 KiB received, 2.83 KiB sent

So I recently encountering a strange issue on two of my servers. I noticed that the load average was increasing approximately every 20 minutes:

Load average graph

I suspected a cronjob, but I don't have any cronjobs that run every 20 mins. Also, CPU usage doesn't actually increase during that period:

Low CPU usage graph

I did some digging and it took a long time to work out what was happening.

So What Is Load Average Anyway?

"Load average" is a term used to describe a measure of how "busy" a system is. Unix-like systems (including Linux) show a load average as three numbers, representing the system load over the previous one minute, five minutes, and fifteen minutes. These numbers represent the number of processes that are using the CPU right now, waiting to use the CPU, or waiting for disk I/O. The Wikipedia article has more details.

Linux updates the load average every 5 seconds. In fact, it actually updates every 5 seconds plus one "tick". The reason for this is to avoid coinciding with other five-second timers:

It turns out that there are a few other five-second timers in the kernel, and if the timers get in sync, the load-average can get artificially inflated by events that just happen to coincide. So just offset the load average calculation it by a timer tick.

From the Linux kernel code:

sched/loadavg.h:

#define LOAD_FREQ	(5*HZ+1) /* 5 sec intervals */

sched/loadavg.c

* The global load average is an exponentially decaying average of nr_running +
 * nr_uninterruptible.
 *
 * Once every LOAD_FREQ:
 *
 *   nr_active = 0;
 *   for_each_possible_cpu(cpu)
 *	nr_active += cpu_of(cpu)->nr_running + cpu_of(cpu)->nr_uninterruptible;
 *
 *   avenrun[n] = avenrun[0] * exp_n + nr_active * (1 - exp_n)

HZ is the kernel timer frequency, which is defined when compiling the kernel. On my system, it's 250:

% grep "CONFIG_HZ=" /boot/config-$(uname -r)
CONFIG_HZ=250

This means that every 5.004 seconds (5 + 1/250), Linux calculates the load average. It checks how many processes are actively running plus how many processes are in uninterruptable wait (eg. waiting for disk IO) states, and uses that to compute the load average, smoothing it exponentially over time.

Say you have a process that starts a bunch of subprocesses every second. For example, Netdata collecting data from some apps. Normally, the process will be very fast and won't overlap with the load average check, so everything is fine. However, every 1251 seconds (5.004 * 250), the load average update interval will be an exact multiple of one second (that is, 1251 is the least common multiple of 5.004 and 1). 1251 seconds is 20.85 minutes, which is exactly the interval I was seeing the load average increase. My educated guess here is that every 20.85 minutes, Linux is checking the load average at the exact time that several processes are being started and are in the queue to run.

I confirmed this by disabling netdata and manually watching the load average:

while true; do uptime; sleep 5; done

After 1.5 hours, I did not see any similar spikes. The spikes only occur when Netdata is running.

It turns out other people have hit similar issues in the past, albeit with different intervals. The following posts were extremely helpful:

In the end, I'm not sure if I'd call this a bug, but perhaps netdata could implement some jitter so that it doesn't perform checks every one second exactly. I posted a GitHub issue so their developers can take a look.

Recently I moved all my sites onto a new server. I use Duplicity and Backupninja to perform weekly backups of my server. While configuring backups on the new server, I kept encountering a strange error:

Error: gpg: using "D5673F3E" as default secret key for signing
Error: gpg: signing failed: Inappropriate ioctl for device
Error: gpg: [stdin]: sign+encrypt failed: Inappropriate ioctl for device

It turns out this error is due to changes in GnuPG 2.1, which only recently landed in Debian Testing. The error occurs because GnuPG 2.1 by default ignores passphrases passed in via environment variables or stdin, and is trying to show a pinentry prompt. "Inappropriate ioctl for device" is thrown because the Backupninja script is not running through a TTY, so there's no way to actually render the prompt.

To solve the problem, you need to enable loopback pinentry mode. Add this to ~/.gnupg/gpg.conf:

use-agent
pinentry-mode loopback

And add this to ~/.gnupg/gpg-agent.conf, creating the file if it doesn't already exist:

allow-loopback-pinentry

Then restart the agent with echo RELOADAGENT | gpg-connect-agent and you should be good to go!

Yesterday, I had a hard drive die on me  😦. It was a 160 GB Seagate Barracuda, and was less than a year old. It started with strange error messages late last night. I went to the server, reset it (need to power it off - It wouldn't shutdown cleanly), and now it won't boot. When I try to boot the system, I can hear the drive head hitting against the side of the drive, which makes me think there is a problem internally with the drive...

I'm looking into recovery options, though I think it will be way too expensive for my needs. If it is too expensive, I'll just have to cut my losses and move on. A lot of my development work was on this server (it was my backup PC as well). Whilst I have local copies of stuff, a lot of it is slightly old. It's kinda depressing to have something like this happen, so I'm not really in a good mood at the moment  😦

Update: About two weeks after the hard drive "failed", I tried it again. It was working, but made a strange sound. I quickly copied all the data off of it, and disconnected it. I was able to get a copy of all the most important data. So, it's not so bad... This story has a happy ending 😄

I checked my email inbox this morning, and guess what I found? The firewall (ConfigServer Security and Firewall) on a server I help run blocked a brute-force attack from Nokia:

Time: Tue May 1 02:28:18 2007

IP: 63.97.248.34 (machine34.nokia.com)

Failures: 5 (sshd)

Interval: 135 seconds

Blocked: Yes

Log entries:

May 1 02:28:08 blue sshd[9363]: Failed password for root from ::ffff:63.97.248.34 port 56057 ssh2

May 1 07:28:08 blue sshd[9364]: Failed password for root from ::ffff:63.97.248.34 port 56057 ssh2

May 1 02:28:11 blue sshd[9368]: Failed password for root from ::ffff:63.97.248.34 port 56436 ssh2

May 1 07:28:11 blue sshd[9369]: Failed password for root from ::ffff:63.97.248.34 port 56436 ssh2

May 1 02:28:13 blue sshd[9370]: Failed password for root from ::ffff:63.97.248.34 port 56591 ssh2

Just thought it was funny 😛

(oh yeah, and I will report it to them!)

This tutorial will show you how to set up a serial console on a Linux system, and connect to it via a null modem cable. This is quite useful if your Linux server is in a headless configuration (no keyboard or monitor), as it allows you to easily get a console on the system if there are any problems with it (especially network problems, when SSH is not available). In the end, the GRUB menu will appear over the serial link, as will the bootup messages (output when booting the system). I'm using Debian Etch on the server and Ubuntu Edgy on my client, although this should work on any Linux distribution.

Read more ⇒

The other day, I was looking for an easy way to restore a MySQL dump (or backup, whatever you like to call it) in PHP. I've previously used a segment of the code from PHP MySQL Backup V 2.2 for this, but it didn't seem to support FULLTEXT indicies that well. So, I searched around, but couldn't find anything. I even asked on the PHP IRC channel, and they suggested to use shell_exec to call mysql (unfortunately, I've disabled shell_exec for security reasons). Looking closer, I noticed that this was actually quite easy to do.

Read more ⇒

In this tutorial, I'll show you how to install Linux-Vserver on Debian Testing (Etch), the easy way. This was the first tutorial I posted to HowtoForge.com, so please tell me if you like it or not. You may find it a bit verbose, as I try to explain things in enough detail so that everyone understands what I mean 😃

What is Linux-Vserver, you ask? It's simple. Basically, Linux-Vserver is an open-source system used to separate a single physical server into multiple virtual servers. From the Linux-Vserver website:

"Linux-VServer allows you to create virtual private servers and security contexts which operate like a normal Linux server, but allow many independent servers to be run simultaneously in one box at full speed. All services, such as ssh, mail, Web, and databases, can be started on such a VPS, without modification, just like on any real server. Each virtual server has its own user account database and root password and doesn't interfere with other virtual servers."

Read more ⇒

Update: Since I made this post in September 2006, things have changed. Compiz-Quinn has changed its name to Beryl, and up-to-date installation instructions are available for Ubuntu Edgy and Ubuntu Feisty

Well, I got my laptop on Friday (15th September). I must say, the Inspiron 6400 was definately a good choice. The only thing I don't really like about it is the reflectiveness of the TrueLife screen, but I suppose I'll get used to it (the colours are definately more vivid, and it has better contrast... The screen looks absolutely awesome when playing games!). Anyways, one of the first things I did was partition the drive, and install Ubuntu Linux 😃. Once I did this, the first thing I did was install i8kutils and Gkrellm (so that I could control the speed of the fan, to make sure it doesn't overheat), and then install the ATI Drivers for the X1400 (called 'FGLRX'). Everything was working excellently (see screenshot)

Desktop screenshot

After everything was done, I thought about what to install. I remember my friend telling me about something called 'Compiz', but I couldn't remember how to install it. After searching for a while, I found a nice tutorial on installing GLX/Compiz. It appeared to be quite easy to do, so I followed the instructions on the tutorial. And, it works perfectly! 😄 It comes with heaps of themes, and there are some similar to the upcoming Windows Vista. Take a look at this:

Compiz screenshot 1

Look at the title bar of the 'Home Directory' window. That's called a 'glass effect', and is quite nice...

Another thing implemented in Compiz is window transparency: You can make a window semi-transparent. For example, look at this screenshot of my home directory on top of Opera (with the Google site open). I've faded the window out a bit, so it's partially transparent:

Compiz screenshot 2 - transparency

Nice, huh? Yet another thing in Compiz is multiple desktops. Sure, Linux has support for multiple workplaces in-built (the selector on the bottom-right hand side of Gnome). However, Compiz improves on it vastly, making all the virtual desktops into one large cube:

Compiz screenshot 3 - Cube

I thought the cube was interesting, but something really interesting is if you run VMWare in fullscreen mode. You can make it appear as if you're running multiple operating systems at once... Take a look:

Compiz screenshot 4 - VMWare

In addition to all of this, Compiz has heaps of other effects which I couldn't take screenshots of (including wobbly windows - windows wobble as you drag them, windows 'zoom in' when you start a program, and 'zoom out' when you exit it, and a lot more)... If you'd like to get Compiz working, and have an ATI graphics card, first get the ATI Linux drivers (FGLRX) working (see http://wiki.cchtml.com/index.php/Ubuntu_Dapper_Installation_Guide for a tutorial. Make sure you follow the second bit of the tutorial, to install version 8.28.8). Then, follow the Compiz tutorial, as found at http://www.compiz.net/topic-389-1.html (I followed the second howto, as I personally think it's better to have Compiz as a seperate session... Have fun! 😃

Anyways, I think this is probably the longest blog post I've written, ever 😛

Previous 1 2