It’s been quite a rocky year after switching jobs but one of the upshots is that I’ve managed to hopefully have finally sorted out this blog. You’ll perhaps notice all the old posts have finally come back since March. Hopefully some of it’s still of some use. Over the next few days and weeks I’ll be making some posts about what I use my new “homelab 😂” for: Personal knowledge management (PKM) How this blog is automatically updated from the PKM (fun) Media server Online book reading Webserver(s) Etc!

Earlier this week, I had an interesting problem someone asked me to solve. A custom analytics tool had been set up which was gathering lots of data at a rate of about 10 rows/second. The analytics table had 5.3 million rows, was innodb, had two fields which were nested JSON (stored as TEXT) and NO INDEXES. The customer wanted to be able to run reports on this table and was unable to as mariadb would just lock up (as expected) when trying to use any kind of clause.

john-hunt.com has been a wordpress site since the early 2000s, mostly a blog of interesting things I’ve discovered during my devops/software engineering career. I’ve recently switched over to using hugo to generate a statically hosted website rather than dealing with security updates that wordpress requires. Hopefully over the next couple of weeks I’ll migrate the old posts over as there were a couple of posts that generated quite a lot of interest (notably the external USB data recovery one.

So for a long time, I’ve been wondering why I can’t use a locally mounted directory within one of my containers - I’d just get an empty directory on the volume mount point inside the container. A lot of googling yielded nothing, but then when I ran ps -aux | grep -i docker I noticed docker was running via snap. It then occurred to me that snap has restrictions on what parts of the filesystem it allows it’s ‘snaps’ to read - this indeed was the problem so a quick uninstall of the snapped version of docker and installed via apt fixed the issue once and for all.

Recently I added a static analysis tool (Bandit - which is an excellent tool and deserves it’s own post) to our Python microservices CI pipeline at work. Upon doing this, I discovered that our requests calls didn’t have timeouts set on them. By default, requests don’t have a timeout set. At first, I thought it was going to be straight forward to just put a timeout on the remote calls, but upon reading the documentation I discovered that the timeout value was not quite what you’d expect - it’s not the wall time to complete the request.

After quite some time I’ve managed to get my server back online. The issue was that running it on a raspberry pi 3 alongside other things pretty much ensured the OOM killer would kick in and start killing things off. I got one of those HP elitedesk PCs for only £40 off eBay to host this and it’s superb! This is now running in a docker container on a low powered 8gb system.

This site is now hosted on my raspberry pi zero w - incredible! Hosting costs are now on a real shoestring budget. I’ll be tinkering with it to get the site to perform a bit better (microssd is not fast..) This was part fun experiment, part cost saving exercise.

This handy snippet is useful for seeing what merged PRs there are between two branches - eg master/release or whatever you’re using. I find it v.handy: git log --merges --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr)%Creset' --abbrev-commit --date=relative origin/production..origin/master

Stick this in your .bash_profile or .bashrc file: function diffopen() { git diff --name-only "$1" | xargs -d '\n' pstorm; } Then in your project folder, say you want to open all files that differ from your master branch: diffopen master Note, on OSX you’ll need the GNU version of xargs for this to work, as I discovered the hard way..

This post is mostly for my own reference. I’m still in the very early stages of understanding and using docker. # The docker cli client app brew install docker # Tool for installing a docker machine (VM and or docker layer for holding containers (the whale)) brew install docker-machine # Create a local docker machine using virtualbox as the VM, call it 'dev' # Assuming this is where it gets clever as we can create docker machines for reference locally way off in the cloud docker-machine create --driver virtualbox dev # Nothing worked properly until I did this, not sure what it actually does, probably sets the active docker environment eval "$(docker-machine env dev)" # Pull down and register the whalesay container docker pull docker/whalesay # Run the cowsay command on the whalesay container with the argument 'boo' docker run docker/whalesay cowsay boo