Over the last couple of months I have relied on a couple of web articles to get me through particularly specific issues. I hope these sites will remain available forever… However, I have “printed” the sites to PDFs to make sure they are available should the sites disappear / migrate / melt… etc
I’ve been using SES for a while for sending email from servers.
A few months ago one of my servers was compromised and I discovered there is no such thing as good logging in SES…. whooops… Amazon suspended my account and this lead me to investigate how I could get some visibility of SES logging. This was considerably harder than I would have thought. There are no SMTP logs, not real recording of any information and no diagnostics. I nearly left SES for this, then I decided to use it as a learning opportunity to understand more about AWS & SES.
I used this guide to get it all working: https://blog.andreev.it/?p=5513
Hopefully this site never goes down, if so I will need to write my own guide!
It’s easy to add extra fields to the database. These are all the objects available https://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html
Finally this docker image allows you to connect to dynamodb easily https://github.com/YoyaTeam/dynamodb-manager
The only thing I haven’t managed to do is find out which IAM user actually sent the email… I need to do more research on this to try and enhance my logging further.
I have had a couple of requirements recently where I have needed to leave a device onsite for remote access. I initially thought that the best way to handle this would be to put a pi onsite that joined my VPN and then I could connect to all the machines on the remote network and do my troubleshooting like that…
I looked at my pi shelf and there looking all little and shiny was a Pi Zero… At this point I thought to myself… what do I actually need from this device.
I have a couple of Emergency Linux VMs on dedicated servers that run a lightweight GUI with pretty much just a web browser. These emergency VMs work with VNC, but VNC isn’t running all the time. You SSH into them and as part of the login it fires up a VNC session and when you log off it tears the connection down. So you SSH in with 2FA, this starts the VNC Server (with a password), when you finish you log out and it clears down the VNC session. I also installed a cron job that checks the devices external IP address and emails you when it changes (for when it goes into a residential setting).
Here’s how it’s setup
- Install Raspberry OS (with desktop)
- Enable SSH
- Add to .bashrc
if [[ -n $SSH_CONNECTION ]] ;
- Add to .bash_logout
vncserver kill :1
- Enable 2FA for SSH login
- Install the following script for emails if IP changes
When you arrive onsite, connect the pi zero to the network, add a port forward for port 22 and 5901 to the pi. Check the IP emailer works (see link above).
Over the last month I have migrated my home server from a Gen 8 HP Microserver to a Lenovo P500 workstation. There are many reasons for my migration the two biggest were that I was being constrained by the amount of RAM the Microserver could take (16Gb vs 512Gb), the processor was also becoming a bit of a bottle neck.
The second was that in my professional life I have moved from VMWare ESXi to Proxmox and my home lab was the only ESXi server that I was left managing, it also meant I wasn’t reflecting my professional install base so making it hard to test things.
Migrations are horrible, no matter how much planning you do, they take time and suck! No matter how many trials and tests you do there will always be something.
I used an old desktop PC with a 500Gb SATA drive and a 240Gb SSD to migrate all servers other than the Windows server (not enough space or grunt).
Although exceptionally boring and probably of no interest to anyone this was my migration plan..
- Shutdown new host
- NIC in new host
- Check Second Network Card
- Restore firewall
- Copy all Proxmox machines from test proxmox host
- Copy latest backup
- Run Full backup c:\backups\backup.bat
- Check USB disk on another PC
- Close OneDrive
- Restart PC
- Check OneDrive is stopped
- Shutdown VM
- Convert System Disk
- Check Proxmox Boot
- Move 2Tb disks to think station
- Create new ZFS 2Tb for File Server
- Boot File Server
- Add 1.8Tb disk
- Setup OneDrive if needed
20/02/2020 20:12 <JUNCTION> data [d:\data]
17/09/2020 08:48 <JUNCTION> media [D:\media]
17/09/2020 09:07 <JUNCTION> server backups [D:\backups]
- Start OneDrive
- Undisable Start with Windows OneDrive
- Check shares
- Remove “to watch” from backups
- USB pass through
- Setup Proxmox Backups (Exclude File Server d drive)
- Remove SSDs from Microserver and check
- Rebuild Test Proxmox as Hobby PC with 240Gb SSD
- Take old Hobby PC
- Check 120Gb SSDs
- 2FA for SSH and Proxmox on New Host
- Add New Host to Nagios
This was all in a text file which I constantly updated and changed during the actual migration. It went well and there were only a couple of hiccups. The testing had paid off.
Hopper – New Host
Running proxmox with a number of Windows, Linux and BSD VMs.
- Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz (1 Socket)
- 48Gb RAM
- 2x 480Gb NAS SSD (ZFS), 2x 3Tb NAS SATA (ZFS), 1x 2Tb SATA (Backups)
The two USB cables – One going to an external HDD for file level backups, the second goes to the Pi KVM (for keyboard and mouse control)
Tron – Pi
4Tb USB Drive
Backup Pi (Rsync and rclone)
IP KVM – Pi 4 (2Gb)
Power/Data Splitter at the back
USB to HDMI Capture Card
Both are cabled into the network.
The Pi 2 only has 100Mbps network, so it’s likely to need replacing soon to keep up with my internet, but for now it works! The Pi 3+ has “Gb network” however due to it using the same USB bus it can only realistically achieve 300Mbps.
This part of the project nearly got it’s own page… However, I don’t have much to say! One of the biggest drawbacks of migrating to the workstation was that I lost iLO (intelligent lights out / IPMI). I use iLO rarely but it is an incredibly useful when you do need it!
I was looking at aftermarket cards and IP based KVMs and they are expensive! I couldn’t justify the cost for a single host or the amount of time I use it.
Then I came across Pi KVM, it looked hugely daunting until I started reading about it. For simple KVM features (and a host of other features) it was incredibly easy to build a Pi 4 KVM (you can use other Pi generations but you will need to do more work). Just one cable and an HDMI capture card and it just works!
They are also developing their own Pi HAT with all the features (including power management (i.e. remote reboot)), I’ll probably buy one when they are released as I can think of a number of locations where a sub £100 KVM would be a life saver, especially with the remote reboot abilities.
Pi KVM can be found here: https://www.pikvm.org/
Another Pi KVM project can be found here: https://tinypilotkvm.com/
Bits I bought to make my Pi KVM
- 2Gb Pi 4 – Pi Hut
- USB to HDMI – Amazon UK
- Power Splitter (I did run it off the host to start, but decided this introduced a dependence I didn’t want) – Tindie UK Based Seller
That was it! I had a case, power supply and SD card knocking about any way… When the hat is released I will need to think about a different case.
Let's Encrypt using DNS on Cloudkey 2+ & Backups
Update: 16/02/2021 – For CK2+ off controller backups see this great how-to: https://lazyadmin.nl/home-network/backup-unifi-controller-to-cloud/
Update: 22/12/2020 – Use this script instead: https://glennr.nl/s/unifi-lets-encrypt
Stolen from the UI community site, but copied here in case Unifi change their forums again or it gets lost to the mists of time (Original URL: https://community.ui.com/questions/How-To-Lets-Encrypt-with-Cloud-Key-and-DNS-Challenge/)
Uses the domain my-domain.xyz, one shortcoming is that my current external DNS provider doesn’t have an API, so I have to manually complete the challenge every 3 months, but the whole process takes just a few minutes so I’m not too concerned.
All steps are performed directly on the Cloud Key.
First, install Git and obtain the Let’s Encrypt code:
cd /home sudo apt-get update sudo apt-get install git git clone https://github.com/letsencrypt/letsencrypt
Next, generate a certificate, specifying that you want to use a DNS challenge for proving ownership of the domain.
certbot certonly --manual --preferred-challenges dns --email firstname.lastname@example.org --domains unifi.my-domain.xyz
In this example, unifi.my-domain.xyz is an internal hostname that resolves to my cloud key, and email@example.com is an email address where I’d like Let’s Encrypt to send me a reminder when the certificate is about to expire.
Since we are using a DNS challenge, you will be prompted to create a TXT record with your DNS provider. Let’s Encrypt will confirm that the DNS record is visible from their cloud infrastructure, thus proving you own the domain, and it will grant your certificate.
Next, stop unifi, since we’re about to mess with its certificates:
service unifi stop
Next, make a backup of the existing certificate data and remove it:
mkdir cert_backup cp -r /etc/ssl/private/ cert_backup rm /etc/ssl/private/cert.tarrm /etc/ssl/private/ssl-cert-snakeoil.key
Next, export your newly-granted Let’s Encrypt certificate into a format that Unifi understands:
cd /etc/letsencrypt/live/unifi.my-domain.xyz/ openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out cert.p12 -name unifi -password pass:your_certificate_password
Here, your_certificate_password is a temporary password of your choosing to protect the exported certificate.
Next, import the Let’s Encrypt certificate into the Unifi keystore:
keytool -importkeystore -deststorepass aircontrolenterprise -destkeypass aircontrolenterprise -destkeystore /usr/lib/unifi/data/keystore -srckeystore cert.p12 -srcstoretype PKCS12 -srcstorepass your_certificate_password -alias unifi
Note that aircontrolenterprise is the Unifi keystore password; it is not a password of your choosing.
In this command, /usr/lib/unifi/data/keystore is actually a symlink pointing to the /etc/ssl/private directory from earlier.
At this point, the Unifi Controller will work with your Let’s Encrypt certificate, but recall that the Cloud Key has a separate internal nginx-based webserver to handle OS configuration options.
Next, replace the default certificates in the location nginx is expecting them, and make sure the permissions are correct:
cp fullchain.pem /etc/ssl/private/cloudkey.crt cp privkey.pem /etc/ssl/private/cloudkey.key chown root:ssl-cert /etc/ssl/private/* chmod 640 /etc/ssl/private/* tar -cvf cert.tar * chown root:ssl-cert cert.tar chmod 640 cert.tar
Finally, restart nginx, protect and start the Unifi controller.
service nginx restart service unifi-protect restart service unifi start
At this point, they should come up and be using your Let’s Encrypt certificate.
Set a calendar reminder for ~2 months from now so you don’t forget to redo this before the certificate expires!
This guide simply re-arranges the hard work that others have done into a solution that fits my specific needs. The author (pcoldren) used bits and pieces from the following resources while writing it:
Yummy, yummy Pi's - December Update
I’ve decided this will become a running update of the Pi’s I am using and what I am doing with them. Updates will be posted to the top of the page.
December 11th – Update
In addition to the Pi’s below, I now have two more in use
Pi 2 – Backup Pi – Using rsync and rclone to manage all my backups locally and to sync to OneDrive for Business.
Pi 4 2Gb – Pi KVM – Find out more in this post
I have also managed to purchase a Pi 2 v1.2 to go on the Pi versions board. This completes my collection of historical Pi Bs. When the next version of Pi’s come out the Pi 4’s will slowly be retired to the board!
I am also working on a project with Pi Zero WH’s to create a multizone audio system using Volumio, this project will make use of HifiBerry’s popular DAC Hats as well as some custom integration work. I currently have 3 Pi “audio zones” and am awaiting the hats to begin testing.
- Pi 2, Pi 3+, Pi 4 8Gb
- Pi Zero WH
- Pi 2 1.2 ready for mounting
- Pi 3+ – First home assistant server migrated to new proxmox host
Why no love for the Pi A or Compute module? Although I have a good collection of old Pi’s you may notice that I don’t have any Pi A or Pi Compute modules on the list. This is because I don’t use them! I’ve never had a use for the compute modules. I do have a Pi A in a wildlife camera, but this currently isn’t being used. I love the Pi B and Zero form factors which is why I use them the most, if I have a project that ever uses the other form factors, I may well collect the back catalogue of those too!
Original Post – November 5th
From the moment they were announced I knew that the way I did computing at home had changed. Ideal as test boxes, development, media players and now even mini ESX servers! I’ve used them for many things…
The Pi’s I currently have in use are:
- Pi 4 1Gb – Kitchen LibreELEC
- Pi 4 2Gb – 2nd Device in Lego Room
- Pi 4 4Gb – Bedroom LibreELEC
- Pi 400 – 2nd Device in Study
- Pi 3+ – CCTV Viewer
- Pi 3+ – Garage
- Pi 3+ – Home Assistant
I have used them for other projects in the past including getting started with Home Assistant, mini ESXi Server, custom automations, OSMC media player, Plex Server, learning things with Ali, Wildlife Cameras, the list goes on. I hope they are around for a long time to come!
In the gallery below you can see the latest Pi 400, my display of Pi’s from the original Pi to the Pi 3 B+ (with space for the Pi4 1, 2, 4Gb version… the 8Gb version will start a new board). Next are my Pi’s ready for use (Pi Zero WH, Pi 3 B+ and Pi 4 8Gb), I also have a Pi 2 in the cupboard should I need something older to play with and yes that is a ZX Spectrum +2 behind them. Finally my Pi Zero Board up to the latest Pi Zero WH.
Rack Mount Lego Server
Stolen from a LinkedIn post.
|Element ID (BrickLink)||Description||Quantity|
|11211||Brick – Modified 1×2 with Studs on 1 Side||2|
|2412a||Tile – Modified 1×2 Grille without bottom lip||9|
|2431||Tile – 1×4||4|
|26603||Tile – 2×3||1|
|3003||Brick – 2×2||1|
|3004||Brick – 1×2||1|
|3010||Brick – 1×4||8|
|3020||Plate – 2×4||2|
|3023||Plate – 1×2||4|
|3024||Plate – 1×1||6|
|3068a||Tile – 2 x 2 without Groove||3|
|3069a||Tile – 1 x 2 without Groove||6|
|3622||Brick – 1×3||2|
|63864||Tile – 1 x 3||2|
|69729||Tile – 2×6 (not many colour choices for this one)||1|
|87079||Tile – 2 x4||1|
|92438||Plate – 8×16||1|
Alternative for the front (avoiding 69729 – Tile – 2×6) for more colour options!
|3005||Brick – 1×1||2|
|3068a||Tile – 2×2||1|
|26603||Tile – 2×3||1|
|87079||Tile – 2×4||1|
|87087||Brick – Modified 1×1 with stud on 1 side||4|
Using the above list you don’t require – 69729, 11211, 3004 or 26603 – from the first list!
The code can also be sent to an already activated Shelly. This will stay on for ttt seconds and then turn off automatically. Turn off and after ttt-seconds automatically turn on: (ttt replace with the desired time in seconds!)