Colourful Norwich skyline illustration

Michael Sage

IT, Digital & Culture

Outcome Based Delivery

Outcome Based Delivery

I hate presenteeism, let me get that out of the way first… I hate the person who is first in and last out and their smugness. It drives me insane. Flex sheets, holiday balances, minimum hours, etc, etc… who cares? I don’t. What I want from my staff is the job to be done, sometimes this will take 2 hours in a day, sometimes 12… Do they have to do those hours in the office? Hell no. Costa, McDonald’s, home all suitable work locations. Working at 2 in the morning or 11 at night, I don’t care as long as the job is done (and you are working when you want too). Why do I feel like this? Mainly because we have unnecessarily shackled ourselves to Victorian values, I accept you have to be in a factory when it’s opening and making widgets, but for any modern office based team there is no need. The tools at our disposal now make it possible to work from anywhere at any time. My preferred way of looking at this is we have reconfigured our day. The slides below is how Microsoft describe how we used to spend our time and how we do now. My day is certainly made up of “moments” now, I go into the office to socialise with the team and to have meetings that are best face to face. I work from home one day a week, this gives me the freedom to see Alice off to school and be there when she gets home. My work pattern on my work from home days goes something like this get up, go for a run, grab coffee, work from 6-7, see Alice to school, work 7.30 – 11, have a break do bits and pieces, work 13-15, see Alice home from school and eat, work 20-22.30… As you can see my day is about the same length as if I go in, but… I get more done as I don’t have to focus on anything else. I also don’t spend 4 hours of the day commuting. Soooo what do I need to give the team the ability to this? It’s simple… The right tools. Personally I have settled on Office 365, but Google’s G-Suite or Amazon WorkXXX or ZoHo will all give you the ability to enable the workforce to do this.

OPNSense & Duo

OPNSense & Duo

Set the OPNSense LDAP connection to use the DN rather than the domain\user format or DUO fails to recognise it as the same user. By default the Authentication Proxy doesn’t require 2FA for the first bind in a connection. This is to support systems that bind as a service account, search for the user account, and then bind as the user. It looks like your system may connect and bind as the service account, then disconnects, then connects again to bind as the end user. Look at the exempt_primary_bind and exempt_ou_1 options  and try settingexempt_primary_bind=false and exempt_ou_1=the DN of the service account.

OPNSense & OVH/SyS

This is much simpler in OPNSense

Add gateway

System -> Gateways -> All

Add Gateway

Change Interface to WAN.

Add your primary OVH IP gateway in the “Gateway” field (this will almost certainly not be the gateway for your IP address block) and tick the “far gateway” box.

Hit save and then reload

Adding more IPs

Go to the firewall -> virtual IPs

Add an IP Alias, add the IPs one at a time with a /32

Your IPs are now ready to use!

LAN Internet

Firewall -> NAT -> Outbound

Manual Outbound NAT rule generation. If it isn’t created automatically add a rule with the Interface of WAN, source of your internal IP (192.168.1.x/24) & any VPN IP leave everything else as default and save. Check that your WAN address is chosen rather than interface address (or your outgoing traffic will come from any of your external IPs)

Monitoring ESXi with Nagios

Monitoring ESXi with Nagios

Using the excellent box292_check_vmware plugin and check_esxi_hardware.py script, you can get really great visibility of your ESXi servers.

Notes
For the box293 plugin, you need to have a VMA (details in the manual below).

For the ESXi Hardware you need to install pywbem (apt-get install python-pywbem) and on ESXi 6.5 you need to enable wbem (on the server type: esxcli system wbem set –enable true)

Firewall ports: 443 & 5989 are needed between the monitoring server, VMA and ESXi Hosts

ESXi Checks
Add a server on the VMA – port 443 needs to be open

/usr/lib/vmware-vcli/apps/general/credstore_admin.pl add --server IP.ADDRESS  --username root 

Check it works

/usr/lib/nagios/plugins/check_by_ssh -E 1 -l vi-admin -H VMA.IP -C "~/box293_check_vmware.pl --server ESXi.IP --check Host_OS_Name_Version"

Host Datastore Check

/usr/lib/nagios/plugins/check_by_ssh -E 1 -l vi-admin -H VMA.IP -C "~/box293_check_vmware.pl --server ESXi.IP --check Datastore_Usage --name "ESXI Datastore Name" --warning datastore_free:750 --critical datastore_free:700"

$ARG1$ server
$ARG2$ datastore name
$ARG3$ warning Gb
$ARG4$ critical Gb

Host Snapshot(s) Check

/usr/lib/nagios/plugins/check_by_ssh -E 1 -l vi-admin -H VMA.IP -C "~/box293_check_vmware.pl --server ESXi.IP --check Guest_Snapshot --host ESXi.IP --warning snapshot_age:5 --critical snapshot_age:15"

$ARG1$ Server
$ARG2$ Warning
$ARG3$ Critical

Patching a stand alone ESXi Host

Patching a stand alone ESXi Host

Before we start download the latest patch for ESXi from VMware.

The patches are cumulative. I tend to only patch on the major updates.
I am using HP hardware so make sure that you download the custom image if you can. Using the standard image broke my install, thankfully the built in roll back took care of things!
Copy the latest patch to one of your datastores, in the example below I have copied it to a directory called patch on Datastore1.

First check which version of ESXi and patch level you are running:
~# esxcli system version get
   Product: VMware ESXi
   Version: 6.5.0
   Build: Releasebuild-4564106
   Update: 0
   Patch: 0
Now ensure all the VMs are powered off and the host node is in maintenance mode. Now we query the image profiles that are contained in the patch
~# esxcli software sources profile list --depot=/vmfs/volumes/datastore1/patch/update-from-esxi6.5.zip

Name                              Vendor        Acceptance Level
--------------------------------  ------------  ----------------
ESXi-6.5.0-20171103001-standard   VMware, Inc.  PartnerSupported
In the HP Custom image there is only one profile, but you may see many options here, unless you know otherwise just use the -standard one. Now we know the name of the profile we can update the host. I recommend doing a dry run first.
~ #  esxcli software profile update --depot=/vmfs/volumes/datastore1/patch/update-from-esxi6.5.zip --dry-run --profile=ESXi-6.5.0-20171103001-standard

Update Result
   Message: Dryrun only, host not changed. The following installers will be applied: [BootBankInstaller]
   Reboot Required: true
   VIBs Installed:...
   VIBs Removed...
Once you are happy you can run it without the dry run option.
~ #  esxcli software profile update --depot=/vmfs/volumes/datastore1/patch/update-from-esxi6.5.zip  --profile=ESXi-6.5.0-20171103001-standard

Update Result
   Message: Dryrun only, host not changed. The following installers will be applied: [BootBankInstaller]
   Reboot Required: true
   VIBs Installed:...
   VIBs Removed...
Once is complete you need to reboot the host.
~ # reboot
When the box is rebooted run the following, to check the new version has applied.
~ # esxcli system version get
    Product: VMware ESXi
    Version: 6.5.0
    Build: Releasebuild-5310538
    Update: 0
    Patch: 19
If the update fails, simply reboot the server and ESXi will roll back.

Hyper-V & OVH – Workgroup

Hyper-V & OVH – Workgroup

On Hyper-V Server

  • Enable Remote Management
  • Enable Remote Desktop
  • Set Computername
  • Enable WSMan and PS-Remoting:
Enable-PSRemoting  
Enable-WSManCredSSP -Role server  

On Managing Client

  • Configure hosts or DNS entry for Hyper-V Server
  • Ensure network profile is “Private”:
Set-NetConnectionProfile -InterfaceAlias Ethernet -NetworkCategory Private  
  • Temporarily start WinRM and add Hyper-V Server as Trusted Host:
Start-Service -Name winrm  
Set-Item WSMan:\localhost\Client\TrustedHosts -Value "hyperv-server.home.local"  
Stop-Service -Name winrm  
  • Enable Hyper-V Management Tools in Windows Features
  • Add Hyper-V Administrator credential with cmdkey:
cmdkey /add:HYPERV-SERVER /user:Administrator /pass:  
  • Launch Hyper-V Manager and connect to server (don’t specify user)
    • or, launch Hyper-V Manager with runas:
runas /user:HYPERV-SERVER\Administrator /netonly "mmc virtmgmt.msc"  

Summary

After lots of mucking around, this is the bare-minimum working configuration I’ve found to remotely manage a Hyper-V server in a non-domain environment.

I’m still not 100% certain why Hyper-V Manager does not work with supplied credentials and the cmdkey or runas workaround is necessary.

Stolen from: https://blog.ropnop.com/remotely-managing-hyper-v-in-a-workgroup-environment

Hyper-V & OVH – Network

Hyper-V & OVH – Network

Network (powershell script)

 

Create an External Switch from the command line

This is the hard part that really had me banging my head of the wall.  If you create an external switch you lose connectivity.

You need to determine the Adapter name to use.  Do to this from a Power Shell prompt run:


Get-NetAdapter

This will list the name of your adapters.  Figure out which one you want to use for your switch.
In my case it was Ethernet 2.

Save the Power Shell Script below to a ps1 file (say c:\MakeSwitch.ps1).  You can run:
Notepad c:\makeswitch.ps1
To do this.  Update it with the correct name determined above.

<#
Make Switch for So You Start Dedicated Server
Usaage:
execute the command Get-NetAdapter
This lists the ethernet adatpers.
Replace “ethernet 2” below with the name of your connected physical adapter
Adapted from:
http://blogs.technet.com/b/heyscriptingguy/archive/2013/10/09/use-powershell-to-create-virtual-switches.aspx
#>

Import-Module Hyper-V
$ethernet = Get-NetAdapter -Name “ethernet 2”
New-VMSwitch -Name externalSwitch -NetAdapterName $ethernet.Name -AllowManagementOS $true -Notes ‘Parent OS, VMs, LAN’
New-VMSwitch -Name privateSwitch -SwitchType Private -Notes ‘Internal VMs only’
New-VMSwitch -Name internalSwitch -SwitchType Internal -Notes ‘Parent OS, and internal VMs’

This actually creates three switches, one of each type.   (external, internal and private).

OVH and pfSense

OVH and pfSense

To create a route up to 192.168.23.254 (your main OVH IP), on an interface having no IP in this range, I use the commands:
route add -net 192.168.23.254/32 -iface em0
route add default 192.168.23.254  
The first line tell the firewall that IP address 192.168.23.254 is on the side of the em0 interface (em0 is my WAN interface), the second one use this address as the default gateway. Install shellcmd into pfSense and add the two commands above, this will make it survive a reboot. Adding more IPs Go to the firewall -> virtual IPs Add an IP Alias, add the IPs one at a time with a /32 Your IPs are now ready to use! LAN Internet Firewall -> NAT -> Outbound Manual Outbound NAT rule generation. If it isn’t created automatically add a rule with the Interface of WAN, source of your internal IP (192.168.1.x/24) leave everything else as default and save.