Monitoring ESXi with Nagios

Monitoring ESXi with Nagios

Using the excellent box292_check_vmware plugin and check_esxi_hardware.py script, you can get really great visibility of your ESXi servers.

Notes
For the box293 plugin, you need to have a VMA (details in the manual below).

For the ESXi Hardware you need to install pywbem (apt-get install python-pywbem) and on ESXi 6.5 you need to enable wbem (on the server type: esxcli system wbem set –enable true)

Firewall ports: 443 & 5989 are needed between the monitoring server, VMA and ESXi Hosts

ESXi Checks
Add a server on the VMA – port 443 needs to be open

/usr/lib/vmware-vcli/apps/general/credstore_admin.pl add --server IP.ADDRESS  --username root 

Check it works

/usr/lib/nagios/plugins/check_by_ssh -E 1 -l vi-admin -H VMA.IP -C "~/box293_check_vmware.pl --server ESXi.IP --check Host_OS_Name_Version"

Host Datastore Check

/usr/lib/nagios/plugins/check_by_ssh -E 1 -l vi-admin -H VMA.IP -C "~/box293_check_vmware.pl --server ESXi.IP --check Datastore_Usage --name "ESXI Datastore Name" --warning datastore_free:750 --critical datastore_free:700"

$ARG1$ server
$ARG2$ datastore name
$ARG3$ warning Gb
$ARG4$ critical Gb

Host Snapshot(s) Check

/usr/lib/nagios/plugins/check_by_ssh -E 1 -l vi-admin -H VMA.IP -C "~/box293_check_vmware.pl --server ESXi.IP --check Guest_Snapshot --host ESXi.IP --warning snapshot_age:5 --critical snapshot_age:15"

$ARG1$ Server
$ARG2$ Warning
$ARG3$ Critical

Patching a stand alone ESXi Host

Patching a stand alone ESXi Host

Before we start download the latest patch for ESXi from VMware.

The patches are cumulative. I tend to only patch on the major updates.
I am using HP hardware so make sure that you download the custom image if you can. Using the standard image broke my install, thankfully the built in roll back took care of things!
Copy the latest patch to one of your datastores, in the example below I have copied it to a directory called patch on Datastore1.

First check which version of ESXi and patch level you are running:
~# esxcli system version get
   Product: VMware ESXi
   Version: 6.5.0
   Build: Releasebuild-4564106
   Update: 0
   Patch: 0
Now ensure all the VMs are powered off and the host node is in maintenance mode. Now we query the image profiles that are contained in the patch
~# esxcli software sources profile list --depot=/vmfs/volumes/datastore1/patch/update-from-esxi6.5.zip

Name                              Vendor        Acceptance Level
--------------------------------  ------------  ----------------
ESXi-6.5.0-20171103001-standard   VMware, Inc.  PartnerSupported
In the HP Custom image there is only one profile, but you may see many options here, unless you know otherwise just use the -standard one. Now we know the name of the profile we can update the host. I recommend doing a dry run first.
~ #  esxcli software profile update --depot=/vmfs/volumes/datastore1/patch/update-from-esxi6.5.zip --dry-run --profile=ESXi-6.5.0-20171103001-standard

Update Result
   Message: Dryrun only, host not changed. The following installers will be applied: [BootBankInstaller]
   Reboot Required: true
   VIBs Installed:...
   VIBs Removed...
Once you are happy you can run it without the dry run option.
~ #  esxcli software profile update --depot=/vmfs/volumes/datastore1/patch/update-from-esxi6.5.zip  --profile=ESXi-6.5.0-20171103001-standard

Update Result
   Message: Dryrun only, host not changed. The following installers will be applied: [BootBankInstaller]
   Reboot Required: true
   VIBs Installed:...
   VIBs Removed...
Once is complete you need to reboot the host.
~ # reboot
When the box is rebooted run the following, to check the new version has applied.
~ # esxcli system version get
    Product: VMware ESXi
    Version: 6.5.0
    Build: Releasebuild-5310538
    Update: 0
    Patch: 19
If the update fails, simply reboot the server and ESXi will roll back.

Hyper-V & OVH – Workgroup

Hyper-V & OVH – Workgroup

On Hyper-V Server

  • Enable Remote Management
  • Enable Remote Desktop
  • Set Computername
  • Enable WSMan and PS-Remoting:
Enable-PSRemoting  
Enable-WSManCredSSP -Role server  

On Managing Client

  • Configure hosts or DNS entry for Hyper-V Server
  • Ensure network profile is “Private”:
Set-NetConnectionProfile -InterfaceAlias Ethernet -NetworkCategory Private  
  • Temporarily start WinRM and add Hyper-V Server as Trusted Host:
Start-Service -Name winrm  
Set-Item WSMan:\localhost\Client\TrustedHosts -Value "hyperv-server.home.local"  
Stop-Service -Name winrm  
  • Enable Hyper-V Management Tools in Windows Features
  • Add Hyper-V Administrator credential with cmdkey:
cmdkey /add:HYPERV-SERVER /user:Administrator /pass:  
  • Launch Hyper-V Manager and connect to server (don’t specify user)
    • or, launch Hyper-V Manager with runas:
runas /user:HYPERV-SERVER\Administrator /netonly "mmc virtmgmt.msc"  

Summary

After lots of mucking around, this is the bare-minimum working configuration I’ve found to remotely manage a Hyper-V server in a non-domain environment.

I’m still not 100% certain why Hyper-V Manager does not work with supplied credentials and the cmdkey or runas workaround is necessary.

Stolen from: https://blog.ropnop.com/remotely-managing-hyper-v-in-a-workgroup-environment

Hyper-V & OVH – Network

Hyper-V & OVH – Network

Network (powershell script)

 

Create an External Switch from the command line

This is the hard part that really had me banging my head of the wall.  If you create an external switch you lose connectivity.

You need to determine the Adapter name to use.  Do to this from a Power Shell prompt run:


Get-NetAdapter

This will list the name of your adapters.  Figure out which one you want to use for your switch.
In my case it was Ethernet 2.

Save the Power Shell Script below to a ps1 file (say c:\MakeSwitch.ps1).  You can run:
Notepad c:\makeswitch.ps1
To do this.  Update it with the correct name determined above.

<#
Make Switch for So You Start Dedicated Server
Usaage:
execute the command Get-NetAdapter
This lists the ethernet adatpers.
Replace “ethernet 2” below with the name of your connected physical adapter
Adapted from:
http://blogs.technet.com/b/heyscriptingguy/archive/2013/10/09/use-powershell-to-create-virtual-switches.aspx
#>

Import-Module Hyper-V
$ethernet = Get-NetAdapter -Name “ethernet 2”
New-VMSwitch -Name externalSwitch -NetAdapterName $ethernet.Name -AllowManagementOS $true -Notes ‘Parent OS, VMs, LAN’
New-VMSwitch -Name privateSwitch -SwitchType Private -Notes ‘Internal VMs only’
New-VMSwitch -Name internalSwitch -SwitchType Internal -Notes ‘Parent OS, and internal VMs’

This actually creates three switches, one of each type.   (external, internal and private).

OVH and pfSense

OVH and pfSense

To create a route up to 192.168.23.254 (your main OVH IP), on an interface having no IP in this range, I use the commands:
route add -net 192.168.23.254/32 -iface em0
route add default 192.168.23.254  
The first line tell the firewall that IP address 192.168.23.254 is on the side of the em0 interface (em0 is my WAN interface), the second one use this address as the default gateway. Install shellcmd into pfSense and add the two commands above, this will make it survive a reboot. Adding more IPs Go to the firewall -> virtual IPs Add an IP Alias, add the IPs one at a time with a /32 Your IPs are now ready to use! LAN Internet Firewall -> NAT -> Outbound Manual Outbound NAT rule generation. If it isn’t created automatically add a rule with the Interface of WAN, source of your internal IP (192.168.1.x/24) leave everything else as default and save.

Nagios MySQL

Nagios MySQL

Monitoring MySQL servers, we will need to pass through a username and password for MySQL when checking the service availability. Firstly we will create a MySQL user: We start on the MySQL user and we will log in to the database server and create a new user for Nagios to use. I normally use the local check_mysql plugin so there is no need to let the user login from other servers.
CREATE USER 'nagios'@'localhost' IDENTIFIED BY 'password'
Next check by running the check_mysql plugin
sudo -s -u nagios /usr/lib/nagios/plugins/check_mysql -H localhost -u nagios -p password
Add this to you nrpe definition file and restart nagios-nrpe-server. Add to your Nagios server and away you go!

Auto OpenVPN Script

Auto OpenVPN Script

Build a new VPS and then follow the instructions here! https://github.com/Angristan/OpenVPN-install I had an issue with my install to do with routing, the trouble was in the iptables NAT rule. Check if IPv4 forward is enable.
# grep net.ipv4.ip_forward /etc/sysctl.conf
net.ipv4.ip_forward=1
# cat /proc/sys/net/ipv4/ip_forward
1
# sysctl -a | grep "net.ipv4.ip_forward "
net.ipv4.ip_forward = 1
If it’s ok, check your iptables rules.
iptables -t nat -L -vn --line
iptables -L FORWARD -vn --line
I had no nat rule in my nat table, sad times! I added the following and all is well. $IP is your public server IP:
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -j SNAT --to $IP

Cert for ESXi

Cert for ESXi

To generate a certificate request for an ESXi 6.0 host:
 
  1. Open a command prompt and navigate to the OpenSSL directory as previously configured in the Configuring OpenSSL article. By default this is  C:\OpenSSL-Win32\bin.
  2. Run the command:

    openssl req -new -nodes -out rui.csr -keyout rui-orig.key

    This creates the certificate request rui.csr.

  3. Convert the Key to be in RSA format by running these command:

    openssl rsa -in rui-orig.key -out rui.key

Installing and configuring the certificate on the ESXi host 

After the certificate is created, complete the installation and configuration of the certificate on the ESXi 6.0 host:
  1. Navigate to the console of the server to enable SSH on the ESXi 6.0 host.
  2. Log in to the host and then navigate to /etc/vmware/ssl.
  3. Copy the files to a backup location, such as a VMFS volume.
  4. Log in to the host with WinSCP or login locally (my preferred method) and navigate to the /etc/vmware/ssl directory.
  5. Delete the existing  rui.crt and  rui.key from the directory.
  6. Copy the newly created  rui.crt and  rui.key or create them using vi (again my preferred method, I also at the intermediate cert to the .crt file) to the directory using Text Mode or ASCII mode to avoid the issue of special characters (  ^M) appearing in the certificate file.
  7. Type vi rui.crt to validate that there are no extra characters.

    Note: There should not be any erroneous  ^M characters at the end of each line.

  8. Restart the management agents

    /etc/init.d/hostd restart

    /etc/init.d/vpxa restart

Z-Push with MXRoute

Z-Push with MXRoute

I decided to set myself a little challenge to get ActiveSync working for free for a project I am working on. Most ActiveSync clients on the market are licensed directly from Microsoft.

I have an email service from the excellent MXRoute http://www.mxroute.com and I am not using the service much, they don’t currently offer a mobile solution so I thought I would see if it is possible to use a small virtual private server (VPS) to connect to IMAP and pass it to my mobile, turns out you can! I am using MXRoutes London servers, you will need to change the config if you are using any other locations.

Using Z-PUSH (2.3.5) to connect to MX Route

Known Issues:

Z-push syncs the shared address book, I haven’t worked out how to stop this, it’s a minor issue, but you do need to be aware of it.

Z-push fails to get the name of the calendar, so on an activesync device the calendar name shows up as unknown, but it works.

Z-push pulls the tasks through as a calendar, I have turned off syncing tasks.

Z-push itself isn’t the most reliable platform. This isn’t an issue per se, but it is something you need to be aware of.

Pre-Reqs

A VPS or server, I am running Ubuntu 16.04 LTS A MXRoute account Apache installed with an SSL cert (I’m using letencrypt) PHP7 with php-cli and php-soap Instructions

Add the z-push repo by creating /etc/apt/sources.list.d/z-push.list with the content

deb http://repo.z-hub.io/z-push:/final/Ubuntu_16.04/ /

Download and add the repo key to the keychain

wget -qO – http://repo.z-hub.io/z-push:/final/Debian_8.0/Release.key | sudo apt-key add –

Run apt-get update

We then need to install the z-push packages that we need

apt-get install z-push-common z-push-config-apache z-push-backend-caldav z-push-backend-carddav z-push-backend-combined z-push-backup-imap z-push-ipc-sharedmemory

This will install and configure z-push ready to go, we now need to edit the config files to tell it where to get the contacts, calendar and email from.

Start by editing /usr/share/z-push/config.php

Change the section “Default Settings” to match your timezone and ensure that the define(‘USE_FULLEMAIL_FOR_LOGIN’,true); is set to true.

Under the logging settings there is a specialLogUsers value, this is really useful to put a user in here when you need to troubleshoot, this generates debug logs for the named user and which can be found in the /var/log/z-push directory, but remember to remove them when you have done your testing as they can generate massive log files.

The only other setting we need to change is the Backend settings.

Change the backend provider setting to:

define(‘BACKEND_PROVIDER’,’BackendCombined’);

Save and exit the file

Next edit the /usr/share/z-push/backend/combined/config.php

Update it to match the following values

‘backends’ => array(
‘i’ => array(

‘name’ => ‘BackendIMAP’,

),

‘d’ => array(

‘name’ => ‘BackendCardDAV’,

),

‘c’ => array(

‘name’ => ‘BackendCalDAV’,

),

),

‘delimiter’ => ‘/’,

//force one type of folder to one backend

//it must match one of the above defined backends

‘folderbackend’ => array(

SYNC_FOLDER_TYPE_INBOX => ‘i’,

SYNC_FOLDER_TYPE_DRAFTS => ‘i’,

SYNC_FOLDER_TYPE_WASTEBASKET => ‘i’,

SYNC_FOLDER_TYPE_SENTMAIL => ‘i’,

SYNC_FOLDER_TYPE_OUTBOX => ‘i’,

// SYNC_FOLDER_TYPE_TASK => ‘c’,
SYNC_FOLDER_TYPE_APPOINTMENT => ‘c’,

SYNC_FOLDER_TYPE_CONTACT => ‘d’,

SYNC_FOLDER_TYPE_NOTE => ‘c’,

SYNC_FOLDER_TYPE_JOURNAL => ‘c’,

SYNC_FOLDER_TYPE_OTHER => ‘i’,

SYNC_FOLDER_TYPE_USER_MAIL => ‘i’,

SYNC_FOLDER_TYPE_USER_APPOINTMENT => ‘c’,

SYNC_FOLDER_TYPE_USER_CONTACT => ‘d’,

SYNC_FOLDER_TYPE_USER_TASK => ‘c’,

SYNC_FOLDER_TYPE_USER_JOURNAL => ‘c’,

SYNC_FOLDER_TYPE_USER_NOTE => ‘c’,

SYNC_FOLDER_TYPE_UNKNOWN => ‘i’,

),

//creating a new folder in the root folder should create a folder in one backend

‘rootcreatefolderbackend’ => ‘i’,

);

}

}

This tells z-push which backend is responsible for which function.

Next we need to edit the individual service files we will start with email.

Edit the file /usr/share/z-push/backend/imap/config.php

Update the following values, this is assuming you are using MXRoute’s London servers.

// Defines the server to which we want to connect

define(‘IMAP_SERVER’, ‘london.mxroute.com’);

// connecting to default port (143)

define(‘IMAP_PORT’, 993);

// best cross-platform compatibility (see http://php.net/imap_open for options)

define(‘IMAP_OPTIONS’, ‘/ssl/norsh’);

// Mark messages as read when moving to Trash.

// BE AWARE that you will lose the unread flag, but some mail clients do this so the Trash folder doesn’t get boldened

define(‘IMAP_AUTOSEEN_ON_DELETE’, false);

// Since I know you won’t configure this, I will raise an error unless you do.

// When configured set this to true to remove the error

define(‘IMAP_FOLDER_CONFIGURED’, true);

// Folder prefix is the common part in your names (3, 4)

define(‘IMAP_FOLDER_PREFIX’, ”);

// Inbox will have the preffix preppend (3 & 4 to true)

define(‘IMAP_FOLDER_PREFIX_IN_INBOX’, false);

// Inbox folder name (case doesn’t matter) – (empty in 4)

define(‘IMAP_FOLDER_INBOX’, ‘INBOX’);

// Sent folder name (case doesn’t matter)

define(‘IMAP_FOLDER_SENT’, ‘inbox.SENT’);

// Draft folder name (case doesn’t matter)

define(‘IMAP_FOLDER_DRAFT’, ‘inbox.DRAFTS’);

// Trash folder name (case doesn’t matter)

define(‘IMAP_FOLDER_TRASH’, ‘inbox.TRASH’);

// Spam folder name (case doesn’t matter). Only showed as special by iOS devices

define(‘IMAP_FOLDER_SPAM’, ‘inbox.junk’);

// Archive folder name (case doesn’t matter). Only showed as special by iOS devices

define(‘IMAP_FOLDER_ARCHIVE’, ‘ARCHIVE’);

You also want to update the method used for sending emails (I am using an SSL connection to MXRoute)

define(‘IMAP_SMTP_METHOD’, ‘smtp’);

global $imap_smtp_params;

$imap_smtp_params = array(‘host’ => ‘ssl://london.mxroute.com’, ‘port’ => 465, ‘auth’ => true, ‘username’ => ‘imap_username’, ‘password’ => ‘imap_password’ );

The above ‘imap_username’ and ‘imap_password’ are variables and do not need changing to the actual username and password.

Save and close the file.

Next we will setup syncing contacts

Edit /usr/share/z-push/backend/carddav/config.php update the following

// Server protocol: http or https

define(‘CARDDAV_PROTOCOL’, ‘https’);

// Server name

define(‘CARDDAV_SERVER’, ‘london.mxroute.com’);

// Server port

define(‘CARDDAV_PORT’, ‘2080’);

// Address book path

define(‘CARDDAV_PATH’, ‘/rpc/addressbooks/%u/’);

// Server path to the default address book

define(‘CARDDAV_DEFAULT_PATH’, ‘contacts/’);

// Support sync-collection

define(‘CARDDAV_SUPPORTS_SYNC’, false);

Save and close the file

Finally the calendar sync

Edit /usr/share/z-push/backend/caldav/config.php

// Server protocol: http or https

define(‘CALDAV_PROTOCOL’, ‘https’);

// Server name

define(‘CALDAV_SERVER’, ‘london.mxroute.com’);

// Server port

define(‘CALDAV_PORT’, ‘2080’);

// Path

define(‘CALDAV_PATH’, ‘/rpc/calendars/%u/’);

// Default CalDAV folder (calendar folder/principal). This will be marked as the default calendar in the mobile

define(‘CALDAV_PERSONAL’, ‘calendar’);

// If the CalDAV server supports the sync-collection operation

// DAViCal, SOGo and SabreDav support it

// SabreDav version must be at least 1.9.0, otherwise set this to false

// Setting this to false will work with most servers, but it will be slower

define(‘CALDAV_SUPPORTS_SYNC’, false);

// Maximum period to sync.

// Some servers don’t support more than 10 years so you will need to change this

define(‘CALDAV_MAX_SYNC_PERIOD’, 2147483647);

Save and close the file

Although not necessarily I restart apache at this point.

service apache2 restart

On your client device (iOS / Android / Outlook 2013+)

As there is no autodiscovery function you will need to enter the settings manually.

Username: Full email address

Password: Users MXRoute Password

Server: Your VPS address (i.e. z-push.yourdomain.com)

Basic Troubleshooting

Check the apache and z-push logs, both of which can be found in /usr/var/log/

Turn on debugging for a single user (see above for details)

Check you can access activesync on your server https://z-push.yourdomain.com/Microsoft-Server-ActiveSync, you should be promoted for a username and password use the email address and the users mxroute password, you should see a page that gives you the activesync information and tell you that “GET is not supported”.

You should now be good to go!