HPE iLO affects ESXi management agents – hosts in “not responding”

The last months we have had several issues with ESXi hosts going in a “Not responding” status. The VMs are still active and online in this scenario, but the ESXi cannot be managed. This also affets backup as it won’t be able to reach the VMs through the APIs.

Previously we have normally just restarted the management agents on the host and it has been able to connect to vCenter and after this we have managed to migrate the VMs off the host. Lately this hasn’t worked and we have been forced to boot the host with the result of the VMs getting rebooted by HA and eventually started on a different host.… continue reading

Exploring monitoring endpoints in the vCenter Server Appliance (VCSA) REST API

For a long time, actually since we migrated to the VCSA in 6.5 last year, I’ve wanted to utilize the REST API in the appliance to have some monitoring of them.

For several reasons I’ve had to put that on hold, one of them being that there seems to be something wrong with the back-end authentication calls. I get authentication errors on certain calls no matter which user I am logged in with (also the vsphere.local admin account).

 

After deciding to check it more closely I eventually found a few errors in the VCSA logs which in turn led me to this article by Ryan Harris.… continue reading

You’re in!

This month I was accepted as a vExpert for the first time! In total VMware announced 233 new vExperts this summer for their second half announcement in the program.

The vExpert program is not a technical certification. VMware states: The judges selected people who were particularly engaged with their community and who had developed a substantial personal platform of influence in those communities.

I am very proud and honored to be included in this community. I have for a long time consumed the content produced by many of these great contributors and talented individuals and now to be a part of the same is humbling but also a great achievement for me.… continue reading

VCSA 6.7 Upgrade error – The mystery of how the installer connected to the wrong VM

When trying to upgrade our lab vcenter from 6.5 to 6.7 this week we encountered a strange error.

Our lab environement is running vSphere 6.5 on VCSA and we are running with an external PSC. So when starting the upgrade of the PSC I got an error early in the process, while connecting to the source VCSA.

Error when deploying appliance

 

I had remembered that I’ve seen some strange errors before if the root password of the appliance was expired. This was not the case here, but I did change the password and reboot the appliance to see if that solved the problem.

As I got the same error on the next try I tested an earlier 6.7 (the GA) version to see if that had the same error and it failed on that as well.… continue reading

Automating disk limits in vSphere

Following up on my last post on Limiting disk i/o in vSphere we will now see how to automate this.

First off we need some way to identify what limit a specific disk should have.

You can do this in multiple ways, you could use an external source, you can use Tags or as we’ve done, Storage Policies.

All our VMs and VMDKs have a Storage Policy to control the Storage it will be compliant with. We have named our policies so we can identify what kind of limit it should have.
We can now use this to set the limit corresponding to the policy.… continue reading

Working with disk limits in PowerCLI

This will be a post following on the previous one on how to control disk i/o in vSphere

That post showed how you set IOPS limits either through the UI or with PowerCLI.

Even though you set the limit on individual disks you need to work through the VM. And when you retrieve the disk limits for a VM you’ll get back limits on disks identified by the device key and not the label.

To make this a bit easier to work with I’ve created some Powershell functions that wraps the *-VMResourceConfiguration cmdlets and one that gives you the label of the disks when you query the limits for a VM.… continue reading

Limiting disk i/o in vSphere

As a Service provider we need to have some way of limiting individual VMs from utilizing too much of our shared resources.

When it comes to CPU and Memory this is rarely an issue as we try to not over-committing these resources, at least not the Memory. For CPU we closely monitor counters like CPU Ready and Latency to ensure that our VMs will have access to the resources they need.

For storage this can be more difficult. Where we usually have 50-60 VMs on a host we will probably have hundreds on a Storage Array (SAN). Of course the SAN should be spec’ed to handle the IOPS and Throughput you need, but you also need to balance the amount of disk space available and maybe most importantly, the cost.… continue reading

Customizing ESXi installation with kickstart files and PXE boot

This blog post will be building on a previous post where I built a small PXE server environment for ESXi installation.

In this post we will enhance the PXE install with customized kickstart files specific for the hardware we want to install.

There’s two new components to discuss here. The kickstart file (ks.cfg) it self and how to point to it during PXE boot.

Let’s take a look at the current environment

The tftp server root is located at /var/lib/tftpboot and my images is stored as directories under this directory. The default (and at this point only) PXE menu is configured in the pxelinux.cfg/default file (used in legacy PXE boot)

Kickstart scripts

We’ll start customizing with adding a generic kickstart file under a new ks directory.… continue reading

Slides and scripts from VMUG sessions

I had the privilege of delivering 3 sessions at VMUG Norway this week in Oslo, Trondheim and Bergen.

With the extremely nice weather in Norway this week in mind the attendance were great and as always the discussions were valuable.

My session on vSphere Performance monitoring were the short version of the blog series I did about how we built our solution for doing performance monitoring of vSphere with InfluxDB and Grafana, and how we easily can customize with adding metrics and datasources.

The main goal of my session was to demonstrate how easy it is to get started with a project like this and get som actual value.… continue reading

Speaking at VMUG Norway

I’ll be speaking at VMUG Norway’s meetings this May.

As always there will be “three sessions in three cities”.

  • Oslo, May 29th
  • Trondheim, May 30th
  • Bergen, May 31st

The topic for my session will be how we have built our own vSphere Performance monitoring solution which I’ve also done a blog series about.

The VMUG meetings are free, for more information check out https://www.vmug.com/norway. I hope you’re able to join!… continue reading