Work update :)

May 22, 2018 – 2:04 pm

Well, as stated in my last post, I am now working with VMware technologies exclusively for the next 9 months or so. Working on vSphere upgrades (team of 3 doing around 300 hosts), environment supporting 7000 VMs. Pretty much the largest deployment I have worked in.

I am also somewhat involved in a new NSX deployment with the VMware TAM, looking forward to get more hands on with it! Been certified for 3 years now and never got a chance to work with it in real life scenarios.

I am also working on a script to document VSAN configurations, disk groups etc… Will post here when completed!


New adventure!

March 12, 2018 – 5:00 pm

I resigned from my current position today, to take on a new challenge: self-employment. So SUBnet192 will officially become “inc.” this week if all goes well! I am taking on a new mandate as a VMware Integrator for a large transport company in Montreal. I expect to be posting a lot more VMware content this year hopefully!

vExpert 2018

March 10, 2018 – 12:41 pm

I have the honor of being recognized for the 2nd year as a VMware vExpert! I will do my best to share even more VMware content this year!

Fixing stale content libraries in vCenter Appliance

February 3, 2018 – 3:11 pm

Last year we had to rebuild our vCenter (I commented about that in a previous post) using a backup/restore procedure provided by VMware. After the restore, the Update Manager was broken (it’s now fixed) and the Content Libraries were also lost. I didn’t have time to fix it until today. I opened a case with VMware and was provided with the procedure to manually clear out the stale entries for the database.

Unable to delete the stale entry of the content library from the web client. The issue is seen upon restoring the VCDB database post re-deploy. The ‘dcli’ commands are also failing to remove the content library since the same is unable to list the stale entry.


  • Snapshot the vCenter appliance
  • SSH to the VCSA appliance, and enter the shell.
  • Open the Postgres console:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Display the Content Library tables content

VCDB=# select id,name from cl_library;

id | name
8bc3326e-3a0b-49e6-9be0-acb18b91d4b2 | Test
3b4280d3-b878-4428-85d0-867cb8252e66 | Stale

VCDB=# select * from cl_library_storage;

library_id | storage_id
8bc3326e-3a0b-49e6-9be0-acb18b91d4b2 | ff3f7160-0731-4d01-9cc0-a2cce03ab3ce
3b4280d3-b878-4428-85d0-867cb8252e66 | 1325cd29-0b3e-4be4-8445-625da0dce452
(2 rows)

VCDB=# select * from cl_storage;

id | storageuri | type
ff3f7160-0731-4d01-9cc0-a2cce03ab3ce | Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888 | Datastore
1325cd29-0b3e-4be4-8445-625da0dce452 | Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888 | Datastore
(2 rows)

VCDB=# select id,name from vpx_datastore where id=11;

id | name
11 | DS1
(1 row)

Note: As per the snippets above, we see the id for the library is “3b4280d3-b878-4428-85d0-867cb8252e66”.

The storage id for the same library is “1325cd29-0b3e-4be4-8445-625da0dce452”.

When we check the location, we see that the library is saved in the datastore with id “11” (Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888)

We can get the datastore name from the datastore table, which is “DS1” here.

To remove stale entry follow below order.

VCDB=# delete from cl_library_storage where library_id=’3b4280d3-b878-4428-85d0-867cb8252e66′;
VCDB=# delete from cl_storage where id=’1325cd29-0b3e-4be4-8445-625da0dce452′;
VCDB=# delete from cl_library where id=’3b4280d3-b878-4428-85d0-867cb8252e66′;

The output for each of the above command would return “DELETE 1”,Once deleted, we can log in to the web client.

Note: Don’t forget the “;” at the end of each line. If your prompt is VCDB-#, that means it’s waiting for an end-of-line character (the “;”).


Skype For Business – Can not update OnPremLineURI as the user has dirsynced onpremise LineURI

September 24, 2017 – 5:24 pm

We had some legacy conference rooms that were renamed over the years, and as part of the infrastructure cleanup project we’re doing, I changed several attributes in the AD account to clean that up (anything that had to do with email/sip address was changed). Resync’ed with AD Connect and all worked well. Several hours later, we changed their licenses from an E3 license to a Skype Plan 2 license. From this point forward, I was unable to run “set-csuser -OnPremLineURI, as I was receiving an error message: “Can not update OnPremLineURI as the user has dirsynced onpremise LineURI.” Looking at the attributes, I could see that the OnPremLineURI and LineURI attributes were the same, and the OnPremLineURIManuallySet was “False” (as opposed to all our user accounts which have this set to “True”).

Say what? Didn’t have that problem a few hours before… So I checked everything I could and couldn’t figure out the issue, so I called Microsoft support. After a week of providing logs, answering irrelevant questions and getting nowhere, I finally got a call back from an “engineer” who told me basically there were 2 options.

  1. We stop syncing those accounts with AD Connect and they could manually update the LineURI for us <- Not acceptable.
  2. Delete the account and recreate. <- really???? For an attribute issue?

So I asked which was the on premises attribute that was converted into the LineURI and got not clear answer from them. So I tried this:

  1. Set the msRTCSIP-Line attribute (it was “not set”) to some dummy value and triggered an AD Connect sync.
  2. Checked in AD Connect to see if the attribute was properly synced and it was.
  3. Checked the online attributes with Get-CSOnlineUser for the OnPremLineURI value and it was the one I had set on premises.
  4. Cleared the attribute on premises and repeated steps 2 and 3.
  5. This time, both the OnPremLineURI and LineURI attributes were blank!
  6.  Tried running the the set-csuser -OnPremLineURI and it worked this time!

Moral of the story:

  1. Microsoft Support “Engineers” are clueless about troubleshooting anything. I have yet to understand the logic when they troubleshoot something. We’re 0/5 so far with cases we’ve opened with them, we always end up finding the solution ourselves. I had one case where while troubleshooting an Intune issue, the support engineer was taking screengrabs of the generic menus (not even our own settings – just the interface! As if he didn’t know what it was supposed to look like. Scary!)
  2. Most issues we’ve encountered so far have been attribute replication issues. Sometimes removing the licenses from the users and putting them back resolved the issues, or forcing a new value then reverting to the actual proper value (like in this case) fixes the issue.

I would have expected them to come up with this scenario as a first option, but they kept insisting that we had the attribute set on premises (which we didn’t) and that there was nothing to do.

Reconfigure Update Manager on VCSA

August 21, 2017 – 10:44 am

We had an issue with our vCenter appliance 6.5 when we upgraded to 6.5u1, that required a full redeployment. We backed up the database and redeployed the appliance. We took that opportunity to rename the appliance, which we were told was the only way to do that (backup/redeploy/restore) by VMware support.

After redeploying, everythin worked except for Update Manager. I narrowed it down to the service trying to use the old name still. So I went in to the MOB configuration and deleted the extension as we used to do in the Windows world. Well, there is no documentation out there to re-register the service on the appliance.

The only thing I had found was:

/usr/lib/vmware-updatemgr/bin/updatemgr-util register-vc

which fails…

So I opened a case with VMware support and 1 hour of drilling down and support came back with this:

Backup the configuration files in case…

cd /usr/lib/vmware-updatemgr/bin
cp extension.xml extension.xml_backup
cp vci-integrity.xml vci-integrity.xml_backup
cp jetty-vum-ssl.xml jetty-vum-ssl.xml_backup

Then re-register the extension with this command:

./vmware-vciInstallUtils -C /usr/lib/vmware-updatemgr/bin/ -L /var/log/vmware/vmware-updatemgr/ -I /usr/lib/vmware-updatemgr/bin/ -v -p 80 -U administrator@vsphere.local -P “password” -S /usr/lib/vmware-updatemgr/bin/extension.xml -O extupdate

chown updatemgr:updatemgr vci-integrity.xml

service-control –start vmware-updatemgr

Rolling back a vSphere upgrade

February 26, 2017 – 8:40 pm

Ahem… Being a vExpert doesn’t mean you don’t make stupid mistakes like everybody else…

I pushed an update to vSphere 6.5 on a host and tada! Purple screen… I rebooted, crossing my fingers that it was a one time thing but no, it came back. Then it dawned on me to check the HCL… The server is an HP DL585 G7 and it isn’t supported.

I opened a case with VMware to see if there’s was a way to recover without having to reconfigure everything (not the end of the world but…) And there is!

I was directed to this KB that points out that during boot there is a rollback feature! Never noticed/knew about that one. Been a while since I upgraded environments. Glad to see this!


vExpert 2017

February 15, 2017 – 8:36 pm

I was awarded a “vExpert” for 2017 by VMware! What an honour! Never thought I’d qualify for it, but I guess my humble efforts are recognized. Thank you VMware!

Introduction to VMware Photon OS

February 8, 2017 – 3:34 pm

I wanted to play with Photon, so I was looking for some ideas/purposes to deploy a Photon VM in a Windows “shop” 🙂

There’s always some docker packages you can find that can be useful (SMTP server for test/dev environments for example). One good use I found is to build our internal NTP server.

Here’s how I do it:

Create DNS record

an A record for the hostname, and a CNAME for an alias to refer to it in your various systems ( for example).

Set Static IP Address

mv /etc/systemd/network/ /etc/systemd/network/

then edit the file using VI








Set Hostname

hostnamectl set-hostname MYNTP.MYLAB.COM

hostname MYNTP



Disable IPTABLES (I know, not the most secure thing but that’s what I do)

Edit /etc/systemd/scripts/iptables

iptables -P INPUT ACCEPT




Update OS

tdnf update


set timezone

ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime


Install nano

tdnf install nano


Enable NTP Server

tdnf install ntp

nano /etc/ntp.conf

— add the following configs:





tinker panic

restrict netmask nomodify notrap

restrict default kod nomodify notrap nopeer


restrict -6 ::1

driftfile /var/lib/ntp/drift/ntp.drift


Start service

systemctl start ntpd

systemctl enable ntpd

systemctl status ntpd


Troubleshooting commands

ntpq -p

date -R

ntpdate -q


Stretched cluster VM localisation

November 29, 2016 – 8:28 pm

A colleague asked me to help him out with an issue he had. On a stretched cluster, some VMs had storage on Site A but were powered-on on site B and that’s a no-no in a normal situation. There’s no built-in mechanism other than using DRS groups to do so. So I assembled bits and pieces I found online to create a script that would serve this purpose by creating 2 DRS groups and populating them based on which datastores the VMs were on.

Here’s the script:

# These groups must already exist
# VMs-On-SiteA
# VMs-On-SiteB
# Hosts-SiteA
# Hosts-SiteB
# And the following rules
# Bind-To-SiteA (should run)
# Bind-To-SiteB (should run)

# Initialize variables

$vCenter = “vCenterName.subnet192.lab”
$vCenterUser = “ServiceAccount@vsphere.local”
$vCenterPass = “MyPassword”
$ClusterName = “ClusterName”
$SiteA_Datastores = “SiteA*”
$SiteB_Datastores = “SiteB*”

$SiteA_DRSGroup = “VMs-On-SiteA”
$SiteB_DRSGroup = “VMs-On-SiteB”

# Update-DrsVMGroup

function Update-DrsVMGroup {
param (

$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$groupVM = New-Object VMware.Vim.ClusterGroupSpec
#Operation edit will replace the contents of the GroupVMName with the new contents seleced below.
$groupVM.operation = “edit”

$groupVM.Info = New-Object VMware.Vim.ClusterVmGroup
$groupVM.Info.Name = $groupVMName

Get-VM $VMs | %{
$groupVM.Info.VM += $_.Extensiondata.MoRef
$spec.GroupSpec += $groupVM

#Apply the settings to the cluster

# vCenter Connection

Connect-VIServer $vCenter -User $vCenterUser -password $vCenterPass -WarningAction SilentlyContinue

# Housekeeping


# Populate the groups

$Cluster = Get-Cluster $ClusterName
$AllVMs = $Cluster| Get-VM

$SiteA_VMs = Get-Datastore $SiteA_Datastores | Get-VM
$SiteB_VMs = Get-Datastore $SiteB_Datastores | Get-VM

Update-DrsVMGroup -VMs $SiteA_VMs -groupVMName $SiteA_DRSGroup
Update-DrsVMGroup -VMs $SiteB_VMs -groupVMName $SiteB_DRSGroup

# Run DRS to move VMS to the proper places

Get-DrsRecommendation -Cluster $Cluster -Refresh
Apply-DrsRecommendation -DrsRecommendation -RunAsync