I resigned from my current position today, to take on a new challenge: self-employment. So SUBnet192 will officially become “inc.” this week if all goes well! I am taking on a new mandate as a VMware Integrator for a large transport company in Montreal. I expect to be posting a lot more VMware content this year hopefully!
I have the honor of being recognized for the 2nd year as a VMware vExpert! I will do my best to share even more VMware content this year!
Last year we had to rebuild our vCenter (I commented about that in a previous post) using a backup/restore procedure provided by VMware. After the restore, the Update Manager was broken (it’s now fixed) and the Content Libraries were also lost. I didn’t have time to fix it until today. I opened a case with VMware and was provided with the procedure to manually clear out the stale entries for the database.
Symptom:
Unable to delete the stale entry of the content library from the web client. The issue is seen upon restoring the VCDB database post re-deploy. The ‘dcli’ commands are also failing to remove the content library since the same is unable to list the stale entry.
Resolution:
Display the Content Library tables content
id | name
————————————–+——-
8bc3326e-3a0b-49e6-9be0-acb18b91d4b2 | Test
3b4280d3-b878-4428-85d0-867cb8252e66 | Stale
VCDB=# select * from cl_library_storage;
library_id | storage_id
————————————–+————————————–
8bc3326e-3a0b-49e6-9be0-acb18b91d4b2 | ff3f7160-0731-4d01-9cc0-a2cce03ab3ce
3b4280d3-b878-4428-85d0-867cb8252e66 | 1325cd29-0b3e-4be4-8445-625da0dce452
(2 rows)
VCDB=# select * from cl_storage;
id | storageuri | type
————————————–+————————————————————-+———–
ff3f7160-0731-4d01-9cc0-a2cce03ab3ce | Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888 | Datastore
1325cd29-0b3e-4be4-8445-625da0dce452 | Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888 | Datastore
(2 rows)
VCDB=# select id,name from vpx_datastore where id=11;
id | name
—-+——
11 | DS1
(1 row)
Note: As per the snippets above, we see the id for the library is “3b4280d3-b878-4428-85d0-867cb8252e66”.
The storage id for the same library is “1325cd29-0b3e-4be4-8445-625da0dce452”.
When we check the location, we see that the library is saved in the datastore with id “11” (Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888)
We can get the datastore name from the datastore table, which is “DS1” here.
To remove stale entry follow below order.
The output for each of the above command would return “DELETE 1”,Once deleted, we can log in to the web client.
Note: Don’t forget the “;” at the end of each line. If your prompt is VCDB-#, that means it’s waiting for an end-of-line character (the “;”).
We had some legacy conference rooms that were renamed over the years, and as part of the infrastructure cleanup project we’re doing, I changed several attributes in the AD account to clean that up (anything that had to do with email/sip address was changed). Resync’ed with AD Connect and all worked well. Several hours later, we changed their licenses from an E3 license to a Skype Plan 2 license. From this point forward, I was unable to run “set-csuser roomname@company.com -OnPremLineURI, as I was receiving an error message: “Can not update OnPremLineURI as the user has dirsynced onpremise LineURI.” Looking at the attributes, I could see that the OnPremLineURI and LineURI attributes were the same, and the OnPremLineURIManuallySet was “False” (as opposed to all our user accounts which have this set to “True”).
Say what? Didn’t have that problem a few hours before… So I checked everything I could and couldn’t figure out the issue, so I called Microsoft support. After a week of providing logs, answering irrelevant questions and getting nowhere, I finally got a call back from an “engineer” who told me basically there were 2 options.
So I asked which was the on premises attribute that was converted into the LineURI and got not clear answer from them. So I tried this:
Moral of the story:
I would have expected them to come up with this scenario as a first option, but they kept insisting that we had the attribute set on premises (which we didn’t) and that there was nothing to do.
We had an issue with our vCenter appliance 6.5 when we upgraded to 6.5u1, that required a full redeployment. We backed up the database and redeployed the appliance. We took that opportunity to rename the appliance, which we were told was the only way to do that (backup/redeploy/restore) by VMware support.
After redeploying, everythin worked except for Update Manager. I narrowed it down to the service trying to use the old name still. So I went in to the MOB configuration and deleted the extension as we used to do in the Windows world. Well, there is no documentation out there to re-register the service on the appliance.
The only thing I had found was:
/usr/lib/vmware-updatemgr/bin/updatemgr-util register-vc
which fails…
So I opened a case with VMware support and 1 hour of drilling down and support came back with this:
Backup the configuration files in case…
cd /usr/lib/vmware-updatemgr/bin
cp extension.xml extension.xml_backup
cp vci-integrity.xml vci-integrity.xml_backup
cp jetty-vum-ssl.xml jetty-vum-ssl.xml_backup
Then re-register the extension with this command:
./vmware-vciInstallUtils -C /usr/lib/vmware-updatemgr/bin/ -L /var/log/vmware/vmware-updatemgr/ -I /usr/lib/vmware-updatemgr/bin/ -v yourvcenter.mycompany.com -p 80 -U administrator@vsphere.local -P “password” -S /usr/lib/vmware-updatemgr/bin/extension.xml -O extupdate
chown updatemgr:updatemgr vci-integrity.xml
service-control –start vmware-updatemgr
Ahem… Being a vExpert doesn’t mean you don’t make stupid mistakes like everybody else…
I pushed an update to vSphere 6.5 on a host and tada! Purple screen… I rebooted, crossing my fingers that it was a one time thing but no, it came back. Then it dawned on me to check the HCL… The server is an HP DL585 G7 and it isn’t supported.
I opened a case with VMware to see if there’s was a way to recover without having to reconfigure everything (not the end of the world but…) And there is!
I was directed to this KB that points out that during boot there is a rollback feature! Never noticed/knew about that one. Been a while since I upgraded environments. Glad to see this!
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033604
I was awarded a “vExpert” for 2017 by VMware! What an honour! Never thought I’d qualify for it, but I guess my humble efforts are recognized. Thank you VMware!
I wanted to play with Photon, so I was looking for some ideas/purposes to deploy a Photon VM in a Windows “shop” 🙂
There’s always some docker packages you can find that can be useful (SMTP server for test/dev environments for example). One good use I found is to build our internal NTP server.
Here’s how I do it:
Create DNS record
an A record for the hostname, and a CNAME for an alias to refer to it in your various systems (time.mylab.com for example).
Set Static IP Address
mv /etc/systemd/network/10-dhcp-eth0.network /etc/systemd/network/static.network
then edit the static.network file using VI
[Match]
Name=eth0
[Network]
Address=10.10.10.10/24
Gateway=10.10.10.1
DNS=10.10.10.50
Domains=mylab.com
Set Hostname
hostnamectl set-hostname MYNTP.MYLAB.COM
hostname MYNTP
reboot
Disable IPTABLES (I know, not the most secure thing but that’s what I do)
Edit /etc/systemd/scripts/iptables
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
Update OS
tdnf update
set timezone
ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime
Install nano
tdnf install nano
Enable NTP Server
tdnf install ntp
nano /etc/ntp.conf
— add the following configs:
server 0.ca.pool.ntp.org
server 1.ca.pool.ntp.org
server 2.ca.pool.ntp.org
server 3.ca.pool.ntp.org
tinker panic
restrict 10.0.0.0 netmask 255.0.0.0 nomodify notrap
restrict default kod nomodify notrap nopeer
restrict 127.0.0.1
restrict -6 ::1
driftfile /var/lib/ntp/drift/ntp.drift
—
Start service
systemctl start ntpd
systemctl enable ntpd
systemctl status ntpd
Troubleshooting commands
ntpq -p
date -R
ntpdate -q 0.ca.pool.ntp.org 1.ca.pool.ntp.org
A colleague asked me to help him out with an issue he had. On a stretched cluster, some VMs had storage on Site A but were powered-on on site B and that’s a no-no in a normal situation. There’s no built-in mechanism other than using DRS groups to do so. So I assembled bits and pieces I found online to create a script that would serve this purpose by creating 2 DRS groups and populating them based on which datastores the VMs were on.
Here’s the script:
# These groups must already exist
# VMs-On-SiteA
# VMs-On-SiteB
# Hosts-SiteA
# Hosts-SiteB
# And the following rules
# Bind-To-SiteA (should run)
# Bind-To-SiteB (should run)
#===================================================================
# Initialize variables
#===================================================================
$vCenter = “vCenterName.subnet192.lab”
$vCenterUser = “ServiceAccount@vsphere.local”
$vCenterPass = “MyPassword”
$ClusterName = “ClusterName”
$SiteA_Datastores = “SiteA*”
$SiteB_Datastores = “SiteB*”
$SiteA_DRSGroup = “VMs-On-SiteA”
$SiteB_DRSGroup = “VMs-On-SiteB”
#===================================================================
# Update-DrsVMGroup
# http://vniklas.djungeln.se/2012/06/28/vsphere-cluster-host-vm-rule-affinity-with-powercli/
#===================================================================
function Update-DrsVMGroup {
param (
$VMs,
$groupVMName)
$spec = New-Object VMware.Vim.ClusterConfigSpecEx
$groupVM = New-Object VMware.Vim.ClusterGroupSpec
#Operation edit will replace the contents of the GroupVMName with the new contents seleced below.
$groupVM.operation = “edit”
$groupVM.Info = New-Object VMware.Vim.ClusterVmGroup
$groupVM.Info.Name = $groupVMName
Get-VM $VMs | %{
$groupVM.Info.VM += $_.Extensiondata.MoRef
}
$spec.GroupSpec += $groupVM
#Apply the settings to the cluster
$cluster.ExtensionData.ReconfigureComputeResource($spec,$true)
}
#===================================================================
# vCenter Connection
#===================================================================
Connect-VIServer $vCenter -User $vCenterUser -password $vCenterPass -WarningAction SilentlyContinue
#===================================================================
# Housekeeping
#===================================================================
Clear
#===================================================================
# Populate the groups
#===================================================================
$Cluster = Get-Cluster $ClusterName
$AllVMs = $Cluster| Get-VM
$SiteA_VMs = Get-Datastore $SiteA_Datastores | Get-VM
$SiteB_VMs = Get-Datastore $SiteB_Datastores | Get-VM
Update-DrsVMGroup -VMs $SiteA_VMs -groupVMName $SiteA_DRSGroup
Update-DrsVMGroup -VMs $SiteB_VMs -groupVMName $SiteB_DRSGroup
#===================================================================
# Run DRS to move VMS to the proper places
#===================================================================
Get-DrsRecommendation -Cluster $Cluster -Refresh
Apply-DrsRecommendation -DrsRecommendation -RunAsync
I regularly post about the latest certification exams I took… And today was the first of two for Cisco’s CCNA Datacenter certification.
In my new role, I am doing a lot of Cisco UCS and a bit of Nexus configurations and since I had little to no exposure to these platforms, I wanted to get up to speed with the platforms. With 25+ years experience, i don’t think certifications mean that much anymore but it’s my way to challenge myself to learn new stuff, and have something to show for all the efforts as well. As long as my employers pay for the material/exams 🙂
On my list for the coming months/year are:
– 2nd CCNA Datacenter Exam
– Renew my VCAP (assuming it also renews the VCP) for vSphere
– If I get good hands-on projects, the VCIX-NV for NSX.
So it seems I have a busy year ahead 🙂
