vSAN Configuration Report

August 15, 2018 – 11:15 am

Here’s a script I put together from various sources to report on the disk group configurations for a VSAN cluster. Hope you find it useful!

########################################################################
# Script : vSAN Configuration Report
# Author : Marc Bouchard
# Version : 1.0
#
# Generate a report of the vSAN Configuration.
# – Clusters with vSAN
# – Disk groups on each host
# – Disk details for each disk group, including HBA each disk is attached to
#
########################################################################
Clear
$VSANArray = @()
$vSANClusters = Get-VsanClusterConfiguration | where {$_.vsanenabled -eq “true”}
# Collect data from all clusters, nodes, disk groups and disks
Foreach ($VSCluster in $VSANClusters) {
    write-host “[$VSCluster]”.ToUpper() -foregroundcolor White
    $AllHosts=Get-VMHost -Location $VSCluster.Name| Sort Name
    Foreach ($ESXiHost in $AllHosts) {
        write-host “————————————————————————————” -foregroundcolor green
        write-host $EsxiHost.Name -foregroundcolor green
        write-host “————————————————————————————” -foregroundcolor green
        $AllvSANDG = Get-vSANDiskGroup -VMHost $EsxiHost.Name | Sort Name
        Write-Host “Number of Disk Groups : ” $AllvSANDG.Count
        $DGCount = 0
        foreach ($DiskGroup in $AllvSANDG) {
            $DGCount += 1
            write-host $DiskGroup.Name -foregroundcolor yellow
            $DGDisks=get-vsandisk -diskgroup $DiskGroup
            Write-Host “Number of disks in this group: ” $DGDisks.Count -foregroundcolor yellow
            $DiskCount = 0
            foreach ($vSANDisk in $DGDisks) {
                $DiskCount += 1
                $Details=$vSANDisk.extensiondata
                $DiskName=$Details.CanonicalName
                Write-Host “Processing disk #$DiskCount : $DiskName” -foregroundcolor cyan
                $SCSILUN = ($ESXiHost | get-scsilun -CanonicalName $Details.CanonicalName)
                $vmHBA = $SCSILUN.RuntimeName.Substring(0, ($SCSILUN.RuntimeName.IndexOf(“:”)))
                $VSANObject = New-Object System.Object
                $VSANObject | Add-Member -type NoteProperty -name Cluster       -Value $VSCluster.Name
                $VSANObject | Add-Member -type NoteProperty -name Host          -Value $ESXiHost.Name
                $VSANObject | Add-Member -type NoteProperty -name vmHBA         -Value $vmHBA
                $VSANObject | Add-Member -type NoteProperty -name GroupID       -Value $DGCount
                $VSANObject | Add-Member -type NoteProperty -name GroupName     -Value $DiskGroup.Name
                $VSANObject | Add-Member -type NoteProperty -name DiskID        -Value $DiskCount
                $VSANObject | Add-Member -type NoteProperty -name DiskName      -Value $Details.CanonicalName
                $VSANObject | Add-Member -type NoteProperty -name State         -Value $Details.OperationalState
                $VSANObject | Add-Member -type NoteProperty -name Vendor        -Value $Details.Vendor
                $VSANObject | Add-Member -type NoteProperty -name Model         -Value $Details.Model
                $VSANObject | Add-Member -type NoteProperty -name Revision      -Value $Details.Revision
                $VSANObject | Add-Member -type NoteProperty -name IsSSD         -Value $Details.SSD
                $VSANObject | Add-Member -type NoteProperty -name QueueDepth    -Value $Details.QueueDepth
                $VSANObject | Add-Member -type NoteProperty -name Capacity      -Value (($Details.Capacity.BlockSize * $Details.Capacity.Block) / 1GB)
                $VSANArray += $VSANObject
            }
        }
    }
}
$VSANArray | Export-CSV vsanConfig.csv

VMware Log Insight issues

August 9, 2018 – 9:55 am

I have encountered an issue in our lab environment with Log Insight. It is configured as a cluster with a master and two worker nodes. We were unable to authenticate through the GUI either with an AD account or the local admin account.

So we tried rebooting the environment to see if that would clear up. After the reboot, the web interface was no longer coming up!

Symptoms:

  • Unable to authenticate using domain account and local administrator account.
  • Root account on console worked
  • After reboot, web interface was no longer responding

Investigation:

  • The Internal Load Balancer’s Virtual IP was not responding to pings
  • The LogInsight service on the Master server was not coming up. It was working fine on the two Workers servers

Resolution:

  • Added a new virtual hard disk to the master since it was the only one where the service didn’t come up.
    • Problem was still there
  • Added a new virtual hard disk to both workers to match the config of the master.
    • Problem solved

Commands used:

  • SSH to the servers and use “df” to show the disk volume space
  • service loginsight restart
  • cd /usr/lib/loginsight/application/lib/apache-cassandra-2.0.17/bin
    ./nodetool status

The first column shows UN : « up/normal ». They were showing DN: “Down/normal”

Work update :)

May 22, 2018 – 2:04 pm

Well, as stated in my last post, I am now working with VMware technologies exclusively for the next 9 months or so. Working on vSphere upgrades (team of 3 doing around 300 hosts), environment supporting 7000 VMs. Pretty much the largest deployment I have worked in.

I am also somewhat involved in a new NSX deployment with the VMware TAM, looking forward to get more hands on with it! Been certified for 3 years now and never got a chance to work with it in real life scenarios.

I am also working on a script to document VSAN configurations, disk groups etc… Will post here when completed!

 

New adventure!

March 12, 2018 – 5:00 pm

I resigned from my current position today, to take on a new challenge: self-employment. So SUBnet192 will officially become “inc.” this week if all goes well! I am taking on a new mandate as a VMware Integrator for a large transport company in Montreal. I expect to be posting a lot more VMware content this year hopefully!

vExpert 2018

March 10, 2018 – 12:41 pm

I have the honor of being recognized for the 2nd year as a VMware vExpert! I will do my best to share even more VMware content this year!

Fixing stale content libraries in vCenter Appliance

February 3, 2018 – 3:11 pm

Last year we had to rebuild our vCenter (I commented about that in a previous post) using a backup/restore procedure provided by VMware. After the restore, the Update Manager was broken (it’s now fixed) and the Content Libraries were also lost. I didn’t have time to fix it until today. I opened a case with VMware and was provided with the procedure to manually clear out the stale entries for the database.

Symptom:
Unable to delete the stale entry of the content library from the web client. The issue is seen upon restoring the VCDB database post re-deploy. The ‘dcli’ commands are also failing to remove the content library since the same is unable to list the stale entry.

Resolution:

  • Snapshot the vCenter appliance
  • SSH to the VCSA appliance, and enter the shell.
  • Open the Postgres console:
/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

Display the Content Library tables content

VCDB=# select id,name from cl_library;

id | name
 ————————————–+——-
8bc3326e-3a0b-49e6-9be0-acb18b91d4b2 | Test
3b4280d3-b878-4428-85d0-867cb8252e66 | Stale

VCDB=# select * from cl_library_storage;

library_id | storage_id
————————————–+————————————–
8bc3326e-3a0b-49e6-9be0-acb18b91d4b2 | ff3f7160-0731-4d01-9cc0-a2cce03ab3ce
3b4280d3-b878-4428-85d0-867cb8252e66 | 1325cd29-0b3e-4be4-8445-625da0dce452
(2 rows)

VCDB=# select * from cl_storage;

id | storageuri | type
————————————–+————————————————————-+———–
ff3f7160-0731-4d01-9cc0-a2cce03ab3ce | Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888 | Datastore
1325cd29-0b3e-4be4-8445-625da0dce452 | Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888 | Datastore
(2 rows)

VCDB=# select id,name from vpx_datastore where id=11;

id | name
—-+——
11 | DS1
(1 row)

Note: As per the snippets above, we see the id for the library is “3b4280d3-b878-4428-85d0-867cb8252e66”.

The storage id for the same library is “1325cd29-0b3e-4be4-8445-625da0dce452”.

When we check the location, we see that the library is saved in the datastore with id “11” (Datastore:datastore-11:f271f3f1-a924-4519-a3f5-e9a75d962888)

We can get the datastore name from the datastore table, which is “DS1” here.

To remove stale entry follow below order.

VCDB=# delete from cl_library_storage where library_id=’3b4280d3-b878-4428-85d0-867cb8252e66′;
VCDB=# delete from cl_storage where id=’1325cd29-0b3e-4be4-8445-625da0dce452′;
VCDB=# delete from cl_library where id=’3b4280d3-b878-4428-85d0-867cb8252e66′;

The output for each of the above command would return “DELETE 1”,Once deleted, we can log in to the web client.

Note: Don’t forget the “;” at the end of each line. If your prompt is VCDB-#, that means it’s waiting for an end-of-line character (the “;”).

 

Skype For Business – Can not update OnPremLineURI as the user has dirsynced onpremise LineURI

September 24, 2017 – 5:24 pm

We had some legacy conference rooms that were renamed over the years, and as part of the infrastructure cleanup project we’re doing, I changed several attributes in the AD account to clean that up (anything that had to do with email/sip address was changed). Resync’ed with AD Connect and all worked well. Several hours later, we changed their licenses from an E3 license to a Skype Plan 2 license. From this point forward, I was unable to run “set-csuser roomname@company.com -OnPremLineURI, as I was receiving an error message: “Can not update OnPremLineURI as the user has dirsynced onpremise LineURI.” Looking at the attributes, I could see that the OnPremLineURI and LineURI attributes were the same, and the OnPremLineURIManuallySet was “False” (as opposed to all our user accounts which have this set to “True”).

Say what? Didn’t have that problem a few hours before… So I checked everything I could and couldn’t figure out the issue, so I called Microsoft support. After a week of providing logs, answering irrelevant questions and getting nowhere, I finally got a call back from an “engineer” who told me basically there were 2 options.

  1. We stop syncing those accounts with AD Connect and they could manually update the LineURI for us <- Not acceptable.
  2. Delete the account and recreate. <- really???? For an attribute issue?

So I asked which was the on premises attribute that was converted into the LineURI and got not clear answer from them. So I tried this:

  1. Set the msRTCSIP-Line attribute (it was “not set”) to some dummy value and triggered an AD Connect sync.
  2. Checked in AD Connect to see if the attribute was properly synced and it was.
  3. Checked the online attributes with Get-CSOnlineUser for the OnPremLineURI value and it was the one I had set on premises.
  4. Cleared the attribute on premises and repeated steps 2 and 3.
  5. This time, both the OnPremLineURI and LineURI attributes were blank!
  6.  Tried running the the set-csuser -OnPremLineURI and it worked this time!

Moral of the story:

  1. Microsoft Support “Engineers” are clueless about troubleshooting anything. I have yet to understand the logic when they troubleshoot something. We’re 0/5 so far with cases we’ve opened with them, we always end up finding the solution ourselves. I had one case where while troubleshooting an Intune issue, the support engineer was taking screengrabs of the generic menus (not even our own settings – just the interface! As if he didn’t know what it was supposed to look like. Scary!)
  2. Most issues we’ve encountered so far have been attribute replication issues. Sometimes removing the licenses from the users and putting them back resolved the issues, or forcing a new value then reverting to the actual proper value (like in this case) fixes the issue.

I would have expected them to come up with this scenario as a first option, but they kept insisting that we had the attribute set on premises (which we didn’t) and that there was nothing to do.

Reconfigure Update Manager on VCSA

August 21, 2017 – 10:44 am

We had an issue with our vCenter appliance 6.5 when we upgraded to 6.5u1, that required a full redeployment. We backed up the database and redeployed the appliance. We took that opportunity to rename the appliance, which we were told was the only way to do that (backup/redeploy/restore) by VMware support.

After redeploying, everythin worked except for Update Manager. I narrowed it down to the service trying to use the old name still. So I went in to the MOB configuration and deleted the extension as we used to do in the Windows world. Well, there is no documentation out there to re-register the service on the appliance.

The only thing I had found was:

/usr/lib/vmware-updatemgr/bin/updatemgr-util register-vc

which fails…

So I opened a case with VMware support and 1 hour of drilling down and support came back with this:

Backup the configuration files in case…

cd /usr/lib/vmware-updatemgr/bin
cp extension.xml extension.xml_backup
cp vci-integrity.xml vci-integrity.xml_backup
cp jetty-vum-ssl.xml jetty-vum-ssl.xml_backup

Then re-register the extension with this command:

./vmware-vciInstallUtils -C /usr/lib/vmware-updatemgr/bin/ -L /var/log/vmware/vmware-updatemgr/ -I /usr/lib/vmware-updatemgr/bin/ -v yourvcenter.mycompany.com -p 80 -U administrator@vsphere.local -P “password” -S /usr/lib/vmware-updatemgr/bin/extension.xml -O extupdate

chown updatemgr:updatemgr vci-integrity.xml

service-control –start vmware-updatemgr

Rolling back a vSphere upgrade

February 26, 2017 – 8:40 pm

Ahem… Being a vExpert doesn’t mean you don’t make stupid mistakes like everybody else…

I pushed an update to vSphere 6.5 on a host and tada! Purple screen… I rebooted, crossing my fingers that it was a one time thing but no, it came back. Then it dawned on me to check the HCL… The server is an HP DL585 G7 and it isn’t supported.

I opened a case with VMware to see if there’s was a way to recover without having to reconfigure everything (not the end of the world but…) And there is!

I was directed to this KB that points out that during boot there is a rollback feature! Never noticed/knew about that one. Been a while since I upgraded environments. Glad to see this!

https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1033604

 

vExpert 2017

February 15, 2017 – 8:36 pm

I was awarded a “vExpert” for 2017 by VMware! What an honour! Never thought I’d qualify for it, but I guess my humble efforts are recognized. Thank you VMware!