Creating local ESXi user in a locked down situation and add it to exception list

So my customer asked for a solution to add local users on ESXi hosts that are in lockdown mode. A side quest was to add these to the lockdown exception list. The use case for this is app volumes, they want to be able to keep using them in case the vCenter server goes down. The trick to this that you need to talk to two different viserver entities. The vCenter server and the local ESXi host since you can add those users via vCenter.

Offcourse PowerCLI to the rescue! I decided to do everything in a try catch construction for some error handling and to give some visual output. These cab be stripped if you want but i like some feedback.

Some of the outtakes:

(get-vmhost $vmhost | get-view).ExitLockdownMode()
(get-vmhost $vmhost | get-view).EnterLockdownMode()

These two disable and enable the current lockdown mode, this is necessary before being able to create the local user.

Try {
$account = Get-VMHostAccount -server $vmhost.name -Id $accountName -ErrorAction Stop |
Set-VMHostAccount -server $vmhost.name -Password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($accountPswd))) -Description $accountDescription 
}
Catch   {
$account = New-VMHostAccount -server $vmhost.name -Id $accountName -Password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($accountPswd))) -Description $accountDescription -UserAccount -GrantShellAccess 
}

Some encryption stuff in here but that’s because I dislike having password visible as plain tekst. This first test if the account exists and then sets the password and description. If the user doesn’t exist it will create the user for you.

 $rootFolder = Get-Folder -server $vmhost.name -Name ha-folder-root
 New-VIPermission -server $vmhost.name -Entity $rootFolder -Principal $account -Role admin

This gives the newly created or edited user the admin role. If you want to use a custom role this could be added to the script, we decided to go for the admin role since app volumes needs an awful lot of rights anyway. In that case i would recommend to use a variable for role name and create it per host using new-VIrole

$HostAccessManager = Get-View -Server $vCenter $vmhost.ExtensionData.ConfigManager.HostAccessManager
$HostAccessManager.UpdateLockdownExceptions($accountName)

This simply adds the user to the lockdown exception list.

So now the complete script:

#-------------------------------------------------
# Create local ESXi user with admin rights and lockdown exception while the host is in lockdown mode
# 
# Requires PowerCLI 6.5 or higher (module based not snappin)
# Based on scripts by Luc Dekens and others
#
# Version 1.0
# 09-10-2017
# Created by: Wouter Kursten
#
#-------------------------------------------------
#
# Load the required VMware modules (for PowerShell only)

Write-Host "Loading VMware PowerCLI Modules" -ForegroundColor Green
try	{
    get-module -listavailable vm* | import-module -erroraction stop
}
catch	{
    write-host "No Powercli found" -ForegroundColor Red
}

#Ask for connection information

$vcenter=Read-Host "Enter vCenter server name"
$Target = Read-Host "Which hosts? (i.e. server*)"
$rootpassword = Read-Host "Enter root Password" -AsSecureString
$accountName = $userPassword = Read-Host "Enter New Username"
$accountDescription = $userPassword = Read-Host "Enter New User description"
$accountPswd = Read-Host "Enter New User Password" -AsSecureString
$rootuser="root"

# Connect to vCenter
$connectedvCenter = $global:DefaultVIServer

if($connectedvCenter.name -ne $vcenter){
	Connect-VIServer $vCenter -wa 0 | Out-Null
	Write-Host "Connected"
	Write-Host " "
}

# Get the host inventory from vCenter
$vmhosts = Get-VMHost $Target | Sort Name

foreach($vmhost in $vmhosts){
    try {
        (get-vmhost $vmhost | get-view).ExitLockdownMode()
        write-host "Lockdown disabled for $vmhost" -foregroundcolor green
    }
    catch   {
        write-host "can't disable lockdown for $vmhost maybe it's already disabled" -foregroundcolor Red
    }

    connect-viserver -server $vmhost -user $rootuser -password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($rootpassword))) -wa 0 -notdefault | Out-Null

    Try {
        $account = Get-VMHostAccount -server $vmhost.name -Id $accountName -ErrorAction Stop |
        Set-VMHostAccount -server $vmhost.name -Password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($accountPswd))) -Description $accountDescription 
    }
    Catch   {
        $account = New-VMHostAccount -server $vmhost.name -Id $accountName -Password ([Runtime.InteropServices.Marshal]::PtrToStringAuto([Runtime.InteropServices.Marshal]::SecureStringToBSTR($accountPswd))) -Description $accountDescription -UserAccount -GrantShellAccess 
    }
    
    $rootFolder = Get-Folder -server $vmhost.name -Name ha-folder-root
    New-VIPermission -server $vmhost.name -Entity $rootFolder -Principal $account -Role admin

    #Adding the new user to the Lockdown Exceptions list
    $HostAccessManager = Get-View -Server $vCenter $vmhost.ExtensionData.ConfigManager.HostAccessManager
    $HostAccessManager.UpdateLockdownExceptions($accountName)
     
      
    Disconnect-VIServer $vmhost.name -Confirm:$false  
    try {	
        (get-vmhost $vmhost | get-view).EnterLockdownMode()
        write-host "Lockdown enabled for $vmhost" -foregroundcolor green
    }
    catch   {
        write-host "can't disable lockdown for $vmhost maybe it's already Enabled?" -foregroundcolor Red}
    }
}

    Disconnect-VIServer -Confirm:$false

Future versions of this script will not be edited on here so always check the latest version on github.

 

Is updating @VMware ESXi at @Nutanix really that easy?

Do you want the short answer or the long answer?

Short: Yes!

Long: Absolutely!

The real story:

  • After doing several Acropolis (CVM) & NCC (Health check) upgrades before on my customer’s Nutanix clusters today was the first time to do an actual hypervisor upgrade. Since we run everything on VMware we wanted to go from 5.5u2 to 6.0u2. The first things to do is to check all compatibility charts and with Acropolis at 4.6.4 and NCC at 2.2.8 we had green lights all over the board.
  • What I always do first before doing anything is running a health check. Since NCC 2.2 you can run some of the checks parallel to save some time:
ncc health_checks run_all -parallel=4

2016-10-04-20_32_24-beheerders-desktop

  • After the check and make sure DRS is set on automated and vSphere HA is turned on as it should otherwise you won’t be updating anything!
  • Next up is heading to Prism, no we’re not doing any cli work when a GUI is better and just as fast!
  • The first thing to do is upload the software (Offline bundle zip file from VMware.com and json file from Nutanix.com)

2016-10-04-15_08_02-documents

  • Go to Software Upgrade

2016-10-04-20_47_38-beheerders-desktop

  • Select Upload the hypervisor Binary

2016-10-04-15_08_16-nutanix-web-console

  • Select the binary and the Json files and hit Upload Now

2016-10-04-15_08_37-nutanix-web-console

2016-10-04-15_08_59-nutanix-web-console

  • When this is done hit the arrow besides the upgrade button and select the pre-check (the real upgrade also does this but it is never wrong to check twice!)

2016-10-04-15_10_49-nutanix-web-console

2016-10-04-15_10_58-nutanix-web-console

  • Enter the IP of the vCenter (not DNS!) and credentials

2016-10-04-15_11_36-nutanix-web-console

2016-10-04-15_12_20-nutanix-web-console

2016-10-04-15_12_28-nutanix-web-console

  • This won’t take long but if you get bored hit the Nothing do to button for a game of 2048 presented to you by our friends at Nutanix

2016-10-04-15_12_39-nutanix-web-console

  • You might need to re-open the Software Upgrade but but somewhere it will be done now

2016-10-04-15_17_53-nutanix-web-console

  • When this finishes successful it’s time to hit the real upgrade button

2016-10-04-15_18_47-nutanix-web-console

  • You know what to do here right?

2016-10-04-15_18_57-nutanix-web-console

2016-10-04-15_19_50-nutanix-web-console

2016-10-04-15_20_48-nutanix-web-console

  • The waiting game has started since there will be a lot of vMotion’s and reboots
  • This it might be time for this again

2016-10-04-15_12_39-nutanix-web-console

2016-10-04-15_21_01-nutanix-web-console

  • If you re-open the Software Upgrade bit it will show the versions of ESXi the cluster is now running

2016-10-04-15_21_45-nutanix-web-console

  • Aaaaaaand we’re done

2016-10-04-15_36_56-nutanix-web-console

So actually creating this post took longer then the preparation and actions for the upgrade themselves. For me that was 5 minutes work in preparation and about 20 minutes per host for the upgrade itself.

 

 

 

 

Back to basics: Daily checks

Something I still hear a lot that system engineers take their vSphere environment for granted and hardly check anything on a daily basis. I always point them at Alan Renouf‘s brilliant health check script while there are other ways to get your daily dose of health this one still rocks for me. You can remove unwanted plugins or make different selections of plugins for daily, weekly or monthly checks. Now and then I still hear people that had issues because of snapshots and there is no need for that anymore and hasn’t been for years! This script has saved me lots of times already + it helped me get management support for limiting other people’s access to the environment because they had no idea what they where doing.

Example of the output you can get:

2016-07-03 20_13_59-192.168.0.11 vCheck

 

Timecheck please!

Something I still see now and then, and have had big issues with in the past, is the time on ESXi hosts. Sometimes no ntp servers have been set or the ESXi hosts can’t connect to them. Other times ntp servers have been set but not the time so they’re still off. Normally this shouldn’t be a problem but since a VM always takes on the time of the hosts it is moving to during a vmotion this can cause issues on database servers.

In my last situation the ntp servers where correct but the time was off and somehow never properly synced to the ntp hosts. To fix this I created 2 scripts, one to check the ntp settings and current time and another to set the time.

$style = "<style>BODY{font-family: Arial; font-size: 10pt;}"
$style = $style + "TABLE{border: 1px solid black; border-collapse: collapse;}"
$style = $style + "TH{border: 1px solid black; background: #dddddd; padding: 5px; }"
$style = $style + "TD{border: 1px solid black; padding: 5px; }"
$style = $style + "</style>"
$esxihosts=Get-VMHost | Sort Name | Select Name,  @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, Timezone, @{N="CurrentTime";E={(Get-View $_.ExtensionData.ConfigManager.DateTimeSystem) | Foreach {$_.QueryDateTime().ToLocalTime()}}}, @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ |Where-Object {$_.key-eq "ntpd"}).Running}} 
$esxihosts | convertto-html -head $style -property  name,NTPServer,TimeZone,CurrentTime,ServiceRunning | out-file timecheck.html
start timecheck.html

Nothing fancy, you need to be connected to your vcenter in advance but it makes and opens a nice html file with your ntp settings and current time on your ESXi hosts.

This is the output it makes:

2016-05-24 20_23_21-Mozilla Firefox

Then it was time to make the other script, since sometimes it might take a few secs to set the time I decided to check my local time before every set of a time on an ESXi host.

Get-VMHost | Where-Object {
$t = Get-Date
$dst = $_ | %{ Get-View $_.ExtensionData.ConfigManager.DateTimeSystem }
$dst.UpdateDateTime((Get-Date($t.ToUniversalTime()) -format u))
}

Again nothing fancy but it does the trick perfectly.