With a college of mine, I’m setting up a new business. Where going to provide a virtual desktop to (small) customers who have the need for a enterprise enviroment (mail, shared calanders, shared storage etc) but don’t have the budget for it. When we combine all those customers together you have a budget to provide those functionality at a fractions of the costs.
Of course we want to use VMware vSphere and View for this environment. Later I will provide some more details.
First step was to setup a test enviroment so we can show some functionality to customers. All went fine until my lab had a power failure. And of course at home I don’t have a UPS who can back this up. I switched on the power again and my servers start up up (very loud,because in my lab I only have one power supply connected.)
When I opened vCenter again I noticed that some virtual machine have red alarm icons on them. But when I looked at the alarm tap there was no triggered alarm.
So what trigger this red alarm icon. After some investigation I discovered that the ESXi servers where up and running before the storage was available. When the datastores where available the virtual machines where started. But the weren’t properly registered. A simply vMotion to another ESXi host solved the problem.
In my lab I changed the IP address of my vCenter 5.1 appliance.
I thought it would be pretty easy. Just connect to the vCenter appliance management port 5490 with HTTPS.
Login as root and go to the Network tab. Change the IP and reboot the vCenter appliance.
But after the reboot all my vSphere ESXi hosts where marked as “Not responding”. When I did a connect on the ESXi host the ESXi server connected. But after a minute the where marked as “Not responding” again. This minute is the default timeout value when vCenter doesn’t receive any heartbeats from the ESXi hosts. I read that vCenter listens at TCP and UDP 902. So I tried to telnet to the new vCenter IP address at port 902 and I didn’t get a response. So obvious vCenter isn’t listing on the TCP 902.
Then I looked at the vCenter settings. Login on the vCenter appliance with a vSphere client. Select the vCenter, select settings, general. Click “Edit”. Under runtime settings you will see the configured IP address. In my case this was the old IP address. After changing the old IP to the new IP address and a reboot my ESXi hosts kept connected.
During a storage migration the SAN team had to create and delete multiple LUNs for the VMware vSphere 5.1 environment.
Accidentally they deleted a LUN that wasn’t deleted in VMware vSphere. Gladly there were no more virtual machine on that datastore. When I tried to unmount the volume I got the following error:
The message is clear. I cannot unmount the datastore because SIOC is enabled. But I can’t disabled SIOC because the datastore ain’t available anymore. There are 2 solutions for this.
- Reboot the hosts. During boot SIOC won’t be enabled on the datastore anymore and you will be able to unmount the datastore.
- SSH to the ESXi console and stop the SIOC deamon by entering:
Give a Rescan all on the storage adapter and enable the SIOC deamon again with the command:
Now you’re able to unmount the datastore.
Option 2 is nice if you don’t want or are unable to reboot your vSphere hosts.
VMware ThinApp 4.7 is an application virtualization and portable application creator which allows users to package conventional applications so that they are portable. “VMware ThinApp 4.7 Essentials” shows you how to deploy ThinApp packages in order to improve the portability, manageability and compatibility of applications by encapsulating them from the underlying operating system on which they are executed.
VMware ThinApp 4.7 is an application virtualization and portable application creator which allows users to package conventional applications so that they are portable. ThinApp eliminates application conflicts, reducing the need and cost of recoding and regression testing.
Want more information? Check here.
I have done multiple installations of the vCenter Virtual Appliance. Every time I run into the same issue and get the same question from customers. So I though to make a post with all my tips and tricks. This post will be updated when I have a new trick. So check back often.
Change default IP setting before running SSO config
When your vCenter starts for the first time. SSH to the temporary IP address and configure the correct IP settings through Yast. How? Read on.
Change the vCenter name localhost to server name
By default your vCenter name is localhost. In order to change this, login in the vSphere Web Client and goto the vCenter Servers view.
- Select your vCenter named localhost.
- Select the manage tab.
- Select General.
- Click Edit.
- Select Runtime settings.
- At the vCenter Server name change localhost to whatever you want.
Don’t disable IPv6. Your vCenter server won’t work anymore after a reboot.
Setting the correct IP settings for the appliance
In the vCenter Virtual Appliance web interface you can alter the IP setting for the vCenter appliance. This is not enough.
- Login with SSH on your vCenter Appliance server.
- Start yast.
- Goto Network Devices | Network Settings
- Goto Hostname/DNS
- Set the correct Hostname and Domain Name
- Reboot host
Replace default SSL certificates
This one is easy. Look at this KB article.
Active Directory authentication
With vCenter 5.1 VMware introduced Single Sign On (SSO). This is a service on vCenter where you can configure multiple authentication sources. 2 of them are default.
- Localos (where the user root comes from)
- System-domain (here you can create vCenter users)
Most of you would like to configure Active Directory authentication. This can be done in 2 ways.
- Active Directory intergration
Login in on the vCenter Appliance with HTTPS at port 5480 and goto to the tab vCenter Server | Authentication. Place a mark at Active Directory Enabled and provide the Domain name with the corresponding credentials.
- LDAP connection
Login with the vSphere web client on your vCenter server. On the home page goto Administration | Sing-On and Discovery | Configuration. At the tab Identity Sources click the + sing. Select Active Directory and provide the Identity source settings.
No domain.local\username in vSphere client
If you have a Active Directory Identity Source configured you have to login as follow: domain.local\username. This is no problem if you have multiple domains who contain the same usernames. But if you have only one domain configured this can be annoying. If you configure this Identity Source as Default Domain you don’t have the provide the domain name any more.
Login with the vSphere client on you vCenter server. On the home page goto Administration | Sing-On and Discovery | Configuration. At the tab Identity Sources select your Identity Source and click Add to Default Domain button at the top. Your domain will appear in the lower section of the screen. There you can select the domain and use the arrow key’s to change the search order. The on that is on top is searched first.
NTP time synchronization
Of course you want time synchronization for your vCenter server and what better way to do this with NTP.
- Login with SSH on your vCenter Appliance
- Start yast and goto Network Services | NTP Configuration
- Make sure that the Start NTP Daemon is set to Now and on Boot
- Select Add and provide the IP or DNS name of you NTP server.
After you save the configuration NTP is started. You can check the time synchronization with the command: watch “ntpq -p”.
The watch command will execute the ntpq command every 2 second. You can stop this whit Ctrl-C.
Using Firefox on Mac won’t show all the available tabs in the vCenter Virtual Appliance web interface.
This is a bug (still with build 18.104.22.16800 Build 880472). Use a older Firefox.
Change SSH host keys after changing the hostname and IP settings
After you changed the hostname and the IP settings of you vCenter server, you have to regenerate the SSH host keys. “Why?” I hear you asking yourself, “everything works?”. Yes everything is working, and it’s secure. But the host keys are generate with the wrong (temporary) values.
- Delete the ssh_host_* files in /etc/sshd/
- Restart SSHD by entering the command rcsshd restart.
You will see that the host key’s a regenerated.
ESXTOP is a great tool for troubleshooting performance issue in VMware vSphere. In the past I written a post about CPU troubleshooting. One of the main values I mention in this blog post was %RDY.Want to know what it means? Read the post
While checking a vSphere 5.1 environment of a customer I got the following ESXTOP results
As you can see the %USED and %RUN differer a lot. Meaning the virtual machine want more CPU resources (%RUN) than it is actually getting (%USED). A small difference is normal but not up till 50%. %RDY was normal but as you can see %LAT_C is very high. But what does the value %LAT_C means? In the man pages %lat_c is explained as:
%LAT_C Percentage of time the resource pool or world was ready to run but was not scheduled to run because of CPU resource contention.
As you can read, there is a high CPU resource contention. But where does it come from? This is a DELL PowerEdge R620 with 2 Intel E2650 8 cores processors who is running 4 virtual machines. All these virtual machine have 8 vCPUs configured. So you may think that the amount of vCPUs is to high for the amount of pCPUs. But I wouldn’t expect these values. Then I looked a the P state off the CPU, seeing P states of 4, 5 even 11 or 12 explained to me that power saving was enabled in the BIOS for the CPUs. After I disabled power saving (in this case set the performance to maximum) I got the following results in ESXTOP.
As you can see %RUN and %USED are quit the same and %LAT_C is low. This is what we want to see.
When I was installing a new vSphere 5.1 cluster for a customer I had to document the WWNN and the WWPN. After copying and pasting the WWNNs and the WWPN I noticed that on a hosts, the same WWNN and WWPN occur. Of course these have to be unique.
As you can see, this is not what you want.
When I list the WWNN and WWPN through the CLI I get the following result.
As you can see, the last 2 numbers/letters are different.
Conclusion, you cannot trust the vSphere Web Client for listing the correct WWNN and WWPN. Use the CLI instead.
vSphere 5.1 is just released and I’m already implementing it in a test environment for a customer of mine. As of vSphere 5.1 the vSphere Web Client is must improved.
While playing around I wanted to test the logbrowser. The logbrowser you can view, search, and export one or more vCenter Server and ESXi log files at a time using the log browser. You can also export, manage, and view different log types.
Right after I clicked the logbrowser option I got the error:
The release note off vSphere 5.1 say’s the following:
When you click Log Browser in the vSphere Web Client, an Unauthorized Access error appears
When you click the Log Browser link in the vSphere Web Client, an error message appears: Exception: https://<system-address>:12443/vmwb/logbrowser: Unauthorized access. This error occurs after you replace the default vCenter Single Sign On server’s SSL certificate, either directly or by regenerating the certificate in the vCenter Server Appliance.
I didn’t replace or recreate the certificate files but the error is the same. VMware has the following work around.
- Log in to the vSphere Web Client as a Single Sign On administrator.
- Navigate to Administration > Sign-on and Discovery > Configuration, and click the STS Certificate tab.
- Click Edit.
- Select the Single Sign On SSL keystore.
- If Single Sign On is running on a Windows system, select the following file:
C:\Program Files\VMware\Infrastructure\SSOServer\security\server-identity.jks (default path)
- If Single Sign On is running on Linux (vCenter Server Appliance), select the following file:
/usr/lib/vmware-sso/security/server.jks (default path)
- If Single Sign On is running on a Windows system, select the following file:
- Open the Single Sign On server.xml file with a text editor or browser.
- On Windows:
C:\Program Files\VMware\Infrastructure\SSOServer\conf\server.xml (default path)
- On Linux:
/usr/lib/vmware-sso/conf/server.xml (default path)
- On Windows:
- Search for keystorePass="..." on the Connector element. The string in quotes is your password.
- Enter the password in the vSphere Web Client when prompted.
- Select only the displayed chain.
- Click OK and enter the password again.
- Restart the following services: the vSphere Web Client, vCenter Server, vCenter Inventory Service, and the VMware Log Browser. You do not need to restart Single Sign On.
After reading the procedure multiple times I didn’t understand step 4 and 5. When I click Edit as in step 3 I didn’t see any keystore. And when I clicked on the Browse button. I browsed my own desktop. And of course the file server-identity.jks from step 4 isn’t on my desktop.
I copied the file from the vCenter appliance to my desktop with secure copy and used it like described in step 4. The rest of the procedure is correct and you can browse the logfiles.
As vSphere 5.1 is just released, you can run into the problem that the default ISO from VMware doesn’t contain the right drivers for you hardware. During installation you don’t have the option to import a driver. And if no network card is detected, installation won’t continue. In order to fix this problem you have to create your own ISO file. This can be done with PowerCLI.
- Download the VMware-ESXi-5.1.0-xxxxxx-depot.zip (where xxxxxx is the build number) from the VMware website and your driver file (something like Vendor_drivername-version-offline_bundle-xxxxxxxx where xxxxxx is the buildnumber. In my case the file name is: BCM-NetXtremeII-1.0-offline_bundle-553511). Place these files in a directory on your harddrive. I placed the ESXi5.1 depot in c:\vmware\esxi51 and the driver file in c:\vmware\bcm-driver.
- Start PowerCLI and cd to c:\vmware.
- Use the Add-ESXSoftwareDepotcommandlet to add both the ESXi offline bundle and async offline bundle as depots. For example:
Output will be something like:
- Add the vSphere 5.1 software depot file with the command:
Add-ESXSoftwareDepot ./esxi51/VMware-ESXi-5.1.0-799733-depot.zipOutput will be something like:
- Verify the available software packages with the command: Get-EsxSoftwarePackage
- Next step is to clone a existing image profile.With the command Get-EsxImageProfile you’ll get a list of the available profiles.
Output will be something like:
Name Vendor Last Modified Acceptance
————————– —— ————- —————
ESXi-5.1.0-799733-standard VMware mm/dd/yyyy PartnerSupported
ESXi-5.1.0-799733-no-tools VMware mm/dd/yyyy PartnerSupported
Where going to clone the ESXi-5.0.0-456551-no-tools profile to a new profile called ESXi-WilmsenIT. This can be done with the command New-EsxImageProfile. For example:
New-EsxImageProfile -CloneProfile “ESXi-5.1.0-799733-no-tools” -name “ESXi-WilmsenIT” -Vendor “WilmsenIT”Output will be something like:
Name Vendor Last Modified Acceptance Level
—- —— ————- —————-
ESXi-WilmsenIT WilmsenIT 2-8-2012 3:0… PartnerSupported
- Now whe’re going to add the driver from step 4.Add-EsxSoftwarePackage -ImageProfile “ESXi-WilmsenIT“ -SoftwarePackage “net-bnx2x”The output appears similar to:Name Vendor Last Modified Acceptance Level
————— —— ————- —————-
ESXi-WilmsenIT WilmsenIT today PartnerSupported
- Last step is to export the imageprofile to a ISO file which you can burn. Use the command Export-EsxImageProfile for this:Export-EsxImageProfile -ImageProfile “ESXi-WilmsenIT” -ExportToISO -filepath C:\ESXi-WilmsenIT.iso
Just a quick post how to mount a NFS share using the ESXi CLI.
You all probably know that when it comes to storage protocols, NFS is often used with NAS storage devices. You can mount a NFS share with the vSphere client. But when mouting a NFS share through the vSphere client fails, it’s handy to try it through the CLI. When mounting a NFS share through the CLI you get more info when the mount fails or successes.
So how can you mount a NFS share using the CLI?
Show all active mounts: esxcfg-nas -l
Delete a NFS share: esxcfg-nas -d [datastorename]
Add a NFS share: esxcfg-nas -a –host [nfsserver dns/ip] –share [share name] [datastorename] –readonly
The option –readonly is optional.
Before you mount a NFS share, make sure you know the full path of the share. It’s possible that you have provided a NFS share name like “VMDatastore1″ but that the full export name is /nfs/VMDatastore1. This can be vendor specific.
Share names are case sensitive.