The operation is not allowed in the current state

It didn’t took me long the discover the problem. The other VM was still deleting his snapshots, preventing the other VM powering up.
Though I find the behavior strange. You cannot snapshot a vApp at once and why doesn’t VMware allow me to power on a VM in a vApp when another VM is deleting his snaphosts?

Is it better to buy a server with a 4 core 3 Ghz CPU or with a 6 core 2.2 Ghz CPU?

In other words, what is better for performance? The total amount of Mhz in the host or the amount of CPU cores?

First let me explain my thoughts. A virtual machine with 1 vCPU can run on 1 core at a time. Although it seems that all virtual machines are running at the same time, this isn’t the fact. The CPU scheduler of the VMkernel places a virtual machine on CPU core and when this VM is done doing his thing another virtual machine can use this core. Of course when you have multiple cores in a host, more virtual machines can run at the same time.
When you have a 2 vCPU virtual machine, this virtual machine wants to ‘lock’ 2 fyscial CPU cores at the same time. When a second core ain’t available the virtual machine is co-stopped (%CSTP).
If a core has more 3 Ghz, the calculation being done on that CPU core faster than on a 2 Ghz CPU core. So a other virtual machine can be put on this CPU core for his calculation.
On the other hand, with more CPU cores in your host. More virtual machines can perform CPU calculations at the same time. Ending up in a lower %READY.

While Googling around on the internet I stumbled on this thread in the VMware Community forums. Jon Hemming explaines how Capacity planner rates a system. The first CPU core counts as 100%, the second CPU core is 90% of the first CPU core. The third CPU core is 90% of the the second CPU.

As you can see, with this calculation the two 3 Ghz 4 core CPU has more power than the two 2 Ghz 6 core CPU.
Jon indicated that this calculation isn’t up-to-date for modern hardware and that the are trying to optimization this algorithm.
This post is back-upped by a Youtube movie from Intel. In this movie Intel claims that the bigger the CPU is, the more and flexibler  it can run virtual machines. In this movie you will see 1 small virtual machine combined with 1 large virtual machine will better fit in a large CPU than in 2 smaller CPUs. But 2 virtual machines cannot run at the same time in a CPU? Lake of CPU cores will result in %CSTP and %READY.

So this left me a bit confused. My opinion: it’s better to have more cores than more Mhz. What’s your opinion?

I’m guessing the result will be a combination of both. But what when have the budget for more 2 Ghz CPUs or less 3 Ghz CPUs?

This time the VCP5 exam was the goal. I combined this with the update of my vSphere4 administration and advanced administration course I give for Ictivity Training. This way I practice the new vSphere 5 features and have the course up-to-date.

Although I have given a vSphere5 administration course (with only the features of that where already available in vSphere4) there were quit some features I didn’t have the time to play with except in the VMware Vsphere: What’s new ESXi 5.0 and vCenter Server 5.0. This includes Storage profiles, Auto deploy and more.

As I follow @esloof on Twitter I read on his blog that the VCP5 exam is different than the VCP4 exam. With the previous VCP exam you had to know the maximum configurations out-of-your-head! There were quite some question: “What is the maximum size of a VMFS volume”. This is not the case any more. The exam is more based on global ESXi 5.0 knowledge with I think is a good thing.

The material I used for my preparation:

• VMware Education (by @esloof) VMware vSphere: What’s new ESXi 5.0 and vCenter Server 5.0
• Scott Lowe (@scott_lowe) Mastering VMware vSphere 5
• Duncan Epping (@DuncanYB) and Frank Denneman (@FrankDenneman) vSphere 5 Clustering Technical Deepdive
• And a lot of testing in my home-lab

Gladly all the work payed-off and I passed my exam with a score of 411 where a minimum score of 300 is the passing rate.

As I’m VCAP4-DCD and VCAP4-DCA certified I will go for VCAP5 when it’s available.
After that maybe I will go for VCDX but I’m wondering if this certification brings me more interesting work here in the Netherlands. There aren’t many corporations in the Netherlands that need a VCDX for there environment.
So will we see. For now I’m glad I can call myself VCP5

As with all upgrade, not all upgrade paths are supported. That’s why I thought to make a list with some upgrade considerations.

1. vSphere5 only comes with the ESXi hypervisor architecture. The main distinction is that ESX comes with a Service Console (also called SC or COS from Console Operating System) and ESXi only has a tech support mode (busybox implementation). If you want to perform command-line administration, you can use vCLI or PowerCLI.
2. You can upgrade from ESX 3.5/4.x and ESXi 4.x installations to ESXi 5.0 preserving your VMFS partitions.
3. New installation and boot devices options are:
   • Unified Extensible Firmware Interface (UEFI) like USB
   • Disk larger than 2TB if the system firmware supports it.
4. A minimum of 2098MB or RAM is required.
5. No graphical installer is available because this requires a Service Console.
6. The (text-base) installer can be used for new installations or upgrades
7. New partitions use the GUID partition table (GPT) instead of the MBR. GUID supports partitions size larger than 2TB.
8. New installations create a 4GB scratch partition. Any remaining disk space is formatted as VMFS datastore.
9. Rolling back to a previous version of ESX/ESXi is not supported.
10. When using a kick-start script (ks.cfg) you can press Shift-O when the ESXi installer screen appears to edit the boot options and provide the kick-start script (Example: ks=nfs//192.168.1.10/vSphere-install/esxi5.cfg nameserver=192.168.1.10 ip=192.168.1.1 netmask=255.255.255.0 gateway=192.168.1.254)
11. The default database for vCenter 5 is Microsoft SQL Server 2008 R2 express. This is bundled with the vCenter DVD. Requirements are:
    • Microsoft Windows Installer version 4.5 (MSI 4.5)
    • 4GB RAM
    • 4GB Disk Storage
    • 64 Bits Operating System
12. A in-place upgrade on Windows XP is not supported
13. When doing a in-place upgrade of your vCenter server, your vCenter server can be down for 40 till 50 minutes. During this downtime, DRS will not function. HA will.
14. A in-place upgrade on a 32Bits Operating System is not supported. You have to perform a migration to a 64Bits Operating System.
15. There is also a vCenter appliance available. This appliance is based on SUSE Linux Enterprise Server 11 (SLES11).
16. Some configuration files are not migrated when performing a upgrade:
    • /etc/sysconfig/mouse
    • /etc/sudoers
    • /etc/yp.conf
    • Custom scripts that are added to /etc/rc.d
17. Configuration files that are migrated are:
    • /etc/vmware/esx.conf
    • /etc/ntp.conf, ntp.drift, ntp.keys
    • /etc/krb.*, /etc/krb5.*
    • /etc/hosts, /etc/resolv.conf
    • /etc/pam.d/*
    • /etc/vmware/vmkiscsid/*
18. Configuration files that are partially migrated are:
    • /etc/passwd, /etc/shadow (only root and vpxusers accounts)
19. When performing a new vSphere5 installation the default partition table that is used will be GPT. When performing a in-place upgrade the MBR partition format will be kept.
20. When migration from ESX the Service Console network interface cards (NICs) are converted to VMkernel NICs and the Service Console port group is removed.
21. Rule set files and customized firewall rules are not preserved.
22. You cannot perform a in-place upgrade from ESX4.x to ESXi 5.0 when the ESX4.x was upgraded from ESX3.x.
23. When performing a in-place upgrade from ESX 4.x the /boot partition has to have more than 350Mb of free space. If the hosts that you are upgrading does not have more than 250MB of free space in the /boot partition, use a scripted or interactive upgrade instead.
24. You can preserve your (local) VMFS datastores when upgrading. Afterward you can upgrade your VMFS3 datastore to VMFS5.

When you are a VMware administrator you constantly have to defend your self that it doesn’t matter if a server is virtualized or that it running on physical hardware. But how can you troubleshoot these issues?

In these Performance Troubleshooting series I’m going to explain how you can troubleshoot performance problems in your virtualized environment.
This series will contain the following chapters:

1. CPU
2. Memory
3. Storage
4. Network
5. Virtual Machine

If you think I left something out. Let me know!

Before you dive into the different chapters, first let me explain how esxtop works.

Running esxtop
You can run esxtop in 2 different ways. One directly in the ESXi Console (the so called busybox) or remote through the remote CLI command called resxtop.
In case you want to run esxtop on the ESXi console,  login as root and start esxtop by typing the command esxtop followed by a enter.

esxtop

If you want to use resxtop start resxtop on a OS where the remote CLI commands are installed with the following parameters:

esxtop –server [dns or ip adres of ESXi host] –user [username probably root] –password [password of the user]

The option –user and –password are not required. If you don’t provide them as a parameter, you will be prompted for them.

Alter esxtop view
When esxtop is started (it doesn’t matter if this is remote or local) you can alter you view.

Running esxtop in batch mode
It’s also possible to start esxtop in batch mode. The results are saved in a CSV file. This can be done with the option -b. With the option -d you can specify the delay for the refresh and the option -n specifies the amount of iterations. Example:

esxtop -c ~/myesxtopconf.cfg -b -d 5 -n 10 > ~/esxtop-output.csv

In this example esxtop is started with a custom configruation file called myesxtopconf.cfg, in batch mode, with a delay of 5, and 10 iterations.
With resxtop you have to provide the IP or DNS name of the ESXi host.

The CSV file can be imported in Microsoft Perfmon for example. Note that if you have a large CSV file the import can take a very long time.

Key tool for CPU performance troubleshooting is esxtop or if your running your esxtop remote resxtop.
I my examples I’m using esxtop directly on the vSphere ESXi console but all of these command also work with resxtop.You just have to provide a extra parameter called –server on witch server you want to run the command. Of course you have to provide a username and password to get access to that server.

If we start esxtop on our vSphere ESXi host we will get the following screen:

Click on image to enlarge

First let me explain what we see here.

Click on image to enlarge

The first three lines:

After the first three lines you will see a table with like the following:

Click on image to enlarge

So let me explain what we see there:

This picture (who I have borrowed from a VMworld presentation) explains the overall relationship between the different variables.

Oke, now we now where the different variables stand for and what there relationship is. The next question will be,  which variables do I have to monitor and what are there thresholds?

So that’s what you need to know about monitoring you vSphere ESXi host with esxtop.

The Broadcom vmnic will show up as a vmhba in the configuration screen of your vSphere5 host.

With the Software iSCSI client provided by VMware the IQN name of this adapter is created when you enable the adapter. This IQN name has a naming convention like: iqn.1998-01.com.vmware:hostname:string where hostname is the name of your vSphere host and string is a random value. Now with the Broadcom card the hostname is where the pain is. As the Broadcom card is enabled by default the hostname is detected as localhost (as seen in the screenshot).
Of course this is not what we want. As when we have a 32 vSphere host all showing up in your iSCSI storage as localhost, it’s now easy to see anymore witch iqn belongs to a vSphere host. Of course the string behind the hostname will be unique and you can document the iqn, but it’s easier and thus better manageable if the iqn contains the real hostname of the server who you have configured during installation.

As of now, the only way I figured out to change the hostname in the iqn name of your vmhba is to go to the properties of you vmhba, on the General tab click configure and change the iqn name in the iSCSI Name field.

After doing so you have to to a rescan of the vmhba in order to update the information in your iSCSI target. If you have used the previous iqn name containing the hostname as localhost you have the setup your lun masking in your iSCSI target again.

Note: I have tested this behavior with Broadcom NICs, I don’t know if the behavior is similar on other NICs who have offloading features.

One more day of work and than after the weekend I’m flying to Copenhagen. Off to VMworld 2011. I’m really excited this year. A lot of great new stuff has been released (vSphere5 and all of its features, View5 etc). So I’m expecting a lot of great session and meet-ups with some people in the community. I noticed that there are a lot of people from the Netherlands. We will see at the VMware Benelux party:-)

If you’re interested in contacting me, use Twitter.

For the people how are interested this is my schedule (for the moment).

Monday
Tuesday
Start: 9:00 AM End: 10:00 AM		SUP1018 Auditorium 15	The Biggest VMware Management Launch Ever! – Find out What’s New & What’s Nex…
Start: 10:30 AM End: 11:30 AM		VSP1999 Room C1-M1	esxtop for Advanced Users
Start: 12:00 PM End: 1:00 PM		VSP3864 Room C1-M2	Best Practices for Virtual Networking
Start: 1:30 PM End: 2:30 PM		VSP3868 Room C1-M4	VMware vStorage Best Practices
Wednesday
Start: 10:30 AM End: 11:30 AM		VSP1933 Room C1-M1	Storage I/O Control for Network-Attached Storage Datastores
Start: 12:00 PM End: 1:00 PM		VSP3205 Auditorium 12	Technology Preview: VMware vStorage APIs for VM and Application Granular Data Management
Start: 1:30 PM End: 2:30 PM		BCO3324 Room C1-M2	Site Recovery Manager 5 and VMware vSphere Replication: Simple and Reliable D…
Start: 3:00 PM End: 4:00 PM		VSP1682 Room C1-M1	VMware vSphere Clustering Q&A
Start: 4:30 PM End: 5:30 PM		VSP2122 Room C1-M1	VMware vMotion in VMware vSphere 5.0: Architecture, Performance and Best Prac…
Thursday
Start: 11:00 AM End: 12:00 PM		VSP1708 Room H-M5	VCDX Panel Defense Preparation
Start: 12:30 PM End: 1:30 PM		VSP1700 Room C1-M2	VMware vSphere 5.0 Storage Features
Start: 2:00 PM End: 3:00 PM		EUC2846 Room C1-M4	VMware View Enterprise Architecture Design and Implementation Best Practices

= Conference Event
= Waiting List

Have a real nice VMworld!

First a little overview. My server who is named ESX2 has 1 VMFS3.46 datastore named vmfslun1 with a block size of 4 MB. On this datastore I have 1 test VM called WinXP Mike. During the upgrade this VM is powered on. The LUN is hosted on a QNAP NAS with iSCSI.

As you have probably heard as of vSphere5 the default and only block size you can select is 1MB. This is because of VMFS5 VMware isn’t using MBR anymore as the default partition table but GPT (GUID Partition Table). So when I upgrade my VMFS3 datastore to VMFS5 is MBR changed to GPT? No, as you probably know, the maximum size of a VMFS3 datastore is 2TB-512 Byte  (without extends), the maximum size of a  VMFS5 datastore is 64TB – 512 Bytes (without extends). This is a major improvement. So when does the MBR partition table change to GPT? Simple, when the VFMS5 datastore grows beyond 2 TB.
Be aware that upgrading your VMFS3 to VMFS5 is a one-way process, there is no turning back.The block size will not change, so if you have a VMFS3 datastore with a 4MB block size, after the upgrade you will have a VMFS5 datastore with a 8MB block size.

That said, you probably asking your self: “Does it matter that I have VMFS5 datastores with different block sizes?” This depends if you have a VAAI capable storage. This can influence the performance of your storage. Best is to have all the datastores the same block size. So a best practice is to create your VMFS datastores with a 1MB block size and move your VMs to them.

If recreating your VMFS datastores isn’t a option, you have to upgrade them.

1. Select your ESXi host and click on the configuration tab.
2. Select Storage and click on the LUN who holds a VMFS3 partition.
3. In the Datastore Details section select Upgrade to VMFS5.
   
4. vCenter will check is the datastore is ready to be upgrade. This means that all the host connected to the datastore have to be ESXi 5.
   
5. The upgrade is perfomed by the ESXi host where you are connected to. Wait until the task is Completed in the resend tasks section.
   
6. After a while your VMFS datastore is upgrade to VMFS5. This while my VM kept running. Nice!
   

More info on VMFS5? Look at the following blog post of Frank Denneman and Jason Boche.

What I didn’t wrote about is that you have to upgrade your vSphere client. This is so straight forward that I’m not going to use a lot a screenshot for it. Just download the vSphere 5 client, start the installation, next, next, next and finish:).

Now we can begin to upgrade the first ESXi host. First step is to import the ESXi 5 ISO file into the repository of Update Manager.

1. Start the Update Manger by clicking on the Update Manager icon in the vCenter Home view.
   
2. In the Update Manager Administration view click on the ESXi Images tab.
   
3. Click on Import ESXi Image.
   
4. Click on Browse and browser to your VMware-VMvisor-Installer-5.0.0-xxxxxx.x86_64.iso and click Open. You’re returned to the Select ESXi image screen. Click Next.
   
   
5. Update Manger will try to connect to the Update Manager server and starts uploading the ISO file into the Update Manager repository. After the upload and import is finished click Next.
   
6. Leave the option Create a baseline using the ESXi image selected and provide a name for the baseline, in my case I call this baseline upgrade_to_ESXi5. If you want you can provide a description for the baseline. Click Finish.
   

Now that the ESXi image is imported and we have created a baseline. We can begin to upgrade the first ESXi host.

1.  In vCenter home view, select Host and Clusters.
   
2. Select you ESXi host and select the Update Manger tab.
   
3. As you can see in my screenshot. It looks like my host is Compliant, but the new baseline is not attached. To attach the new baseline click on Attach.
   
4. A new window will open and you’re able to select the new baseline. Select the upgrade_to_ESXi5 baseline and click Attach.
   
5. Your Update Manager screen will refresh and the compliant state will show up as Unknown. Click on Scan.
   
6. In the Confirm Scan screen deselect Patches and Extensions and select Upgrades. Click on Scan.
   
7. The scan will start and after a while your host compliance status will show up as Incompatible. My host shows up as Incompatible because I’ve installed custom NIC drivers. These drivers will be removed when the host is upgraded.
   
8. Click on Remediate to start the upgrade process.
9. In Baseline Groups and Types select Upgrade Baselines. When you selected Upgrade Baselines you’re able to select the baseline you’ve created. Click Next.
   
10. Read the license agreement and place a mark before I accept the terms and license agreement. Click next.
    
11. As I have installed custom NIC drivers and I’m not sure those are compatible with ESXi 5 I place a mark before Remove installed third-party software that is incompatible with the upgrade, and continue with the remediation. Click next.
    
12. In the schedule screen, you can give this upgrade a Task Name and a Task Description. If you want you can start this upgrade at a specific time, but I want to start the upgrade Immediately. Click next.
    
13. Before the upgrade can start, the host has to be in maintenance mode. New with Update Manager 5 is that you can power off virtual machines before remediation. I don’t want that so I leave the default Do Not Change VM Power Sate selected. Click next.
    
14. In the Cluster remediation option I only select Disable High Availabilility admission control if it is enabled for any of the selected clusters. The other option are not necessary for me. Click next.
    
15. You’re provided with a summary of the selected options you’ve made. Review the option and if you’re sure you haven’t made a wrong choice, click finish and your upgrade will start.
    
16. You can keep track of the upgrade in the Resent Task screen. After a while you ESXi host will reboot and the upgrade is finised.
    

As you can see, my home ESXi is now version 5.0.0 build 469512.

There are many new cool features with vSphere5, who are well documenten by VMware in this pdf.

Next blog post will be upgrading the Virtual Machines.