Jan
8
2012
The last and first week of the year I normally reserve for some jobs around the house, some time with wife and kid and some studying.
This time the VCP5 exam was the goal. I combined this with the update of my vSphere4 administration and advanced administration course I give for Ictivity Training. This way I practice the new vSphere 5 features and have the course up-to-date.
Although I have given a vSphere5 administration course (with only the features of that where already available in vSphere4) there were quit some features I didn’t have the time to play with except in the VMware Vsphere: What’s new ESXi 5.0 and vCenter Server 5.0. This includes Storage profiles, Auto deploy and more.
As I follow @esloof on Twitter I read on his blog that the VCP5 exam is different than the VCP4 exam. With the previous VCP exam you had to know the maximum configurations out-of-your-head! There were quite some question: “What is the maximum size of a VMFS volume”. This is not the case any more. The exam is more based on global ESXi 5.0 knowledge with I think is a good thing.
The material I used for my preparation:
- VMware Education (by @esloof) VMware vSphere: What’s new ESXi 5.0 and vCenter Server 5.0
- Scott Lowe (@scott_lowe) Mastering VMware vSphere 5
- Duncan Epping (@DuncanYB) and Frank Denneman (@FrankDenneman) vSphere 5 Clustering Technical Deepdive
- And a lot of testing
in my home-lab
Gladly all the work payed-off and I passed my exam with a score of 411 where a minimum score of 300 is the passing rate.
As I’m VCAP4-DCD and VCAP4-DCA certified I will go for VCAP5 when it’s available.
After that maybe I will go for VCDX but I’m wondering if this certification brings me more interesting work here in the Netherlands. There aren’t many corporations in the Netherlands that need a VCDX for there environment.
So will we see. For now I’m glad I can call myself VCP5
Jan
5
2012
While preparing for my VCP5 exam I was reading a lot of material including the upgrade guide.
As with all upgrade, not all upgrade paths are supported. That’s why I thought to make a list with some upgrade considerations.
- vSphere5 only comes with the ESXi hypervisor architecture. The main distinction is that ESX comes with a Service Console (also called SC or COS from Console Operating System) and ESXi only has a tech support mode (busybox implementation). If you want to perform command-line administration, you can use vCLI or PowerCLI.
- You can upgrade from ESX 3.5/4.x and ESXi 4.x installations to ESXi 5.0 preserving your VMFS partitions.
- New installation and boot devices options are:
- Unified Extensible Firmware Interface (UEFI) like USB
- Disk larger than 2TB if the system firmware supports it.
- A minimum of 2098MB or RAM is required.
- No graphical installer is available because this requires a Service Console.
- The (text-base) installer can be used for new installations or upgrades
- New partitions use the GUID partition table (GPT) instead of the MBR. GUID supports partitions size larger than 2TB.
- New installations create a 4GB scratch partition. Any remaining disk space is formatted as VMFS datastore.
- Rolling back to a previous version of ESX/ESXi is not supported.
- When using a kick-start script (ks.cfg) you can press Shift-O when the ESXi installer screen appears to edit the boot options and provide the kick-start script (Example: ks=nfs//192.168.1.10/vSphere-install/esxi5.cfg nameserver=192.168.1.10 ip=192.168.1.1 netmask=255.255.255.0 gateway=192.168.1.254)
- The default database for vCenter 5 is Microsoft SQL Server 2008 R2 express. This is bundled with the vCenter DVD. Requirements are:
- Microsoft Windows Installer version 4.5 (MSI 4.5)
- 4GB RAM
- 4GB Disk Storage
- 64 Bits Operating System
- A in-place upgrade on Windows XP is not supported
- When doing a in-place upgrade of your vCenter server, your vCenter server can be down for 40 till 50 minutes. During this downtime, DRS will not function. HA will.
- A in-place upgrade on a 32Bits Operating System is not supported. You have to perform a migration to a 64Bits Operating System.
- There is also a vCenter appliance available. This appliance is based on SUSE Linux Enterprise Server 11 (SLES11).
- Some configuration files are not migrated when performing a upgrade:
- /etc/sysconfig/mouse
- /etc/sudoers
- /etc/yp.conf
- Custom scripts that are added to /etc/rc.d
- Configuration files that are migrated are:
- /etc/vmware/esx.conf
- /etc/ntp.conf, ntp.drift, ntp.keys
- /etc/krb.*, /etc/krb5.*
- /etc/hosts, /etc/resolv.conf
- /etc/pam.d/*
- /etc/vmware/vmkiscsid/*
- Configuration files that are partially migrated are:
- /etc/passwd, /etc/shadow (only root and vpxusers accounts)
- When performing a new vSphere5 installation the default partition table that is used will be GPT. When performing a in-place upgrade the MBR partition format will be kept.
- When migration from ESX the Service Console network interface cards (NICs) are converted to VMkernel NICs and the Service Console port group is removed.
- Rule set files and customized firewall rules are not preserved.
- You cannot perform a in-place upgrade from ESX4.x to ESXi 5.0 when the ESX4.x was upgraded from ESX3.x.
- When performing a in-place upgrade from ESX 4.x the /boot partition has to have more than 350Mb of free space. If the hosts that you are upgrading does not have more than 250MB of free space in the /boot partition, use a scripted or interactive upgrade instead.
- You can preserve your (local) VMFS datastores when upgrading. Afterward you can upgrade your VMFS3 datastore to VMFS5.
Dec
13
2011
“The server has poor performance because the server is running virtual.”
“Virtual server are always slower than physical servers.”
“My vSphere server has 60% CPU utilization but the performance of my Virtual Server is poor.”
When you are a VMware administrator you constantly have to defend your self that it doesn’t matter if a server is virtualized or that it running on physical hardware. But how can you troubleshoot these issues?
In these Performance Troubleshooting series I’m going to explain how you can troubleshoot performance problems in your virtualized environment.
This series will contain the following chapters:
- CPU
- Memory
- Storage
- Network
- Virtual Machine
If you think I left something out. Let me know!
Before you dive into the different chapters, first let me explain how esxtop works.
Running esxtop
You can run esxtop in 2 different ways. One directly in the ESXi Console (the so called busybox) or remote through the remote CLI command called resxtop.
In case you want to run esxtop on the ESXi console, login as root and start esxtop by typing the command esxtop followed by a enter.
esxtop
If you want to use resxtop start resxtop on a OS where the remote CLI commands are installed with the following parameters:
esxtop –server [dns or ip adres of ESXi host] –user [username probably root] –password [password of the user]
The option –user and –password are not required. If you don’t provide them as a parameter, you will be prompted for them.
Alter esxtop view
When esxtop is started (it doesn’t matter if this is remote or local) you can alter you view.
| Option | Result |
| V (Capital) | The esxtop screen will only view Virtual Machines. If V is pressed again all other worlds are displayed again. |
| f | Alters the field list. This enables you to create a custom view. |
| s | Changes the refresh time of the screen. This is default 5 sec. |
| h | Help screen |
| c | CPU view |
| m | Memory view |
| n | Network view |
| i | Interrupt view |
| d | Disk Adapter view |
| u | Disk Devices view |
| v | Disk VM view |
| p | Power State view |
| # | Limits the number of rows displayed in esxtop |
| w | Write the alter config. If you just press enter after pressing w, the default configuration is altered. If you provide a path and file-name you can use this configuration the next time you start esxtop. You will have to provide the option -c with the path of the configuration file. Example: esxtop -c ~/myesxtopconf.cfg |
Running esxtop in batch mode
It’s also possible to start esxtop in batch mode. The results are saved in a CSV file. This can be done with the option -b. With the option -d you can specify the delay for the refresh and the option -n specifies the amount of iterations. Example:
esxtop -c ~/myesxtopconf.cfg -b -d 5 -n 10 > ~/esxtop-output.csv
In this example esxtop is started with a custom configruation file called myesxtopconf.cfg, in batch mode, with a delay of 5, and 10 iterations.
With resxtop you have to provide the IP or DNS name of the ESXi host.
The CSV file can be imported in Microsoft Perfmon for example. Note that if you have a large CSV file the import can take a very long time.
Dec
13
2011
Even do vSphere is telling you that the overall CPU utilization is no more than 60%, this doesn’t indicated that your VM isn’t running low on CPU resources.
In this part where going to deepdive in troubleshooting CPU performance related issues on your vSphere environment. Although I’m using vSphere 5.0 for my screenshot, most of the options used also go for vSphere 3 and 4.
Key tool for CPU performance troubleshooting is esxtop or if your running your esxtop remote resxtop.
I my examples I’m using esxtop directly on the vSphere ESXi console but all of these command also work with resxtop.You just have to provide a extra parameter called –server on witch server you want to run the command. Of course you have to provide a username and password to get access to that server.
If we start esxtop on our vSphere ESXi host we will get the following screen:
Click on image to enlarge
First let me explain what we see here.
Click on image to enlarge
The first three lines:
| 1:06:05pm | The current time of your ESXi server. Notice that the time is in UTC. |
| up 18 days 3:35 | How long your ESXi server has been up. |
| 307 worlds | a world is a process thats running in your VMkernel |
| 5 VMs | The amount of VMs running on your ESXi host |
| 12 vCPUs | The amount of vCPU provided to VMs |
| CPU load average | The average CPU load per 5, 10 and 15 minutues. If the average load is higher than the amount of CPU cores, your system has not enough CPU recourses. |
| PCPU USED(%) | Real-time amount of CPU usage per CPU core in percentages. As you can see my system is a 8 core system (2 quad core CPUs). AVG: is the average of all pCPU cores. (4,3 + 2,4 + 0,0 + 0,3 +5,1 + 1,7 + 2,7 + 3,3) / 8 = 25 |
| PCPU UTIL(%) | Real-time amount of CPU utilizaton per CPU core in percentage. AVG is the average of all pCPU cores. (4,4 + 7,2 + 100 + 5,2 + 5,9 + 2,3 + 2,4 + 4,4) / 8 = 16 |
After the first three lines you will see a table with like the following:
Click on image to enlarge
So let me explain what we see there:
| ID | The recourse world id. A world is an ESXi VMkernel schedulable entity, similar to a process or thread in other operating systems. | |
| GID | The resource group world id. A group contains more worlds. If you press e in esxtop and enter the number of the GID, this GID will expand itself in multiple world’s for the same group. Every VM consists of minimum 4 worlds:
|
|
| NAME | The name of the world or world recourse pool. | |
| NWLD | The amount of world’s in the world recourse pool. | |
| %USED | The percentage of physical CPU core cycles used by the recourse pool/world. | |
| %USED = #vCPU*100% indicates that the VM occupies all the CPU cycles he can takes. Indicates that the VM is running at 100%. |
||
| %RUN | The percentage of time scheduled. This value can be twice as large as %USED. | |
| %RUN > %USED the pCPU is not running at its rated clock frequency. Probably due Power saving. | ||
| %SYS | The percentage of time spend in the ESXi VMkernel on behave of the recourse pool/world to process interrupts and to perform other system activities. | |
| If higher than 25 the VM is a high IO VM. If you are aware of this,OK. If not check other statistics. | ||
| %WAIT |
|
|
| %VMWAIT | ||
| =%WAIT-%IDLE | ||
| %RDY | The percentage of time the Resource pool/world was ready to run. | |
| >20% indicated that the amount of pCPU cores is to low. | ||
| %IDLE | The percentage of time the Resource pool/world was idle. | |
| %OVRLP | The Percentage of system time that was spent on behalf of some other Resource Pool/World while Resource Pool/World was scheduled. | |
| %CSTP | The percentage of time the Resource pool/world spent in ready, co-deschedule state. | |
| >5% This accours when a VM as more vCPUs and one vCPU has to wait on another vCPU in order to catch up. | ||
| %MLMTD | Percentage of time the ESX VMKernel deliberately did not run the Resource Pool/World because that would violate the Resource Pool/World’s limit setting. | |
| %SWPWT |
This picture (who I have borrowed from a VMworld presentation) explains the overall relationship between the different variables.
Oke, now we now where the different variables stand for and what there relationship is. The next question will be, which variables do I have to monitor and what are there thresholds?
| Variable | Threshold | Resolution |
| %RDY | >10% | If higher than 10% for a long time, add more CPU cores tho your vSphere host |
| %CSTP | >5% | This only occurs in a VM with more than 1 vCPU. Add more pCPU to the host or decrease the amount of vCPUs in the VM |
| %MLMTD | >0% | If higher than 0% the vCPU is throttled because of CPU limits |
| %SYS | >20% | If higher than 20% the VM is like a high I/O VM. Check guest OS for problems |
| %RUN | >%USED | The pCPU is not running at its rated clock frequency. Probably due Power saving. |
So that’s what you need to know about monitoring you vSphere ESXi host with esxtop.
Oct
25
2011
Just a quick post about something I noticed while configuring a new vSphere5 host with a Broadcom iSCSI Adapter. This is not a real iSCSI HBA but a vmnic with TCP/IP offloading. This offloading feature benefits the performance of iSCSI because the vmkernel doesn’t have to do as must work as with a normal vmnic without the offloading feature.
The Broadcom vmnic will show up as a vmhba in the configuration screen of your vSphere5 host.
With the Software iSCSI client provided by VMware the IQN name of this adapter is created when you enable the adapter. This IQN name has a naming convention like: iqn.1998-01.com.vmware:hostname:string where hostname is the name of your vSphere host and string is a random value. Now with the Broadcom card the hostname is where the pain is. As the Broadcom card is enabled by default the hostname is detected as localhost (as seen in the screenshot).
Of course this is not what we want. As when we have a 32 vSphere host all showing up in your iSCSI storage as localhost, it’s now easy to see anymore witch iqn belongs to a vSphere host. Of course the string behind the hostname will be unique and you can document the iqn, but it’s easier and thus better manageable if the iqn contains the real hostname of the server who you have configured during installation.
As of now, the only way I figured out to change the hostname in the iqn name of your vmhba is to go to the properties of you vmhba, on the General tab click configure and change the iqn name in the iSCSI Name field.
After doing so you have to to a rescan of the vmhba in order to update the information in your iSCSI target. If you have used the previous iqn name containing the hostname as localhost you have the setup your lun masking in your iSCSI target again.
Note: I have tested this behavior with Broadcom NICs, I don’t know if the behavior is similar on other NICs who have offloading features.
| 
Comments Off
Comments Off Oct
14
2011
One more day of work and than after the weekend I’m flying to Copenhagen. Off to VMworld 2011. I’m really excited this year. A lot of great new stuff has been released (vSphere5 and all of its features, View5 etc). So I’m expecting a lot of great session and meet-ups with some people in the community. I noticed that there are a lot of people from the Netherlands. We will see at the VMware Benelux party:-)
If you’re interested in contacting me, use Twitter.
For the people how are interested this is my schedule (for the moment).
VMworld Europe 2011
Oct 14, 2011 Personal Schedule
Printed below is your personal schedule.
| Monday | |||
|---|---|---|---|
| Tuesday | |||
|
Start: 9:00 AM
End: 10:00 AM
|
SUP1018
Auditorium 15
|
The Biggest VMware Management Launch Ever! – Find out What’s New & What’s Nex… | |
|
Start: 10:30 AM
End: 11:30 AM
|
VSP1999
Room C1-M1
|
esxtop for Advanced Users | |
|
Start: 12:00 PM
End: 1:00 PM
|
VSP3864
Room C1-M2
|
Best Practices for Virtual Networking | |
|
Start: 1:30 PM
End: 2:30 PM
|
VSP3868
Room C1-M4
|
VMware vStorage Best Practices | |
| Wednesday | |||
|
Start: 10:30 AM
End: 11:30 AM
|
VSP1933
Room C1-M1
|
Storage I/O Control for Network-Attached Storage Datastores | |
|
Start: 12:00 PM
End: 1:00 PM
|
VSP3205
Auditorium 12
|
Technology Preview: VMware vStorage APIs for VM and Application Granular Data Management | |
|
Start: 1:30 PM
End: 2:30 PM
|
BCO3324
Room C1-M2
|
Site Recovery Manager 5 and VMware vSphere Replication: Simple and Reliable D… | |
|
Start: 3:00 PM
End: 4:00 PM
|
VSP1682
Room C1-M1
|
VMware vSphere Clustering Q&A | |
|
Start: 4:30 PM
End: 5:30 PM
|
VSP2122
Room C1-M1
|
VMware vMotion in VMware vSphere 5.0: Architecture, Performance and Best Prac… | |
| Thursday | |||
|
Start: 11:00 AM
End: 12:00 PM
|
VSP1708
Room H-M5
|
VCDX Panel Defense Preparation | |
|
Start: 12:30 PM
End: 1:30 PM
|
VSP1700
Room C1-M2
|
VMware vSphere 5.0 Storage Features | |
|
Start: 2:00 PM
End: 3:00 PM
|
EUC2846
Room C1-M4
|
VMware View Enterprise Architecture Design and Implementation Best Practices | |
= Conference Event
= Waiting List
Have a real nice VMworld!
|
Comments Off
Comments Off Sep
15
2011
In part 3 of the series Upgrading my home vSphere 4 cluster to a vSphere5 cluster we upgraded my ESXi4 hosts to ESXi5 with the help of Update Manger. Now that we’re running ESXi 5 where able to upgrade the VMFS3 datastores to VMFS5.
First a little overview. My server who is named ESX2 has 1 VMFS3.46 datastore named vmfslun1 with a block size of 4 MB. On this datastore I have 1 test VM called WinXP Mike. During the upgrade this VM is powered on. The LUN is hosted on a QNAP NAS with iSCSI.
As you have probably heard as of vSphere5 the default and only block size you can select is 1MB. This is because of VMFS5 VMware isn’t using MBR anymore as the default partition table but GPT (GUID Partition Table). So when I upgrade my VMFS3 datastore to VMFS5 is MBR changed to GPT? No, as you probably know, the maximum size of a VMFS3 datastore is 2TB-512 Byte (without extends), the maximum size of a VMFS5 datastore is 64TB – 512 Bytes (without extends). This is a major improvement. So when does the MBR partition table change to GPT? Simple, when the VFMS5 datastore grows beyond 2 TB.
Be aware that upgrading your VMFS3 to VMFS5 is a one-way process, there is no turning back.The block size will not change, so if you have a VMFS3 datastore with a 4MB block size, after the upgrade you will have a VMFS5 datastore with a 8MB block size.
That said, you probably asking your self: “Does it matter that I have VMFS5 datastores with different block sizes?” This depends if you have a VAAI capable storage. This can influence the performance of your storage. Best is to have all the datastores the same block size. So a best practice is to create your VMFS datastores with a 1MB block size and move your VMs to them.
If recreating your VMFS datastores isn’t a option, you have to upgrade them.
- Select your ESXi host and click on the configuration tab.
- Select Storage and click on the LUN who holds a VMFS3 partition.
- In the Datastore Details section select Upgrade to VMFS5.
- vCenter will check is the datastore is ready to be upgrade. This means that all the host connected to the datastore have to be ESXi 5.
- The upgrade is perfomed by the ESXi host where you are connected to. Wait until the task is Completed in the resend tasks section.
- After a while your VMFS datastore is upgrade to VMFS5. This while my VM kept running. Nice!
More info on VMFS5? Look at the following blog post of Frank Denneman and Jason Boche.
Sep
15
2011
It has been a while when I wrote part 2 but I’ve been really busy with vSphere5 that I didn’t had the time to complete my series. But now part 3 of upgrading my home vSphere 4 cluster to a vSphere 5 cluster. In this part where going to upgrade my 2 ESXi 4.1 server to ESXi 5 with Update Manager.
What I didn’t wrote about is that you have to upgrade your vSphere client. This is so straight forward that I’m not going to use a lot a screenshot for it. Just download the vSphere 5 client, start the installation, next, next, next and finish:).
Now we can begin to upgrade the first ESXi host. First step is to import the ESXi 5 ISO file into the repository of Update Manager.
- Start the Update Manger by clicking on the Update Manager icon in the vCenter Home view.
- In the Update Manager Administration view click on the ESXi Images tab.
- Click on Import ESXi Image.
- Click on Browse and browser to your VMware-VMvisor-Installer-5.0.0-xxxxxx.x86_64.iso and click Open. You’re returned to the Select ESXi image screen. Click Next.
- Update Manger will try to connect to the Update Manager server and starts uploading the ISO file into the Update Manager repository. After the upload and import is finished click Next.
- Leave the option Create a baseline using the ESXi image selected and provide a name for the baseline, in my case I call this baseline upgrade_to_ESXi5. If you want you can provide a description for the baseline. Click Finish.
Now that the ESXi image is imported and we have created a baseline. We can begin to upgrade the first ESXi host.
- In vCenter home view, select Host and Clusters.
- Select you ESXi host and select the Update Manger tab.
- As you can see in my screenshot. It looks like my host is Compliant, but the new baseline is not attached. To attach the new baseline click on Attach.
- A new window will open and you’re able to select the new baseline. Select the upgrade_to_ESXi5 baseline and click Attach.
- Your Update Manager screen will refresh and the compliant state will show up as Unknown. Click on Scan.
- In the Confirm Scan screen deselect Patches and Extensions and select Upgrades. Click on Scan.
- The scan will start and after a while your host compliance status will show up as Incompatible. My host shows up as Incompatible because I’ve installed custom NIC drivers. These drivers will be removed when the host is upgraded.
- Click on Remediate to start the upgrade process.
- In Baseline Groups and Types select Upgrade Baselines. When you selected Upgrade Baselines you’re able to select the baseline you’ve created. Click Next.
- Read the license agreement
and place a mark before I accept the terms and license agreement. Click next.
- As I have installed custom NIC drivers and I’m not sure those are compatible with ESXi 5 I place a mark before Remove installed third-party software that is incompatible with the upgrade, and continue with the remediation. Click next.
- In the schedule screen, you can give this upgrade a Task Name and a Task Description. If you want you can start this upgrade at a specific time, but I want to start the upgrade Immediately. Click next.
- Before the upgrade can start, the host has to be in maintenance mode. New with Update Manager 5 is that you can power off virtual machines before remediation. I don’t want that so I leave the default Do Not Change VM Power Sate selected. Click next.
- In the Cluster remediation option I only select Disable High Availabilility admission control if it is enabled for any of the selected clusters. The other option are not necessary for me. Click next.
- You’re provided with a summary of the selected options you’ve made. Review the option and if you’re sure you haven’t made a wrong choice, click finish and your upgrade will start.
- You can keep track of the upgrade in the Resent Task screen. After a while you ESXi host will reboot and the upgrade is finised.
As you can see, my home ESXi is now version 5.0.0 build 469512.
There are many new cool features with vSphere5, who are well documenten by VMware in this pdf.
Next blog post will be upgrading the Virtual Machines.
|
Comments Off
Comments Off Sep
13
2011
After 5 year of blogging on blog.wilmsenit.nl it’s time to move another website name and URL.
As of now the primary URL of my blog will be www.virtual-hike.com. The content of my ‘old’ blog is still available through the old URL, but you will be redirected to the new URL. I also made some minor modification in the template layout. As you can see, I added the vExpert logo (who I’m proud off) and added a section for sponsors. The first sponsor I on board (thanks Toine for your support:-)).
In the passed I blogged about VMware and Linux, now my main focus will be VMware and virtualization. Hope you find the new layout nice.
|
Comments Off
Comments Off Sep
12
2011
Although I’m not the biggest fan of programming and there fore PowerCLI, with vSphere5 you cannot do without it. So, if you want to create a small PowerCLI script for daily tasks this reference card can come in handy.
Want to know more about PowerCLI? Follow @alanrenouf
|
Comments Off
Comments Off 
