Name: Mike

Email:

Posts by Mike:

My VCP5 exam experience

January 8th, 2012

The last and first week of the year I normally reserve for some jobs around the house, some time with wife and kid and some studying.

This time the VCP5 exam was the goal. I combined this with the update of my vSphere4 administration and advanced administration course I give for Ictivity Training. This way I practice the new vSphere 5 features and have the course up-to-date.

Although I have given a vSphere5 administration course (with only the features of that where already available in vSphere4) there were quit some features I didn’t have the time to play with except in the VMware Vsphere: What’s new ESXi 5.0 and vCenter Server 5.0. This includes Storage profiles, Auto deploy and more.

As I follow @esloof on Twitter I read on his blog that the VCP5 exam is different than the VCP4 exam. With the previous VCP exam you had to know the maximum configurations out-of-your-head! There were quite some question: “What is the maximum size of a VMFS volume”. This is not the case any more. The exam is more based on global ESXi 5.0 knowledge with I think is a good thing.

The material I used for my preparation:

• VMware Education (by @esloof) VMware vSphere: What’s new ESXi 5.0 and vCenter Server 5.0
• Scott Lowe (@scott_lowe) Mastering VMware vSphere 5
• Duncan Epping (@DuncanYB) and Frank Denneman (@FrankDenneman) vSphere 5 Clustering Technical Deepdive
• And a lot of testing in my home-lab

Gladly all the work payed-off and I passed my exam with a score of 411 where a minimum score of 300 is the passing rate.

As I’m VCAP4-DCD and VCAP4-DCA certified I will go for VCAP5 when it’s available.
After that maybe I will go for VCDX but I’m wondering if this certification brings me more interesting work here in the Netherlands. There aren’t many corporations in the Netherlands that need a VCDX for there environment.
So will we see. For now I’m glad I can call myself VCP5

1 Comment "

Some vSphere5 and vCenter5 upgrade considerations

January 5th, 2012

While preparing for my VCP5 exam I was reading a lot of material including the upgrade guide.

As with all upgrade, not all upgrade paths are supported. That’s why I thought to make a list with some upgrade considerations.

1. vSphere5 only comes with the ESXi hypervisor architecture. The main distinction is that ESX comes with a Service Console (also called SC or COS from Console Operating System) and ESXi only has a tech support mode (busybox implementation). If you want to perform command-line administration, you can use vCLI or PowerCLI.
2. You can upgrade from ESX 3.5/4.x and ESXi 4.x installations to ESXi 5.0 preserving your VMFS partitions.
3. New installation and boot devices options are:
   • Unified Extensible Firmware Interface (UEFI) like USB
   • Disk larger than 2TB if the system firmware supports it.
4. A minimum of 2098MB or RAM is required.
5. No graphical installer is available because this requires a Service Console.
6. The (text-base) installer can be used for new installations or upgrades
7. New partitions use the GUID partition table (GPT) instead of the MBR. GUID supports partitions size larger than 2TB.
8. New installations create a 4GB scratch partition. Any remaining disk space is formatted as VMFS datastore.
9. Rolling back to a previous version of ESX/ESXi is not supported.
10. When using a kick-start script (ks.cfg) you can press Shift-O when the ESXi installer screen appears to edit the boot options and provide the kick-start script (Example: ks=nfs//192.168.1.10/vSphere-install/esxi5.cfg nameserver=192.168.1.10 ip=192.168.1.1 netmask=255.255.255.0 gateway=192.168.1.254)
11. The default database for vCenter 5 is Microsoft SQL Server 2008 R2 express. This is bundled with the vCenter DVD. Requirements are:
    • Microsoft Windows Installer version 4.5 (MSI 4.5)
    • 4GB RAM
    • 4GB Disk Storage
    • 64 Bits Operating System
12. A in-place upgrade on Windows XP is not supported
13. When doing a in-place upgrade of your vCenter server, your vCenter server can be down for 40 till 50 minutes. During this downtime, DRS will not function. HA will.
14. A in-place upgrade on a 32Bits Operating System is not supported. You have to perform a migration to a 64Bits Operating System.
15. There is also a vCenter appliance available. This appliance is based on SUSE Linux Enterprise Server 11 (SLES11).
16. Some configuration files are not migrated when performing a upgrade:
    • /etc/sysconfig/mouse
    • /etc/sudoers
    • /etc/yp.conf
    • Custom scripts that are added to /etc/rc.d
17. Configuration files that are migrated are:
    • /etc/vmware/esx.conf
    • /etc/ntp.conf, ntp.drift, ntp.keys
    • /etc/krb.*, /etc/krb5.*
    • /etc/hosts, /etc/resolv.conf
    • /etc/pam.d/*
    • /etc/vmware/vmkiscsid/*
18. Configuration files that are partially migrated are:
    • /etc/passwd, /etc/shadow (only root and vpxusers accounts)
19. When performing a new vSphere5 installation the default partition table that is used will be GPT. When performing a in-place upgrade the MBR partition format will be kept.
20. When migration from ESX the Service Console network interface cards (NICs) are converted to VMkernel NICs and the Service Console port group is removed.
21. Rule set files and customized firewall rules are not preserved.
22. You cannot perform a in-place upgrade from ESX4.x to ESXi 5.0 when the ESX4.x was upgraded from ESX3.x.
23. When performing a in-place upgrade from ESX 4.x the /boot partition has to have more than 350Mb of free space. If the hosts that you are upgrading does not have more than 250MB of free space in the /boot partition, use a scripted or interactive upgrade instead.
24. You can preserve your (local) VMFS datastores when upgrading. Afterward you can upgrade your VMFS3 datastore to VMFS5.

 

2 Comments "

vSphere Performance troubleshooting

December 13th, 2011

“The server has poor performance because the server is running virtual.”
“Virtual server are always slower than physical servers.”
“My vSphere server has 60% CPU utilization but the performance of my Virtual Server is poor.”

When you are a VMware administrator you constantly have to defend your self that it doesn’t matter if a server is virtualized or that it running on physical hardware. But how can you troubleshoot these issues?

In these Performance Troubleshooting series I’m going to explain how you can troubleshoot performance problems in your virtualized environment.
This series will contain the following chapters:

1. CPU
2. Memory
3. Storage
4. Network
5. Virtual Machine

If you think I left something out. Let me know!

Before you dive into the different chapters, first let me explain how esxtop works.

Running esxtop
You can run esxtop in 2 different ways. One directly in the ESXi Console (the so called busybox) or remote through the remote CLI command called resxtop.
In case you want to run esxtop on the ESXi console,  login as root and start esxtop by typing the command esxtop followed by a enter.

esxtop

If you want to use resxtop start resxtop on a OS where the remote CLI commands are installed with the following parameters:

esxtop –server [dns or ip adres of ESXi host] –user [username probably root] –password [password of the user]

The option –user and –password are not required. If you don’t provide them as a parameter, you will be prompted for them.

Alter esxtop view
When esxtop is started (it doesn’t matter if this is remote or local) you can alter you view.

Option	Result
V (Capital)	The esxtop screen will only view Virtual Machines. If V is pressed again all other worlds are displayed again.
f	Alters the field list. This enables you to create a custom view.
s	Changes the refresh time of the screen. This is default 5 sec.
h	Help screen
c	CPU view
m	Memory view
n	Network view
i	Interrupt view
d	Disk Adapter view
u	Disk Devices view
v	Disk VM view
p	Power State view
#	Limits the number of rows displayed in esxtop
w	Write the alter config. If you just press enter after pressing w, the default configuration is altered. If you provide a path and file-name you can use this configuration the next time you start esxtop. You will have to provide the option -c with the path of the configuration file. Example: esxtop -c ~/myesxtopconf.cfg

Running esxtop in batch mode
It’s also possible to start esxtop in batch mode. The results are saved in a CSV file. This can be done with the option -b. With the option -d you can specify the delay for the refresh and the option -n specifies the amount of iterations. Example:

esxtop -c ~/myesxtopconf.cfg -b -d 5 -n 10 > ~/esxtop-output.csv

In this example esxtop is started with a custom configruation file called myesxtopconf.cfg, in batch mode, with a delay of 5, and 10 iterations.
With resxtop you have to provide the IP or DNS name of the ESXi host.

The CSV file can be imported in Microsoft Perfmon for example. Note that if you have a large CSV file the import can take a very long time.

2 Comments "

vSphere Performance troubleshooting Part1: CPU

December 13th, 2011

Even do vSphere is telling you that the overall CPU utilization is no more than 60%, this doesn’t indicated that your VM isn’t running low on CPU resources.
In this part where going to deepdive in troubleshooting CPU performance related issues on your vSphere environment. Although I’m using vSphere 5.0 for my screenshot, most of the options used also go for vSphere 3 and 4.

Key tool for CPU performance troubleshooting is esxtop or if your running your esxtop remote resxtop.
I my examples I’m using esxtop directly on the vSphere ESXi console but all of these command also work with resxtop.You just have to provide a extra parameter called –server on witch server you want to run the command. Of course you have to provide a username and password to get access to that server.

If we start esxtop on our vSphere ESXi host we will get the following screen:



Click on image to enlarge

 

 

 

 

 

First let me explain what we see here.



Click on image to enlarge

 

 

 

The first three lines:

1:06:05pm	The current time of your ESXi server. Notice that the time is in UTC.
up 18 days 3:35	How long your ESXi server has been up.
307 worlds	a world is a process thats running in your VMkernel
5 VMs	The amount of VMs running on your ESXi host
12 vCPUs	The amount of vCPU provided to VMs
CPU load average	The average CPU load per 5, 10 and 15 minutues. If the average load is higher than the amount of CPU cores, your system has not enough CPU recourses.
PCPU USED(%)	Real-time amount of CPU usage per CPU core in percentages. As you can see my system is a 8 core system (2 quad core CPUs). AVG: is the average of all pCPU cores. (4,3 + 2,4 + 0,0 + 0,3 +5,1 + 1,7 + 2,7 + 3,3) / 8 = 25
PCPU UTIL(%)	Real-time amount of CPU utilizaton per CPU core in percentage. AVG is the average of all pCPU cores. (4,4 + 7,2 + 100 + 5,2 + 5,9 + 2,3 + 2,4 + 4,4) / 8 = 16

 

 

 

 

 

 

 

 

 

 

 

 

 

After the first three lines you will see a table with like the following:

 



Click on image to enlarge

 

 

 

So let me explain what we see there:

ID	The recourse world id. A world is an ESXi VMkernel schedulable entity, similar to a process or thread in other operating systems.
GID	The resource group world id. A group contains more worlds. If you press e in esxtop and enter the number of the GID, this GID will expand itself in multiple world’s for the same group. Every VM consists of minimum 4 worlds: vmx: This world is used for vCPU world explained in vmx-vcpu-#. vmast.#: Ths world is used for memory scanning. vmx-mks: This world is used for mouse, keyboard and monitor. vmx-vcpu-#: This world is used for every vCPU of the VM. The amount of vCPU worlds is the same as the amount of vCPU configured for this VM.
NAME	The name of the world or world recourse pool.
NWLD	The amount of world’s in the world recourse pool.
%USED	The percentage of physical CPU core cycles used by the recourse pool/world.
	%USED = #vCPU*100% indicates that the VM occupies all the CPU cycles he can takes. Indicates that the VM is running at 100%.
%RUN	The percentage of time scheduled. This value can be twice as large as %USED.
	%RUN > %USED the pCPU is not running at its rated clock frequency. Probably due Power saving.
%SYS	The percentage of time spend in the ESXi VMkernel on behave of the recourse pool/world to process interrupts and to perform other system activities.
	If higher than 25 the VM is a high IO VM. If you are aware of this,OK. If not check other statistics.
%WAIT	The total percentage of time the Resource pool/world spent in wait state.
%VMWAIT
	=%WAIT-%IDLE
%RDY	The percentage of time the Resource pool/world was ready to run.
	>20% indicated that the amount of pCPU cores is to low.
%IDLE	The percentage of time the Resource pool/world was idle.
%OVRLP	The Percentage of system time that was spent on behalf of some other Resource Pool/World while Resource Pool/World was scheduled.
%CSTP	The  percentage of time the Resource pool/world spent in ready, co-deschedule state.
	>5% This accours when a VM as more vCPUs and one vCPU has to wait on another vCPU in order to catch up.
%MLMTD	Percentage  of time the ESX VMKernel deliberately did not run the Resource Pool/World because that would violate the Resource Pool/World’s limit setting.
%SWPWT

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This picture (who I have borrowed from a VMworld presentation) explains the overall relationship between the different variables.



Oke, now we now where the different variables stand for and what there relationship is. The next question will be,  which variables do I have to monitor and what are there thresholds?

Variable	Threshold	Resolution
%RDY	>10%	If higher than 10% for a long time, add more CPU cores tho your vSphere host
%CSTP	>5%	This only occurs in a VM with more than 1 vCPU. Add more pCPU to the host or decrease the amount of vCPUs in the VM
%MLMTD	>0%	If higher than 0% the vCPU is throttled because of CPU limits
%SYS	>20%	If higher than 20% the VM is like a high I/O VM. Check guest OS for problems
%RUN	>%USED	The pCPU is not running at its rated clock frequency. Probably due Power saving.

 

So that’s what you need to know about monitoring you vSphere ESXi host with esxtop.

No Comments "

vSphere5 and Broadcom iSCSI Adapter IQN name contains localhost

October 25th, 2011

Just a quick post about something I noticed while configuring a new vSphere5 host with a Broadcom iSCSI Adapter. This  is not a real iSCSI HBA but a vmnic with TCP/IP offloading. This offloading feature benefits the performance of iSCSI because the vmkernel doesn’t have to do as must work as with a normal vmnic without the offloading feature.

The Broadcom vmnic will show up as a vmhba in the configuration screen of your vSphere5 host.



With the Software iSCSI client provided by VMware the IQN name of this adapter is created when you enable the adapter. This IQN name has a naming convention like: iqn.1998-01.com.vmware:hostname:string where hostname is the name of your vSphere host and string is a random value. Now with the Broadcom card the hostname is where the pain is. As the Broadcom card is enabled by default the hostname is detected as localhost (as seen in the screenshot).
Of course this is not what we want. As when we have a 32 vSphere host all showing up in your iSCSI storage as localhost, it’s now easy to see anymore witch iqn belongs to a vSphere host. Of course the string behind the hostname will be unique and you can document the iqn, but it’s easier and thus better manageable if the iqn contains the real hostname of the server who you have configured during installation.

As of now, the only way I figured out to change the hostname in the iqn name of your vmhba is to go to the properties of you vmhba, on the General tab click configure and change the iqn name in the iSCSI Name field.



After doing so you have to to a rescan of the vmhba in order to update the information in your iSCSI target. If you have used the previous iqn name containing the hostname as localhost you have the setup your lun masking in your iSCSI target again.

Note: I have tested this behavior with Broadcom NICs, I don’t know if the behavior is similar on other NICs who have offloading features.

Comments Off

VMworld Copenhagen

October 14th, 2011



One more day of work and than after the weekend I’m flying to Copenhagen. Off to VMworld 2011. I’m really excited this year. A lot of great new stuff has been released (vSphere5 and all of its features, View5 etc). So I’m expecting a lot of great session and meet-ups with some people in the community. I noticed that there are a lot of people from the Netherlands. We will see at the VMware Benelux party:-)

If you’re interested in contacting me, use Twitter.

For the people how are interested this is my schedule (for the moment).

VMworld Europe 2011

Oct 14, 2011 Personal Schedule

Printed below is your personal schedule.

Monday
Tuesday
Start: 9:00 AM End: 10:00 AM		SUP1018 Auditorium 15	The Biggest VMware Management Launch Ever! – Find out What’s New & What’s Nex…
Start: 10:30 AM End: 11:30 AM		VSP1999 Room C1-M1	esxtop for Advanced Users
Start: 12:00 PM End: 1:00 PM		VSP3864 Room C1-M2	Best Practices for Virtual Networking
Start: 1:30 PM End: 2:30 PM		VSP3868 Room C1-M4	VMware vStorage Best Practices
Wednesday
Start: 10:30 AM End: 11:30 AM		VSP1933 Room C1-M1	Storage I/O Control for Network-Attached Storage Datastores
Start: 12:00 PM End: 1:00 PM		VSP3205 Auditorium 12	Technology Preview: VMware vStorage APIs for VM and Application Granular Data Management
Start: 1:30 PM End: 2:30 PM		BCO3324 Room C1-M2	Site Recovery Manager 5 and VMware vSphere Replication: Simple and Reliable D…
Start: 3:00 PM End: 4:00 PM		VSP1682 Room C1-M1	VMware vSphere Clustering Q&A
Start: 4:30 PM End: 5:30 PM		VSP2122 Room C1-M1	VMware vMotion in VMware vSphere 5.0: Architecture, Performance and Best Prac…
Thursday
Start: 11:00 AM End: 12:00 PM		VSP1708 Room H-M5	VCDX Panel Defense Preparation
Start: 12:30 PM End: 1:30 PM		VSP1700 Room C1-M2	VMware vSphere 5.0 Storage Features
Start: 2:00 PM End: 3:00 PM		EUC2846 Room C1-M4	VMware View Enterprise Architecture Design and Implementation Best Practices

= Conference Event
= Waiting List

 

Have a real nice VMworld!

Comments Off

Upgrading my home vSphere 4 cluster to a vSphere5 cluster (Part 4 upgrading VMFS)

September 15th, 2011

In part 3 of the series Upgrading my home vSphere 4 cluster to a vSphere5 cluster we upgraded my ESXi4 hosts to ESXi5 with the help of Update Manger. Now that we’re running ESXi 5 where able to upgrade the VMFS3 datastores to VMFS5.

First a little overview. My server who is named ESX2 has 1 VMFS3.46 datastore named vmfslun1 with a block size of 4 MB. On this datastore I have 1 test VM called WinXP Mike. During the upgrade this VM is powered on. The LUN is hosted on a QNAP NAS with iSCSI.



As you have probably heard as of vSphere5 the default and only block size you can select is 1MB. This is because of VMFS5 VMware isn’t using MBR anymore as the default partition table but GPT (GUID Partition Table). So when I upgrade my VMFS3 datastore to VMFS5 is MBR changed to GPT? No, as you probably know, the maximum size of a VMFS3 datastore is 2TB-512 Byte  (without extends), the maximum size of a  VMFS5 datastore is 64TB – 512 Bytes (without extends). This is a major improvement. So when does the MBR partition table change to GPT? Simple, when the VFMS5 datastore grows beyond 2 TB.
Be aware that upgrading your VMFS3 to VMFS5 is a one-way process, there is no turning back.The block size will not change, so if you have a VMFS3 datastore with a 4MB block size, after the upgrade you will have a VMFS5 datastore with a 8MB block size.

That said, you probably asking your self: “Does it matter that I have VMFS5 datastores with different block sizes?” This depends if you have a VAAI capable storage. This can influence the performance of your storage. Best is to have all the datastores the same block size. So a best practice is to create your VMFS datastores with a 1MB block size and move your VMs to them.

If recreating your VMFS datastores isn’t a option, you have to upgrade them.

1. Select your ESXi host and click on the configuration tab.
2. Select Storage and click on the LUN who holds a VMFS3 partition.
3. In the Datastore Details section select Upgrade to VMFS5.
   
4. vCenter will check is the datastore is ready to be upgrade. This means that all the host connected to the datastore have to be ESXi 5.
   
5. The upgrade is perfomed by the ESXi host where you are connected to. Wait until the task is Completed in the resend tasks section.
   
6. After a while your VMFS datastore is upgrade to VMFS5. This while my VM kept running. Nice!
   

More info on VMFS5? Look at the following blog post of Frank Denneman and Jason Boche.

2 Comments "

Upgrading my home vSphere 4 cluster to a vSphere5 cluster (Part 3)

September 15th, 2011

It has been a while when I wrote part 2 but I’ve been really busy with vSphere5 that I didn’t had the time to complete my series. But now part 3 of upgrading my home vSphere 4 cluster to a vSphere 5 cluster. In this part where going to upgrade my 2 ESXi 4.1 server to ESXi 5 with Update Manager.

What I didn’t wrote about is that you have to upgrade your vSphere client. This is so straight forward that I’m not going to use a lot a screenshot for it. Just download the vSphere 5 client, start the installation, next, next, next and finish:).

Now we can begin to upgrade the first ESXi host. First step is to import the ESXi 5 ISO file into the repository of Update Manager.

1. Start the Update Manger by clicking on the Update Manager icon in the vCenter Home view.
   
2. In the Update Manager Administration view click on the ESXi Images tab.
   
3. Click on Import ESXi Image.
   
4. Click on Browse and browser to your VMware-VMvisor-Installer-5.0.0-xxxxxx.x86_64.iso and click Open. You’re returned to the Select ESXi image screen. Click Next.
   
   
5. Update Manger will try to connect to the Update Manager server and starts uploading the ISO file into the Update Manager repository. After the upload and import is finished click Next.
   
6. Leave the option Create a baseline using the ESXi image selected and provide a name for the baseline, in my case I call this baseline upgrade_to_ESXi5. If you want you can provide a description for the baseline. Click Finish.
   

Now that the ESXi image is imported and we have created a baseline. We can begin to upgrade the first ESXi host.

1.  In vCenter home view, select Host and Clusters.
   
2. Select you ESXi host and select the Update Manger tab.
   
3. As you can see in my screenshot. It looks like my host is Compliant, but the new baseline is not attached. To attach the new baseline click on Attach.
   
4. A new window will open and you’re able to select the new baseline. Select the upgrade_to_ESXi5 baseline and click Attach.
   
5. Your Update Manager screen will refresh and the compliant state will show up as Unknown. Click on Scan.
   
6. In the Confirm Scan screen deselect Patches and Extensions and select Upgrades. Click on Scan.
   
7. The scan will start and after a while your host compliance status will show up as Incompatible. My host shows up as Incompatible because I’ve installed custom NIC drivers. These drivers will be removed when the host is upgraded.
   
8. Click on Remediate to start the upgrade process.
9. In Baseline Groups and Types select Upgrade Baselines. When you selected Upgrade Baselines you’re able to select the baseline you’ve created. Click Next.
   
10. Read the license agreement and place a mark before I accept the terms and license agreement. Click next.
    
11. As I have installed custom NIC drivers and I’m not sure those are compatible with ESXi 5 I place a mark before Remove installed third-party software that is incompatible with the upgrade, and continue with the remediation. Click next.
    
12. In the schedule screen, you can give this upgrade a Task Name and a Task Description. If you want you can start this upgrade at a specific time, but I want to start the upgrade Immediately. Click next.
    
13. Before the upgrade can start, the host has to be in maintenance mode. New with Update Manager 5 is that you can power off virtual machines before remediation. I don’t want that so I leave the default Do Not Change VM Power Sate selected. Click next.
    
14. In the Cluster remediation option I only select Disable High Availabilility admission control if it is enabled for any of the selected clusters. The other option are not necessary for me. Click next.
    
15. You’re provided with a summary of the selected options you’ve made. Review the option and if you’re sure you haven’t made a wrong choice, click finish and your upgrade will start.
    
16. You can keep track of the upgrade in the Resent Task screen. After a while you ESXi host will reboot and the upgrade is finised.
    

As you can see, my home ESXi is now version 5.0.0 build 469512.

There are many new cool features with vSphere5, who are well documenten by VMware in this pdf.

Next blog post will be upgrading the Virtual Machines.

Comments Off

New website

September 13th, 2011

After 5 year of blogging on blog.wilmsenit.nl it’s time to move another website name and URL.

As of now the primary URL of my blog will be www.virtual-hike.com. The content of my ‘old’ blog is still available through the old URL, but you will be redirected to the new URL. I also made some minor modification in the template layout. As you can see, I added the vExpert logo (who I’m proud off) and added a section for sponsors. The first sponsor I on board (thanks Toine for your support:-)).

In the passed I blogged about VMware and Linux, now my main focus will be VMware and virtualization. Hope you find the new layout nice.

Comments Off

Free PowerCLI 5.0 reference card

September 12th, 2011

Although I’m not the biggest fan of programming and there fore PowerCLI, with vSphere5 you cannot do without it. So, if you want to create a small PowerCLI script for daily tasks this reference card  can come in handy.

Want to know more about PowerCLI? Follow @alanrenouf

Comments Off