Performance testing is generally not suggested on Virtualized machine. But if we allocate resources properly, create test cases properly and evaluate performance accordingly then it is fairly good to go for virtualized machines. Since virtualization is slow and we may not always have best HW(cost matters) we may need to tune the virtual machines and their operating systems very much.
If you want to build up virtual performance testing setup here are the check lists and steps for Oracle Virtual box, VMware and Oracle VM tuning so that you get maximum of available Hardware and software. Most of the suggestions I've tested practically.
Check List :
1. If using Hardware virtualized machine HWM or HWM with paravirtualized drivers then
A) Does your processor supports HW virtualization of (VT-X feature, most do) and your CPU has physical virtualization enabled.
B) Does your processor supports HW virtualizated I/O (VT-D feature),quite new HW feature(improves I/O tremendously) and your CPU has enabled it.
C) Does your processor supports nested paged table(most do) and your Hyper-visor is using it.
D) Have you installed the necessary add on services like VBOX add on or VMWARE tools.
E) Are you using paravirtualized Ethernet cards like VMXNET on vmware or Paravirtualized network- vitrio-net instead of Intel Pro in case of Oracle virtual box.
F) Have you checked which I/O scheduler is best for you. Most of the hyper-visors like VMWARE ESX-i takes care of disk I/O submission so NOOP gives best performance with less CPU overhead.
G) Make sure you are not over committing memory or processors than available, which is allowed by some virtualization like VMWARE.
2) A) If using pure paravirtualized machine then is your kernel uptodate.
B) If required have you disabled the HW virtualization supported by processor in BIOS.
C) Are you monitoring virtual machine from metal-ware i.e. Dom0 using specialized commands xentop etc in Oracle VM server.
3) If using Inetl i7 then disable hyper threading as hyper threads are actual not core physical Arithmetic and Logical units and virtual box performance slows down due it. Intel i5 may give performance as good as i7
Tune GUEST :
- In order to save resources are you running your Linux in run-level 3 or without x-window.
- remove unnecessary packages
- use virtual disks with fixed storage unless space is issue.
- Have you disabled the services not required.
- Instead of creating many disks under many controllers create few virual disks under few controllers ,preferably on separate physical hard disks. If using Virtual Box you can change mother board type to ACHI
- Have you tuned your file system( like ext4 or removing atime attribute ) and using the one offering best performance and less overheads[ Be aware of its crash and recovery as well ]
- when not required please put Virtual machines GUI in background[ not possible in Oracle Virtual box]
- If using Virtual Box then make sure you are using appropriate disk controller. SATA disk controller has advantage of less overhead while IDEL controller can leverage the DISK caching. Use SCSI only when you are in need to match it to your prod system you need to simulate.
- If you have enabled the disk caching then in abrupt termination of machine there is high chance of DB corruption so be ready with recovery solutions.
- Make sure there be fewer paging and no swapping at all in Guest. Anticipatory I/O scheduler does not go well with swapping. Make sure you are not using this one though this is the case only with older kernel say 2.4.1 Swapping at host level may be better than swapping at GUEST so may be over-committing memory better in few cases.
- If Oracle SGA is large say 4GB and many concurrent sessions say 100 then seriously consider using huge pages but do not pre-page SGA. Make sure whole SGA fits into Huge pages pool else it will rather slow down. Please monitor system for a good duration after setting up. Disable transparent Huge pages RHEL 6.
It is not helpful. Please see post : Configure Huge Pages on Linux
- if using Oracle VM then increase VM metal-ware memory from default 512MB to 1024MB.
- If using Paravirtualized machine and RHEL then change I/O scheduler from CFQ to DEADLINE. You can changed write time-out of deadline scheduler attribute.
- If you installed Linux originally with 1 CPU and adding CPUs later it after then you need to install kernel-smp package also and use apic and smp switch in /etc/grub.conf . Most of the cases single processors or two processors [ or two cores ] out performs the four processors [ or four core ] due minimized context switching.
- Snapshots slows down the performance due required extra maintenance of machine state so make sure this is disabled while performance testing.
- Try to use less colors say 16 colors and minimize the video memory say from 16MB to 8MB.
- use large pages(not the OS one ) in Oracle Virtual box. Increase it from command line using vboxsrvc command. It can not be increased from GUI. Please read manual.
- Use appropriate kernel network parameters like default and maximum send and receive buffer size.
- Decrease the ticker frequncy by adding divider=10 in /etc/grub.conf boot loader for corresponding kernel line. RHEL 6 is tickless so not required for that.
- OEL RHEL compatible kernel offers slightly better processor speed 3% in single task and 1% in multi tasking than OEL-UEK (not sure why - due processor scheduler or any reason) but it offers sligjly less performance than OEL UEK kernel (even though both schedulers are kept same deadline for testing). So you can test which one suits your CPU/I-O
- If using performance testing then use CPU affinity to assign CPU to each node according to capacity so that if respective performance of nodes or cluster has to be checked then CPU's usage should be correct. Avoid CPU over commit as well.
- Do not add many virtual hard disks if it is on same physical box or it is not essential for test as doing so will increase virtualization overhead. If Virtual machine freezes under heavy I/O then consider increasing host memory and guest memory. In such cases you will get warning like 'ATA disk waits exceed 61 secsetc.'
- Few paging can be tolerated but swapping should not be. Increase Linux VM parameter to increase unit of swapped pages in single call which is logarithmic(natual).
- if You disk freezes(too slow I/O) [confirmed by Virtual machine log reporting too much disk time warning for e.g. Virtual box then detach disks,readd them in different controller with changed caching and then add back to original controller.Else machine will continue to freeze even after reboot ]
- Some versions of Oracle has mutex bugs- manly due process scheduling issue which causes significant CPU wait- make sure you are not being hit by same.
--- Read manual of virtual machine provide and OS Tunning
-- Just for manageability : UEFI linux does not go well with Windows 7 with dual boot for want of signing boot-loader for security so when installing Linux make sure you does not boot from UEFI disk.
If you want to build up virtual performance testing setup here are the check lists and steps for Oracle Virtual box, VMware and Oracle VM tuning so that you get maximum of available Hardware and software. Most of the suggestions I've tested practically.
Check List :
1. If using Hardware virtualized machine HWM or HWM with paravirtualized drivers then
A) Does your processor supports HW virtualization of (VT-X feature, most do) and your CPU has physical virtualization enabled.
B) Does your processor supports HW virtualizated I/O (VT-D feature),quite new HW feature(improves I/O tremendously) and your CPU has enabled it.
C) Does your processor supports nested paged table(most do) and your Hyper-visor is using it.
D) Have you installed the necessary add on services like VBOX add on or VMWARE tools.
E) Are you using paravirtualized Ethernet cards like VMXNET on vmware or Paravirtualized network- vitrio-net instead of Intel Pro in case of Oracle virtual box.
F) Have you checked which I/O scheduler is best for you. Most of the hyper-visors like VMWARE ESX-i takes care of disk I/O submission so NOOP gives best performance with less CPU overhead.
G) Make sure you are not over committing memory or processors than available, which is allowed by some virtualization like VMWARE.
2) A) If using pure paravirtualized machine then is your kernel uptodate.
B) If required have you disabled the HW virtualization supported by processor in BIOS.
C) Are you monitoring virtual machine from metal-ware i.e. Dom0 using specialized commands xentop etc in Oracle VM server.
3) If using Inetl i7 then disable hyper threading as hyper threads are actual not core physical Arithmetic and Logical units and virtual box performance slows down due it. Intel i5 may give performance as good as i7
Tune GUEST :
- In order to save resources are you running your Linux in run-level 3 or without x-window.
- remove unnecessary packages
- use virtual disks with fixed storage unless space is issue.
- Have you disabled the services not required.
- Instead of creating many disks under many controllers create few virual disks under few controllers ,preferably on separate physical hard disks. If using Virtual Box you can change mother board type to ACHI
- Have you tuned your file system( like ext4 or removing atime attribute ) and using the one offering best performance and less overheads[ Be aware of its crash and recovery as well ]
- when not required please put Virtual machines GUI in background[ not possible in Oracle Virtual box]
- If using Virtual Box then make sure you are using appropriate disk controller. SATA disk controller has advantage of less overhead while IDEL controller can leverage the DISK caching. Use SCSI only when you are in need to match it to your prod system you need to simulate.
- If you have enabled the disk caching then in abrupt termination of machine there is high chance of DB corruption so be ready with recovery solutions.
- Make sure there be fewer paging and no swapping at all in Guest. Anticipatory I/O scheduler does not go well with swapping. Make sure you are not using this one though this is the case only with older kernel say 2.4.1 Swapping at host level may be better than swapping at GUEST so may be over-committing memory better in few cases.
- If Oracle SGA is large say 4GB and many concurrent sessions say 100 then seriously consider using huge pages but do not pre-page SGA. Make sure whole SGA fits into Huge pages pool else it will rather slow down. Please monitor system for a good duration after setting up. Disable transparent Huge pages RHEL 6.
It is not helpful. Please see post : Configure Huge Pages on Linux
- if using Oracle VM then increase VM metal-ware memory from default 512MB to 1024MB.
- If using Paravirtualized machine and RHEL then change I/O scheduler from CFQ to DEADLINE. You can changed write time-out of deadline scheduler attribute.
- If you installed Linux originally with 1 CPU and adding CPUs later it after then you need to install kernel-smp package also and use apic and smp switch in /etc/grub.conf . Most of the cases single processors or two processors [ or two cores ] out performs the four processors [ or four core ] due minimized context switching.
- Snapshots slows down the performance due required extra maintenance of machine state so make sure this is disabled while performance testing.
- Try to use less colors say 16 colors and minimize the video memory say from 16MB to 8MB.
- use large pages(not the OS one ) in Oracle Virtual box. Increase it from command line using vboxsrvc command. It can not be increased from GUI. Please read manual.
- Use appropriate kernel network parameters like default and maximum send and receive buffer size.
- Decrease the ticker frequncy by adding divider=10 in /etc/grub.conf boot loader for corresponding kernel line. RHEL 6 is tickless so not required for that.
- OEL RHEL compatible kernel offers slightly better processor speed 3% in single task and 1% in multi tasking than OEL-UEK (not sure why - due processor scheduler or any reason) but it offers sligjly less performance than OEL UEK kernel (even though both schedulers are kept same deadline for testing). So you can test which one suits your CPU/I-O
- If using performance testing then use CPU affinity to assign CPU to each node according to capacity so that if respective performance of nodes or cluster has to be checked then CPU's usage should be correct. Avoid CPU over commit as well.
- Do not add many virtual hard disks if it is on same physical box or it is not essential for test as doing so will increase virtualization overhead. If Virtual machine freezes under heavy I/O then consider increasing host memory and guest memory. In such cases you will get warning like 'ATA disk waits exceed 61 secsetc.'
- Few paging can be tolerated but swapping should not be. Increase Linux VM parameter to increase unit of swapped pages in single call which is logarithmic(natual).
- if You disk freezes(too slow I/O) [confirmed by Virtual machine log reporting too much disk time warning for e.g. Virtual box then detach disks,readd them in different controller with changed caching and then add back to original controller.Else machine will continue to freeze even after reboot ]
- Some versions of Oracle has mutex bugs- manly due process scheduling issue which causes significant CPU wait- make sure you are not being hit by same.
--- Read manual of virtual machine provide and OS Tunning
-- Just for manageability : UEFI linux does not go well with Windows 7 with dual boot for want of signing boot-loader for security so when installing Linux make sure you does not boot from UEFI disk.
Posts
