ASMLib installations on RHEL 6 require that the 'kmod-oracleasm','oracleasmlib' and 'oracleasm-support' packages be installed on the system. The 'oracleasmlib' and 'oracleasm-support' packages are available for download from oracle otn site but the kernel driver package 'kmod-oracleasm' is available only from Red Hat "RHEL Server Supplementary (v. 6 64-bit x86_64)" channel on Red Hat Network (RHN). Use the ASMLib kernel driver that corresponds to the version of the Red Hat Enterprise Linux kernel you're running. Since you may not have the Red Hat support you may have to resort to use ASM dev rules instead of using asmlib for ASM use.
When you use ASMLIB Oracle creates new device entries under /dev/oracleasm/disks for each disk
created by command oracleasm createdisk . The key benefits of ASMlib are that it creates consistent device names(See bottom of this post), assigns the correct user/group ownership for files (dbauser/dbagroup, grid user , asmgroup whatever you choose), and assigns the correct permissions (660 i.e. rw). You can achieve same using udev rules.
EXAMPLE: You have added disk /dev/sdb and you want to create asm disk /dev/asm-disk1
step 0: Make SCSI Devices Trusted
Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.
options=-g
step 1: find the SCSI ID (or name) of scsi disk, if your disk device name is /dev/sdb then command is
/sbin/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VB6b224856-6ec55511
[ if it was rhel 5 you would give switch -s in place of -g]
step 2: create udev rules file as below. By default there is no 99-oracle-asmdevices.rules file.
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB6b224856-6ec55511", NAME="asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
Note here owner, group and permission. Since I prefer to separate owner for grid infrastructure and oracle rdbms so I'm not using oinstall and oracle here.
If it was rhel 5 you would specify PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent" in place of PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",
This whole entry will go in one line
When the kernel finds a “sd?1” device, and Bus SCSI with the serial “1ATA_VBOX_HARDDISK_VB6b224856-6ec55511″, it creates a device file ‘/dev/asm-disk1′, owner grid and group asmadmin, etc.
step 3:
/sbin/partprobe /dev/sdb1
step 4:
check you can read data from disk blocked partition /dev/sdb/sbd1
/sbin/udevadm test /block/sdb/sdb1
step 5:
/sbin/udevadm control --reload-rules
step 6:
start udev service
/sbin/start_udev
step 7:
check you can asm disk has been created at OS level
ls -al /dev/asm*
[root@node2 ~]# ls -al /dev/asm*
brw-rw----. 1 grid asmadmin 8, 17 Aug 11 09:01 /dev/asm-disk1
This should show disk with asm names mentioned above in udev rules present
step 8:
In grid infrasrucure install change disk discovery path to /dev/asm-* as grid installer will not show your asm disk defined by udev rules by default
What do you mean by persistent device naming:
For Linux platforms the LUNs presented from a SAN, NAS or SCSI devices are identified with the path /dev/sdX. (/dev/sda, dev/sdb, etc). There is a common issue when disks are removed from the system (due to failures), changing the name.
Example: having /dev/sda, /dev/sdb, /dev/sdc, if disk /dev/sdb fails on the next reboot the system will discover /dev/sda and /dev/sdb, but /dev/sdb had the content originally referenced by /dev/sdc.
This is more a problem when using RAC environments. If the devices are bound to raw, it wont change because there is a static file (/etc/sysconfig/rawdevices or /etc/rc.local), where the bindind is executed. At the application level it could be a problem. For example, Oracle, will reference /dev/raw/raw2 for the OCR or Voting disk, but starting CRS will fail because of the content of the device is not the OCR or the voting disk.
Similar situation could apply to disks used by ASM where after the reboot the disk is not an ASM disk. ASMLIB is not a problem because it reads the content of the header and creates the block device under /dev/oracleasm/disks after finding the correct ASMLIB label.
The solution to avoid this type of situations is to guarantee consistent names for the devices through reboots. Linux 2.6 introduced udev as a mechanism to dynamically manage all type of devices, including disks.
This document presents details about how to configure udev and create persistent names and use these names in a RAC configuration, for OCR, Voting Disks and ASM disks
When you use ASMLIB Oracle creates new device entries under /dev/oracleasm/disks for each disk
created by command oracleasm createdisk . The key benefits of ASMlib are that it creates consistent device names(See bottom of this post), assigns the correct user/group ownership for files (dbauser/dbagroup, grid user , asmgroup whatever you choose), and assigns the correct permissions (660 i.e. rw). You can achieve same using udev rules.
EXAMPLE: You have added disk /dev/sdb and you want to create asm disk /dev/asm-disk1
step 0: Make SCSI Devices Trusted
Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.
options=-g
step 1: find the SCSI ID (or name) of scsi disk, if your disk device name is /dev/sdb then command is
/sbin/scsi_id -g -u -d /dev/sdb
1ATA_VBOX_HARDDISK_VB6b224856-6ec55511
[ if it was rhel 5 you would give switch -s in place of -g]
step 2: create udev rules file as below. By default there is no 99-oracle-asmdevices.rules file.
vi /etc/udev/rules.d/99-oracle-asmdevices.rules
cat /etc/udev/rules.d/99-oracle-asmdevices.rules
KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB6b224856-6ec55511", NAME="asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"
Note here owner, group and permission. Since I prefer to separate owner for grid infrastructure and oracle rdbms so I'm not using oinstall and oracle here.
If it was rhel 5 you would specify PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent" in place of PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent",
This whole entry will go in one line
When the kernel finds a “sd?1” device, and Bus SCSI with the serial “1ATA_VBOX_HARDDISK_VB6b224856-6ec55511″, it creates a device file ‘/dev/asm-disk1′, owner grid and group asmadmin, etc.
step 3:
/sbin/partprobe /dev/sdb1
step 4:
check you can read data from disk blocked partition /dev/sdb/sbd1
/sbin/udevadm test /block/sdb/sdb1
step 5:
/sbin/udevadm control --reload-rules
step 6:
start udev service
/sbin/start_udev
step 7:
check you can asm disk has been created at OS level
ls -al /dev/asm*
[root@node2 ~]# ls -al /dev/asm*
brw-rw----. 1 grid asmadmin 8, 17 Aug 11 09:01 /dev/asm-disk1
This should show disk with asm names mentioned above in udev rules present
step 8:
In grid infrasrucure install change disk discovery path to /dev/asm-* as grid installer will not show your asm disk defined by udev rules by default
What do you mean by persistent device naming:
For Linux platforms the LUNs presented from a SAN, NAS or SCSI devices are identified with the path /dev/sdX. (/dev/sda, dev/sdb, etc). There is a common issue when disks are removed from the system (due to failures), changing the name.
Example: having /dev/sda, /dev/sdb, /dev/sdc, if disk /dev/sdb fails on the next reboot the system will discover /dev/sda and /dev/sdb, but /dev/sdb had the content originally referenced by /dev/sdc.
This is more a problem when using RAC environments. If the devices are bound to raw, it wont change because there is a static file (/etc/sysconfig/rawdevices or /etc/rc.local), where the bindind is executed. At the application level it could be a problem. For example, Oracle, will reference /dev/raw/raw2 for the OCR or Voting disk, but starting CRS will fail because of the content of the device is not the OCR or the voting disk.
Similar situation could apply to disks used by ASM where after the reboot the disk is not an ASM disk. ASMLIB is not a problem because it reads the content of the header and creates the block device under /dev/oracleasm/disks after finding the correct ASMLIB label.
The solution to avoid this type of situations is to guarantee consistent names for the devices through reboots. Linux 2.6 introduced udev as a mechanism to dynamically manage all type of devices, including disks.
This document presents details about how to configure udev and create persistent names and use these names in a RAC configuration, for OCR, Voting Disks and ASM disks
Posts
