Translate

Search This Blog

get Advisors, ADDM and AWR stored queries


set linesize 200
col BEGIN_INTERVAL_TIME format a70
select * from (select snap_id,begin_interval_time from dba_hist_snapshot order by begin_interval_time desc) where rownum < 3;
---------------------------------------------
Set pages 1000
Set lines 75
Select a.execution_end, b.type, b.impact, d.rank, d.type,
'Message           : '||b.message MESSAGE,
'Command To correct: '||c.command COMMAND,
'Action Message    : '||c.message ACTION_MESSAGE
From dba_advisor_tasks a, dba_advisor_findings b,
Dba_advisor_actions c, dba_advisor_recommendations d
Where a.owner=b.owner and a.task_id=b.task_id
And b.task_id=d.task_id and b.finding_id=d.finding_id
And a.task_id=c.task_id and d.rec_id=c.rec_Id
And a.task_name like 'ADDM%' and a.status='COMPLETED'
and a.description like '%4782%'
Order by b.impact, d.rank;

----------------------------------------------------------- cat

get_addm_report.sql which gets each task from the last snapshot from dba_advisor_tasks


set long  10000000
set pagesize 50000
column get_clob format a80

select dbms_advisor.get_task_report (task_name) as ADDM_report
from dba_advisor_tasks
where task_id = (
        select max(t. task_id)
        from dba_advisor_tasks t, dba_advisor_log l
        where t.task_id = l.task_id
        and t.advisor_name = 'ADDM'
        and l.status = 'COMPLETED');


---------------------------------------------------------------
export ORACLE_SID=$1
export ORAENV_ASK=NO
. oraenv
sqlplus -S / as sysdba <set echo off
set lines 100
set pages 200
set trimspool on
set termout off
set feedback off

column dcol new_value mydate noprint
select to_char(sysdate,'YYMMDD') dcol from dual;

spool /home/oraprd/scripts/dbreport_$1_&mydate..txt
ttitle 'Average Active Sessions in the last week: Instance $1'
column sample_hour format a16
select
   to_char(round(sub1.sample_time, 'HH24'), 'YYYY-MM-DD HH24:MI') as sample_hour
,    round(avg(sub1.on_cpu),1) as cpu_avg
,    round(avg(sub1.waiting),1) as wait_avg
,    round(avg(sub1.active_sessions),1) as act_avg
,    round( (variance(sub1.active_sessions)/avg(sub1.active_sessions)),1) as act_var_mean
from
   ( -- sub1: one row per second, the resolution of SAMPLE_TIME
     select
        sample_id,
        sample_time,
        sum(decode(session_state, 'ON CPU', 1, 0))  as on_cpu,
        sum(decode(session_state, 'WAITING', 1, 0)) as waiting,
        count(*) as active_sessions
     from
        dba_hist_active_sess_history
     where
        sample_time > sysdate - 7
     group by
        sample_id,
        sample_time
   ) sub1
group by
   round(sub1.sample_time, 'HH24')
order by
   round(sub1.sample_time, 'HH24');


ttitle 'Most expensive queries in the last week: Instance $1'
-- gets most expensive queries
-- (by time spent, change "order by" to use another metric)
-- after a specific date
select
   sub.sql_id,
   sub.seconds_used,
   sub.executions,
   sub.gets
from
   ( -- sub to sort before rownum
     select
        sql_id,
        round(sum(elapsed_time_delta)/1000000) as seconds_used,
        sum(executions_delta) as executions,
        sum(buffer_gets_delta) as gets
     from
        dba_hist_snapshot natural join dba_hist_sqlstat
     where
        begin_interval_time > sysdate - 7
     group by
        sql_id
     order by
        2 desc
   ) sub
where
   rownum < 30
;


set long 32768
ttitle 'Text for most expensive SQL in the last week: Instance $1'
select sql_text
from dba_hist_sqltext
where sql_id =
(
select sub.sql_id
from
   ( -- sub to sort before rownum
     select
        sql_id,
        round(sum(elapsed_time_delta)/1000000) as seconds_since_date,
        sum(executions_delta) as execs_since_date,
        sum(buffer_gets_delta) as gets_since_date
     from
        dba_hist_snapshot natural join dba_hist_sqlstat
     where
        begin_interval_time > sysdate - 7
     group by
        sql_id
     order by
        2 desc
   ) sub
where
   rownum = 1
);

spool off;
exit

top query from awr(revised)

Previous query was blogged in http://orababy.blogspot.in/2013/07/simplification-of-use-of-oracle-10g11g.html  and http://orababy.blogspot.in/2013/08/find-top-resource-intensive-queries-and.html

Here is revised top query, this time presented with sqlplus formatting:

set feedback off
set pagesize 50000
set linesize 8000
set trimspool on
set long 65535
set verify off
set verify off
set serveroutput on size 1000000
alter session set "_optimizer_cartesian_enabled"=false;
alter session set optimizer_mode=RULE;
alter session set nls_date_format = 'YYYY/MM/DD HH24:MI:SS';
alter session set nls_timestamp_format = 'YYYY/MM/DD HH24:MI:SS';
Set Heading On
Set Feedback On
PROMPT
PROMPT **Top 50 Sql last 7 Days(s)**
PROMPT
set pages 10000 linesize 500
set trimspool on
COLUMN parsing_schema_name JUSTIFY C FORMAT a15 HEADING 'Parsing|Schema'
COLUMN '%TOT%' JUSTIFY C FORMAT a6 HEADING '%Tot%'
COLUMN 'SEC/EXE' JUSTIFY C FORMAT 99999.999 HEADING 'Sec|Exec'
COLUMN sql_text JUSTIFY C FORMAT a95 word_wrap HEADING 'SQLText'
COLUMN EXECUTIONS JUSTIFY C format 9,999,999,999 HEADING Executions
COLUMN DISK_READS JUSTIFY C format 9,999,999,999 HEADING DiskReads
COLUMN BUFFER_GETS JUSTIFY C format 9,999,999,999 HEADING BufferGets
COLUMN ELAPSED_TIME JUSTIFY C format 9,999,999,999 HEADING ElapsedTime
COLUMN CPU_TIME JUSTIFY C format 9,999,999,999 HEADING CpuTime
COLUMN ROWS_PROCESSED JUSTIFY C format 9,999,999,999 HEADING
RowsProcessed
COLUMN RANK noprint


SELECT
DISTINCT sub.parsing_schema_name,
sub.sql_id,
REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
DBMS_LOB.SUBSTR(DHST.sql_text,4000,1),CHR(10),' '),CHR(9),' '),' ',' '),'FROM',CHR(10)||'FROM'),
'from',CHR(10)||'from'),'AND ',CHR(10)||'AND '),'and ',CHR(10)||'and '),'WHERE ',CHR(10)||'WHERE '),
'where ',CHR(10)||'where ') SQL_TEXT,
sub.EXECUTIONS,ROUND((sub.EXECUTIONS/sub2.EXECUTIONS_TOTAL)*100)||'%' "%TOT%",
sub.DISK_READS,ROUND((sub.DISK_READS/sub2.DISK_READS_TOTAL)*100)||'%' "%TOT%",
sub.BUFFER_GETS,ROUND((sub.BUFFER_GETS/sub2.BUFFER_GETS_TOTAL)*100)||'%' "%TOT%",
sub.ELAPSED_TIME,ROUND((sub.ELAPSED_TIME/sub2.ELAPSED_TIME_TOTAL)*100)||'%' "%TOT%",
sub.CPU_TIME,ROUND((sub.CPU_TIME/sub2.CPU_TIME_TOTAL)*100)||'%' "%TOT%",
ROWS_PROCESSED,
sub.SEC_PER_EXEC "SEC/EXE",
ROUND((sub.EXECUTIONS/sub2.EXECUTIONS_TOTAL)*100)+
ROUND((sub.DISK_READS/sub2.DISK_READS_TOTAL)*100)+
ROUND((sub.BUFFER_GETS/sub2.BUFFER_GETS_TOTAL)*100)+
ROUND((sub.ELAPSED_TIME/sub2.ELAPSED_TIME_TOTAL)*100) RANK
FROM DBA_HIST_SQLTEXT DHST,
(
SELECT DISTINCT
SQL_ID,
PARSING_SCHEMA_NAME,
ROUND(SUM(EXECUTIONS_DELTA)) AS EXECUTIONS,
ROUND(SUM(PARSE_CALLS_DELTA)) AS PARSE_CALLS,
ROUND(SUM(DISK_READS_DELTA)) AS DISK_READS,
ROUND(SUM(BUFFER_GETS_DELTA)) AS BUFFER_GETS,
ROUND(SUM(ROWS_PROCESSED_DELTA)) AS ROWS_PROCESSED,
ROUND(SUM(CPU_TIME_DELTA/1000000)) AS CPU_TIME,
ROUND(SUM(ELAPSED_TIME_DELTA/1000000)) ELAPSED_TIME,
ROUND(SUM(IOWAIT_DELTA)/1000000) AS IOWAIT,
SUM(ELAPSED_TIME_DELTA/1000000)/DECODE(SUM(EXECUTIONS_DELTA),0,1,SUM(EXECUTIONS_DELTA)) SEC_PER_EXEC
FROM
dba_hist_snapshot
NATURAL join
dba_hist_sqlstat DHS
NATURAL join
dba_hist_sql_plan DHSP
WHERE
BEGIN_INTERVAL_TIME >= SYSDATE-7
AND
parsing_schema_name NOT IN ('SYS','SYSTEM')
AND
object_owner NOT IN ('SYS','SYSTEM')
GROUP BY
SQL_ID,PARSING_SCHEMA_NAME
) sub,
(
SELECT DECODE(ROUND(SUM(EXECUTIONS_DELTA)),0,1,ROUND(SUM(EXECUTIONS_DELTA))) AS EXECUTIONS_TOTAL,
DECODE(ROUND(SUM(DISK_READS_DELTA)),0,1,ROUND(SUM(DISK_READS_DELTA))) AS DISK_READS_TOTAL,
DECODE(ROUND(SUM(BUFFER_GETS_DELTA)),0,1,ROUND(SUM(BUFFER_GETS_DELTA))) AS BUFFER_GETS_TOTAL,
DECODE(ROUND(SUM(ELAPSED_TIME_DELTA/1000000)),0,1,ROUND(SUM(ELAPSED_TIME_DELTA/1000000))) AS ELAPSED_TIME_TOTAL,
DECODE(ROUND(SUM(CPU_TIME_DELTA/1000000)),0,1,ROUND(SUM(CPU_TIME_DELTA/1000000))) AS CPU_TIME_TOTAL
FROM
dba_hist_snapshot
NATURAL join
dba_hist_sqlstat DHS
NATURAL join
dba_hist_sql_plan DHSP
WHERE
BEGIN_INTERVAL_TIME >= SYSDATE-7
AND
parsing_schema_name NOT IN ('SYS','SYSTEM')
AND
object_owner NOT IN ('SYS','SYSTEM')
) sub2
WHERE DHST.sql_id = sub.sql_id
AND ROUND((sub.EXECUTIONS/sub2.EXECUTIONS_TOTAL)*100)+
ROUND((sub.DISK_READS/sub2.DISK_READS_TOTAL)*100)+
ROUND((sub.BUFFER_GETS/sub2.BUFFER_GETS_TOTAL)*100)+
ROUND((sub.ELAPSED_TIME/sub2.ELAPSED_TIME_TOTAL)*100) > 4
AND ROWNUM < 51
AND sub.SEC_PER_EXEC >= .001
ORDER BY RANK DESC

direct path read waits

Direct path read waits are associated with direct read operations. This wait event falls under the User I/O wait class. An Oracle direct read operation reads data directly into the session's PGA , bypassing the SGA. The data in PGA is not shared with other sessions.

Direct reads may be performed in synchronous or asynchronous mode depending on the platform , and the value of the DISK_ASYNC_IO parameter

A significant number of direct path read waits is most likely an application issue

Common causes, Diagnosis and Actions:

Till Oracle 9i this wait events occured for SQL statements with functions that require sorts such as ORDER BY, GROUP BY, UNION,DISTINCT and ROLLUP, HASH PARTITIONS that do not fit into SQL Work Area.

From Oracle 10g this wait event has been divided in two wait events 1. direct path read and 2. direct path read temp

Direct path read waits  in 10g onward occurs for read operations by parallel slaves used in parallel query. While direct path read temp waits occur for sort operations, hashes,index creation etc

This wait event has three parameters: FILE#, FIRST BLOCK#, and BLOCK COUNT.

If level 8 or level 12 10046 trace is enabled for session performing direct path read then number of blocks counts is shown by parameter p3 .

If session performing direct path read is traced with strace , truss, tusc or trace then size of block chunk of direct read can be seen

Note: There is a separate directr read wait event for LOB segments: direct path read(lob). This wait event applies for LOBs that are stored as NOCACHE. When LOBs are stored as CACHE reads and writes go through the SGA and show up as db file sequential reads.

Avoid sorting or hashing large tables else Increase PGA_AGGREGATE_TARGET or MEMORY_MAX_TARGET  after querying V$PGASTAT and V$PGA_TARGET_ADVICE or V$MEMORY_TARGET_ADVICE



read by other session wait events

Read by other session or buffer busy waits occur a when another session is reading the block into the  buffer  OR Another session holds the buffer in an  incompatible mode to our request.   This wait event was known as buffer busy wait event before oracle 10.

These waits indicate read/read, read/write, or write/write contention. The Oracle session is waiting to pin a buffer. A buffer must be pinned before it can be read or modified. Only one process can pin a buffer at any one time.

This wait can be intensified by a large block  size as more rows can be contained within
the block  This wait happens when a session wants to  access a database block in the buffer cache
but it cannot as the buffer is "busy  It is also often due to several processes  repeatedly reading the same blocks (eg: if lots of people scan the same index or data  block)

These waits are common in an I/O bound system.  These wait events may indicates presence of  hot blocks even in tuned queries and presence of un-selective or right hand indexes.

Queries in 10g to find the segments whose data blocks have read contention:

1. SELECT p1 "file#", p2 "block#", p3 "class#",sql_id
FROM v$session
WHERE event = 'read by other session' and username='&USER_NAME';

Parameters P1 and P2 represents the file# and block# while parameter p3 represents the wait_class id. In 9i Parameter P3 represented reason code


2. Pass above selected file# and block# to below query

SELECT relative_fno, owner, segment_name, segment_type
FROM dba_extents
WHERE file_id = &FILE
AND &BLOCK BETWEEN block_id
AND block_id + blocks - 1;


This block of contention can belong to data block,segement header or undo block.


The main way to reduce buffer busy waits is to reduce the total I/O on the system by tuning the query  Depending on the block type, the actions will differ

Data Blocks:
-Eliminate HOT blocks from the application.

-Reduce the number of rows per block( by moving table to tablespace with smaller block size or by below techniques)

-Try rebuilding the object with a higher PCTFREE so that you reduce the number of rows per block.

-Increase INITRANS and MAXTRANS and reduce PCTUSED This will make the table less dense .

-Check for repeatedly scanned /unselective indexes.

-Check for 'right- hand-indexes' (indexes that get inserted into at the same point by many processes).

Segemnt Header:
Use Automate segment management that is bit maps or  increase of number of FREELISTs and FREELIST GROUPs

Undo Header:
Increase the number of Rollback Segments


block contention wait events are also recorded in specific view V$WAITSTAT and since V$SESSION has all the wait events data integrated with it from 10g and it also have the row wait information, below query can also be used to find the sql statements.


SELECT
      s.p1 file_id, s.p2 block_id,o.object_name obj,
       o.object_type otype,
       s.SQL_ID,
       w.CLASS,event
FROM v$session s,
     ( SELECT ROWNUM CLASS#, CLASS FROM v$waitstat ) w,
     all_objects o
WHERE
 event IN ('read by other session')
AND
    w.CLASS#(+)=s.p3
   AND o.object_id (+)= s.row_wait_OBJ#
ORDER BY 1;

SELECT SQL_FULLTEXT from V$SQL WHERE sql_id=&amp

SQL_FULLTEXT is CLOB column which displays full query

ASH samples the active sessions every one second and so we can query v$active_session_history also to get buffer busy waits or read by other session.

SELECT
     p1 file_id ,  p2  block_id ,o.object_name obj,
       o.object_type otype,
       ash.SQL_ID,
       w.CLASS
FROM v$active_session_history ash,
     ( SELECT ROWNUM CLASS#, CLASS FROM v$waitstat ) w,
      all_objects o
WHERE event='read by other session'
   AND w.CLASS#(+)=ash.p3
   AND o.object_id (+)= ash.CURRENT_OBJ#
      AND ash.sample_time > SYSDATE - &MIN/(60*24)
ORDER BY 1;

top resource intensive queries and unused indexes from AWR in oracle

Below top query based on awr is based on earlier post
This query can be modified to sort by any parameter and use any filter as commented.
SELECT T1.*, DBA_HIST_SQLTEXT.sql_text,dba_hist_snapshot.BEGIN_INTERVAL_TIME
FROM(
SELECT
sub.module, parsing_schema_name,
ROUND(sub.seconds_since_date/60,2) elapsed_time_delta_mins,
sub.execs_since_date,
sub.gets_since_date,
sub.snap_id,
ROUND(sub.seconds_since_date/DECODE(execs_since_date,0,1,
execs_since_date)/60,2) avg_exec_time,
sub.sql_id
FROM
( -- sub to sort before rownum
SELECT module, parsing_schema_name,
sql_id,
ROUND(SUM(elapsed_time_delta)/1000000) AS seconds_since_date,
SUM(executions_delta) AS execs_since_date,
SUM(buffer_gets_delta) AS gets_since_date,
snap_id
FROM
dba_hist_snapshot NATURAL JOIN dba_hist_sqlstat
WHERE
dba_hist_sqlstat.parsing_schema_name ='&USER_NAME'
AND begin_interval_time BETWEEN sysdate-1 AND sysdate
AND module NOT LIKE '%exp%' AND module NOT LIKE '%imp%'
AND module NOT LIKE '%TOAD%'
GROUP BY
module,sql_id,snap_id,parsing_schema_name
ORDER BY snap_id DESC
) sub
WHERE ROWNUM <=10
)T1 , DBA_HIST_SQLTEXT , dba_hist_snapshot
WHERE T1.sql_id=DBA_HIST_SQLTEXT.sql_id
AND T1.snap_id=dba_hist_snapshot.snap_id
--AND ( LOWER(DBA_HIST_SQLTEXT.sql_text) LIKE '%delete%' --OR
-- LOWER(DBA_HIST_SQLTEXT.sql_text) LIKE '%insert%' OR LOWER(DBA_HIST_SQLTEXT.sql_text) LIKE '%delete%')
AND sql_text NOT LIKE '%DBMS_STATS%'
AND sql_text NOT LIKE  '%parallel(t,2)%'
AND sql_text NOT LIKE '%maxbkt%'
AND sql_text NOT LIKE '%substrb%'
ORDER BY elapsed_time_delta_mins ;--T1.snap_id
 
Same query sql_id can take different execution plan over time. So in order to find out
which plan for the given SQL_ID was in effect at what time
, you can query  plan_hash_value
from DBA_HIST_SQLSTAT


select distinct plan_hash_value, min(begin_interval_time)
first,
 max(end_interval_time) last
from dba_hist_sqlstat natural join dba_hist_snapshot
where sql_id='&SQL_ID'
group by sql_id, plan_hash_value
order by 3;

 

Query to find unused indexes in oracle:

Indexes can be put in monitoring mode and v$db_object_usage view can be queried to know if index has been used or not but the problem with this approach is if any table has been analyzed with cascade=>true clause then corresponding indexes are marked as used.
So we can again use query based on AWR view to know if index has been used.

-- below query is based on DBA_HIST_SQL_PLAN
-- query can be amended to exclude foreign key indexes as they are not mainly for --performance rather they are to lessen restriction of locks

undefine USER_NAME
undefine start_snap_id
undefine stop_snap_id
 
SELECT owner,  table_name ,  index_name,  index_type ,  LAST_ANALYZED
FROM DBA_INDEXES
WHERE index_name IN
  (SELECT index_name  FROM
    (SELECT owner, index_name
    FROM DBA_INDEXES di
    WHERE di.index_type != 'LOB'
    AND owner            ='&&USER_NAME'
    MINUS
    SELECT index_owner owner, index_name
    FROM DBA_CONSTRAINTS dc
    WHERE index_owner ='&&USER_NAME'
    MINUS
    SELECT p.object_owner owner, p.object_name index_name
    FROM DBA_HIST_SNAPSHOT sn,
      DBA_HIST_SQL_PLAN p
    WHERE sn.snap_id BETWEEN  &start_snap_id AND  &stop_snap_id
    AND p.object_type = 'INDEX'

    )
  )
AND owner ='&&USER_NAME'
ORDER BY 1

OWNER TABLE_NAME    INDEX_NAME    INDEX_TYPE    LAST_ANALYZED
SCOTT    T3                       IDXT3                       NORMAL    27-04-2013 18:11:08
SCOTT    ZIGGY_STUFF  ZIGGY_STUFF_CODE_ID_I    NORMAL    06-07-2013 14:23:03
 
Based on last_analyzed you may like to analyze the ununsed index and run the query after workload of few days to see if index is still unused and it needs to be dropped.

Similarly query can be written to find the frequency  of  index usage :
 

SELECT
 p.object_name search_columns,
 ROUND (COUNT(*)/15 ,2 ) COUNT
 FROM
 DBA_HIST_SNAPSHOT sn,
 DBA_HIST_SQL_PLAN p,
 DBA_HIST_SQLSTAT st
 WHERE
 st.sql_id = p.sql_id
 AND
 sn.snap_id = st.snap_id
AND
p.object_type = 'INDEX'
 AND sn.snap_id BETWEEN &start_snap_id AND &stop_snap_id
AND p.object_owner ='&USER_NAME'
GROUP BY
 p.object_name ORDER BY 2 DESC, 1

temporary tablespace usage in oracle - find sessiosn performing sorting

Oracle provides two basic dynamic performance views for online monitoring of temporary tablespace usage. These are v$sort_usage to query online sessiosn using temporary segments and v$sort_segment to query the size of the temporary segements

V$SORT_USAGE has below main colums:

USERNAME – database user name
SESSION_ADDR – address of the session, can be used to identify the session in V$SESSION according to the SADDR column
SQL_ID – the identifier of the SQL that requires the sort or join, can be used to identify the SQL from V$SQL according to the SQL_ID column
EXTENTS – number of extents in the temporary segment being used by this session
BLOCKS – number of blocks in the temporary segment being used by this session

--based on v$sort_usage
--temp segment usage per session

SELECT S.sid || ‘,’ || S.serial# sid_serial, S.username, S.osuser, P.spid, S.module,
P.program, SUM (T.blocks) * TBS.block_size / 1024 / 1024 mb_used, T.tablespace,
COUNT(*) statements
FROM v$sort_usage T, v$session S, dba_tablespaces TBS, v$process P
WHERE T.session_addr = S.saddr
AND S.paddr = P.addr
AND T.tablespace = TBS.tablespace_name
GROUP BY S.sid, S.serial#, S.username, S.osuser, P.spid, S.module,
P.program, TBS.block_size, T.tablespace
ORDER BY sid_serial;

This sid and serial# can be used to find current query that is taking temp segment using view v$sqltext and passing sql_id of the sid and serial#

-- based on v$sort_segment
-- listing of temp segments

SELECT A.tablespace_name tablespace, D.mb_total,
SUM (A.used_blocks * D.block_size) / 1024 / 1024 mb_used,
D.mb_total-SUM (A.used_blocks * D.block_size) / 1024 / 1024 mb_free
FROM v$sort_segment A,
(
SELECT B.name, C.block_size, SUM (C.bytes) / 1024 / 1024 mb_total
FROM v$tablespace B, v$tempfile C
WHERE B.ts#= C.ts#
GROUP BY B.name, C.block_size
) D
WHERE A.tablespace_name = D.name
GROUP by A.tablespace_name, D.mb_total;

 

patch or upgrade an Oracle clustered (RAC) database

Patching or upgrading RAC database is as easy as patching or upgrading non RAC or non Clustered database, only difference is cluster_database parameter needs to be set to false while running the catupgrd.sql script(10g,11g) or catpatch.sql(9i). Other difference is one and only one instance must be opened in upgrade mode(10,g,11g) or migrate mode(9i) when running upgrade script,which is obviously the only option as setting cluster_database=false means only one instance can be mounted or opened.

OUTLINE:

1. set init parameter cluster_database=false
2. start only one instance in migrate or upgrade mode
    startup upgrade
3. run the catupgrd.sql script in same way it is run on non rac, non clustered database
4. after the catupgrd.sql script has been successfully run i.e you have queried dba_registry and it is showing all required database components have been upgraded then
 a) set the init parameter cluster_database=true
 b) start all the clustered/rac instanaces
  


 

put rac database in archivelog mode

Putting Oracle clustered(RAC) database in archivelog is very easy,only step that differs from single instance is set cluster_database=false while running statement alter database archivelog

1. set init parameter cluster_database =false
2. shutdown the database
3. startup mount
4. alter database archivelog;
5. set init parameter cluster_database =true
6. satrtup open
 

disable daily nightly automatic statistics gathering job in oracle 10g

Disable and enable daily nightly automatic statistics gathering job in oracle 10g:

SELECT * FROM DBA_SCHEDULER_JOBS WHERE JOB_NAME = 'GATHER_STATS_JOB';

Disable:

BEGIN
  DBMS_SCHEDULER.disable('GATHER_STATS_JOB');
END;
Enable:

BEGIN
DBMS_SCHEDULER.enable('GATHER_STATS_JOB');
END;
Query how much table data has changed daily:

exec DBMS_STATS.FLUSH_DATABASE_MONITORING_INFO
select * from  DBA_TAB_MODIFICATIONS where table_owner='user1'

 

gather and restore system statistics in oracle

First query system statistics from sys schema:

select * from aux_stats$
or
select  pname, pval1  from sys.aux_stats$

--now before starting work load
execute dbms_stats.gather_system_stats('Start');
-- run the work load
--after work load has run for representative and significant time
execute dbms_stats.gather_system_stats('Stop');

--to delete system statistics
execute dbms_stats.delete_system_stats

--to restore system statistics

--first find the time to which you want system statistics to restore to
select * from DBA_OPTSTAT_OPERATIONS order by start_time

exec dbms_stats.restore_system_stats(as_of_timestamp=>to_timestamp('27-APR-13 02.42.59'))
execute dbms_stats.restore_table_stats ('SCOTT','EMP','25-JUL-07 12.01.20.766591 PM +02:00');

change host name in linux

Host name is defined in file /etc/sysconfig/network  in linux ,so edit this file
1. vi /etc/sysconfig/network

2.  NETWORKING=yes
    HOSTNAME= NewHostname.localdomain

3. save

4. reboot

5. after reboot run the command hostname to check new hotname is in effect

-bash: cd: /media/sf_ : Permission denied mount host shard folder in oracle virtual box

Error:   -bash: cd: /media/sf_vmshare: Permission denied

Created a host folder as shared directory in Oracle Virtual box and specified auto mount option. When non root user tried to access thsi directoty above error -bash: cd: /media/sf_vmshare: Permission denied was returned. Remedy was to create a directory in that non root user and mount the shared hsot directory in that directory.  Ofcourse first the auto mount directory has to be dismounted.

E
[oracle@node1 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              13G  7.1G  5.0G  59% /
tmpfs                 1.2G  223M  907M  20% /dev/shm
/dev/sda1              73M   29M   41M  41% /boot
/dev/sda2            1008M  260M  698M  28% /tmp
vmshare               1.7T  193G  1.5T  12% /media/sf_vmshare
[oracle@node1 ~]$ cd /media/sf_vmshare
-bash: cd: /media/sf_vmshare: Permission denied

REMEDY:

 [oracle@node1 ~]$ mkdir ~/vmshare
[oracle@node1 ~]$ su - root
Password:
[root@node1 ~]# umount vmshare
[root@node1 ~]# mount -t vboxsf   vmshare  /home/oracle/vmshare/
[root@node1 ~]# exit
logout
[oracle@node1 ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              13G  7.1G  5.0G  59% /
tmpfs                 1.2G  223M  907M  20% /dev/shm
/dev/sda1              73M   29M   41M  41% /boot
/dev/sda2            1008M  260M  698M  28% /tmp
vmshare               1.7T  193G  1.5T  12% /home/oracle/vmshare/
[oracle@node1 ~]$ cd /home/oracle/vmshare/
[oracle@node1 vmshare]$ touch abc

ethernet card of same subnet must have same names on all RAC nodes

Error: ethernet card of same subnet must have same names on all RAC nodes

Remedy: #1delete and add ethernet card repeatedly till ethernet card names are consistent on all nodes

Alternate Remedyy #2 is edit /etc/udev/rules.d/70-persistent-net.rules

You can modify entries like NAME="etho" etc to set correct network card name

In my case old file is below and file after updation is next:

cat: /etc/udev/rules.d/70-persistent-net.rules.bak1.: No such file or directory
[root@node2 ~]# cat /etc/udev/rules.d/70-persistent-net.rules.bak.1
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:cf:18:88", ATTR{type}=="1", KERNEL=="eth*", NAME="eth0"
# PCI device 0x8086:0x100e (e1000)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:25:a3:e6", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
# PCI device 0x8086:0x100e (e1000)

SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:21:6f:e4", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

You can note I renamed the ethernet card name from eth0 to eth1

[root@node2 ~]# cat /etc/udev/rules.d/70-persistent-net.rules
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:cf:18:88", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"
# PCI device 0x8086:0x100e (e1000)
SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:25:a3:e6", ATTR{type}=="1", KERNEL=="eth*", NAME="eth1"
# PCI device 0x8086:0x100e (e1000)
#UBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="08:00:27:21:6f:e4", ATTR{type}=="1", KERNEL=="eth*", NAME="eth2"

kmod-oracleasm package for RHEL 6, alternate use dev rules

ASMLib installations on RHEL 6  require that the 'kmod-oracleasm','oracleasmlib' and 'oracleasm-support' packages be installed on the system. The 'oracleasmlib' and 'oracleasm-support' packages are available for download from oracle otn site but the kernel driver package 'kmod-oracleasm' is available only from Red Hat "RHEL Server Supplementary (v. 6 64-bit x86_64)" channel on Red Hat Network (RHN).  Use the ASMLib kernel driver that corresponds to the version of the Red Hat Enterprise Linux kernel you're running. Since you may not have the Red Hat support you may have to resort to use ASM dev rules instead of using asmlib for ASM use.

When you use ASMLIB Oracle creates new device entries under /dev/oracleasm/disks for each disk
created by command oracleasm createdisk . The key benefits of ASMlib are that it creates consistent device names(See bottom of this post), assigns the correct user/group ownership for files (dbauser/dbagroup, grid user , asmgroup whatever you choose), and assigns the correct permissions (660 i.e. rw). You can achieve same using udev rules.

EXAMPLE: You have added disk /dev/sdb and you want to create asm disk /dev/asm-disk1
 
step 0:  Make SCSI Devices Trusted

Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted. Create the file if it doesn't already exist.

options=-g

step 1: find the SCSI ID (or name) of scsi disk, if your disk device name is /dev/sdb then command is
 /sbin/scsi_id -g -u -d /dev/sdb

1ATA_VBOX_HARDDISK_VB6b224856-6ec55511

[ if it was rhel 5 you would give switch -s in place of -g]

step 2: create udev rules file as below. By default there is no 99-oracle-asmdevices.rules file.

vi /etc/udev/rules.d/99-oracle-asmdevices.rules
cat /etc/udev/rules.d/99-oracle-asmdevices.rules

KERNEL=="sd?1", BUS=="scsi", PROGRAM=="/sbin/scsi_id -g -u -d /dev/$parent", RESULT=="1ATA_VBOX_HARDDISK_VB6b224856-6ec55511", NAME="asm-disk1", OWNER="grid", GROUP="asmadmin", MODE="0660"

Note here owner, group and permission. Since I prefer to separate owner for grid infrastructure and oracle rdbms so I'm not using oinstall and oracle here.

If it was rhel 5 you would specify PROGRAM=="/sbin/scsi_id -g -u -s /block/$parent" in place of PROGRAM=="/sbin/scsi_id -g -u -d  /dev/$parent",  

This whole entry will go in one line

When the kernel finds a “sd?1” device, and  Bus SCSI  with the serial “1ATA_VBOX_HARDDISK_VB6b224856-6ec55511″, it creates a device file ‘/dev/asm-disk1′, owner grid and group asmadmin, etc.

step 3:

/sbin/partprobe /dev/sdb1

step 4:

check you can read data from disk blocked partition /dev/sdb/sbd1

/sbin/udevadm test /block/sdb/sdb1

step 5:

/sbin/udevadm control --reload-rules

step 6:

start udev service 

/sbin/start_udev

step 7:

check you can asm disk has been created at OS level

ls -al /dev/asm*

[root@node2 ~]# ls -al /dev/asm*
brw-rw----. 1 grid asmadmin 8, 17 Aug 11 09:01 /dev/asm-disk1
This should show disk with asm names mentioned above in udev rules present

step 8:

In grid infrasrucure install change disk discovery path to /dev/asm-* as grid installer will not show your asm disk defined by udev rules by default

What do you mean by persistent device  naming:

For Linux platforms the LUNs presented from a SAN, NAS or SCSI devices are identified with the path /dev/sdX. (/dev/sda, dev/sdb, etc). There is a common issue when disks are removed from the system (due to failures), changing the name.
Example: having /dev/sda, /dev/sdb, /dev/sdc, if disk /dev/sdb fails on the next reboot the system will discover /dev/sda and /dev/sdb, but /dev/sdb had the content originally referenced by /dev/sdc.

This is more a problem when using RAC environments. If the devices are bound to raw, it wont change because there is a static file (/etc/sysconfig/rawdevices or /etc/rc.local), where the bindind is executed. At the application level it could be a problem. For example, Oracle, will reference /dev/raw/raw2 for the OCR or Voting disk, but starting CRS will fail because of the content of the device is not the OCR or the voting disk.

Similar situation could apply to disks used by ASM where after the reboot the disk is not an ASM disk. ASMLIB is not a problem because it reads the content of the header and creates the block device under /dev/oracleasm/disks after finding the correct ASMLIB label.
The solution to avoid this type of situations is to guarantee consistent names for the devices through reboots. Linux 2.6 introduced udev as a mechanism to dynamically manage all type of devices, including disks.

This document presents details about  how to configure udev and create persistent names and use these names in a RAC configuration, for OCR, Voting Disks and ASM disks

cluster database creation requires default listener configured and running in grid infrastructure home

When you create RAC database in 11g it requires a default listenr running in Grid home.

Remedy for "cluster database creation requires default listener configured and running in grid infrastructure home.." is create a listener with name LISTENER and port 1521 in grid infrastructure home using  NETCA the graphical tool. Simple, just for netca and create listener. NETCA also registers the listener as cluster resource on all RAC nodes.

note: This error ideally should come in a good installatio of grid infrastructure and rdbms as it creates this required default listeners also besides the scan listeners. 

error in invoking target 'irman ioracle' of makefile ins_rdbms.mk

Encountered below error in linking part in installation of 11gR2:
error in invoking target 'irman ioracle' of makefile ins_rdbms.mk

This error comes when required library is missing, disk temp space is low or swap space is low

 I had ensured all oracle rpms were installed prior to starting installation and oracle installer did not give any warning in prerequisites check so I was expecting something unusual. Swap space was filled 95% and it had only 5MB free space. So I increased swap space and selected retried option and it worked.

[oracle@node1 db_1]$ free -m
Swap:          511        458         53
 [root@node1 ~]# dd if=/dev/zero of=/swapfile1 bs=1024 count=524288
524288+0 records in
524288+0 records out
536870912 bytes (537 MB) copied, 2.304 s, 233 MB/s

[root@node1 ~]# mkswap /swapfile1
Setting up swapspace version 1, size = 524284 KiB
no label, UUID=5a8d2159-82d7-4a71-b13e-06dda43e128c
[root@node1 ~]# swapon /swapfile1
[root@node1 ~]# free -m
Swap:         1023        458        565
optionally if added swap space has to be made permanent then entry for /swapfile can be made in /etc/fstab

[root@node1 ~]# vi /etc/fstab
[root@node1 ~]# cat /etc/fstab
 .
.
/swapfile1 swap swap defaults 0 0
.
.
 

INS 35354 the system on which you are attempting to install oracle rac is not part of a valid cluster

Error: the system on which you are attempting to install oracle rac is not part of a valid cluster

[root@node1 ~]# cat /etc/oraInst.loc
inventory_loc=/u01/app/oraInventory
inst_group=oinstall
[root@node1 ~]# cd /u01/app/oraInventory/
[root@node1 oraInventory]# find . -name 'inventory.xml'
./ContentsXML/inventory.xml
[root@node1 oraInventory]# cd ./ContentsXML/
[root@node1 ContentsXML]# cat inventory.xml|grep CRS
[grid@node1 ~]$ cd $ORACLE_HOME
[grid@node1 grid]$ pwd
/u01/11.2.0/grid
[grid@node1 grid]$ cd /u01/app/oraInventory/ContentsXML/
[grid@node1 ContentsXML]$
[grid@node1 ContentsXML]$ cat inventory.xml

You will find entry crs=true is not there in this file

[grid@node1 ContentsXML]$ $ORACLE_HOME/oui/bin/runInstaller -updateNodeList ORACLE_HOME="/u01/11.2.0/grid" CRS=true
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB.   Actual 511 MB    Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[grid@node1 ContentsXML]$ cat inventory.xml

CRS="true"

This entry now exists  
    
Repeat activity on all nodes in cluster
  




installation errors in RAC root.sh CRS-1013:The OCR location in an ASM disk group is inaccessible, clscfg.bin: error while loading shared libraries: libcap.so.1


Error 1:

 clscfg.bin: error while loading shared libraries: libcap.so.1: cannot open shared object file: No such file or directory

Failed to create keys in the OLR, rc = 127, 32512

Remedy
 root@node1 ~]# cd /rhel6.4_media/Packages/
[root@node1 Packages]# ls -ltrh compat-libcap1-1.10-1.x86_64.rpm
-r--r--r--. 1 root root 18K Aug  9 14:19 compat-libcap1-1.10-1.x86_64.rpm
1.  install the required package
[root@node1 Packages]# rpm -ivh compat-libcap1-1.10-1.x86_64.rpm
warning: compat-libcap1-1.10-1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID fd431d51: NOKEY
Preparing...                ########################################### [100%]
   1:compat-libcap1         ########################################### [100%]
2. clean the run of root.sh
$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force

if it is the last node run following
$GRID_HOME/crs/install/rootcrs.pl -verbose -deconfig -force -lastnode

Error 2:

 [client(13005)]CRS-2317:Fatal error: cannot get local GPnP security keys (wallet).
2013-08-10 13:43:10.896
[client(13005)]CRS-2316:Fatal error: cannot initialize GPnP, CLSGPNP_ERR (Generic GPnP error).
2013-08-10 13:43:10.907
[client(13005)]CRS-1013:The OCR location in an ASM disk group is inaccessible. Details in /u01/11.2.0/grid/log/node1/client/clscfg.log.

[root@node2 grid]# cat /u01/11.2.0/grid/log/node2/client/clscfg.log
Oracle Database 11g Clusterware Release 11.2.0.2.0 - Production Copyright 1996, 2010 Oracle. All rights reserved.
2013-08-10 14:08:17.318: [  CLSCFG][326362880]clscfg_main: Configuration type [4]

remedy - that is probably due bug, it was fixed by installing the upgraded version of cluster after cleaning from above script roocrs.pl and removing grid installation files

 

Create local yum repository in rhel 6.4

resolving pckages dependency in packages installation via yum is lot easier than installing packages from plain rpm command. Here are simple steps to create a local yum repository pointing to rhel 6.4 dvd contents copied to file system. Same steps can be applied for creating yum repository pointing to DVD drive.

1. Copy all linux dvd media into a directory say  /rhel6.4_media

2. install package createrepo

cd /rhel6.4_media/ Packages/
rpm -ivh deltarpm*
rpm -ivh python-deltarpm*
rpm -ivh createrepo*.rpm

3. create file  /etc/yum.repos.d/localrepo.repo with below content

[localrepo]
name=Unixmen Repository
baseurl=file:///rhel6.4_media
gpgcheck=0
enabled=1

3. create local repository

createrepo -v  /rhel6.4_media/
yum clean all
yum update
yum repolist

4.  yum repolist

Loaded plugins: product-id, refresh-packagekit, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
localrepo                                                                                                                                        | 2.9 kB     00:00 ...
localrepo/primary_db                                                                                                                             | 3.1 MB     00:00 ...
repo id                                                                     repo name                                                                             status
localrepo                                                                   Unixmen Repository                                                                    3,720
repolist: 3,720


this will show number of packages available under above created repository

yum is avaialable - you can search and install any packge avaialble in dvd media you copied - you do not need to worry about how to resolve dependency.

yum install glibc-devel

add new hard disk and ext4 file system in linux on oracle virtual box

1. create hdd from virtual media manager option

2. oepn settings of virtaul machine and add new hdd in sata controller

3. partition the /dev/sdb from fdisk:
    fdisk /dev/sdb1
4. create file system

 mkfs.ext4 /dev/sdb1

5. mount the new file system into newly created mount directory

  mkdir /u02
 mount /dev/sdb1 /u02

6. add entry for new mount point in /etc/fstab

/dev/sdb1 /u02 ext4 defaults 0 0
 


 

ORA-00020: maximum number of processes (%s) exceeded

If you are not able to log into oracle database because of error ORA-00020: maximum number of processes (%s) exceeded then remedy is open prelim connection as below and issue shutdown command or else try to kill some oracle processes and see if get luckk

sqlplus -prelim / as sysdba
SQL>shutdown immediate
SQL>startup

 

ORA-00845: MEMORY_TARGET not supported on this system - automatic memory management

Oracle error "ORA-00845: MEMORY_TARGET not supported on this system" is encountered in oracle 11g and 12c databases when instance tries to use automatic memory management and tmpfs mount point is less than the value specified in oracle database instance parameter memory_max_target.  Remedy is unmount the tmpfs and mount with increased space for tmps but also recommended is same change in /etc/fstab for permanent effect to take place.

SQL> alter system set memory_max_target=1024m scope=spfile;
System altered.
SQL> startup force
ORA-00845: MEMORY_TARGET not supported on this systemSQL> exit
Disconnected from Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options
[oracle@orcl12c ~]$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              13G   10G  2.2G  83% /
tmpfs                 939M   72K  939M   1% /dev/shm/dev/sda1              73M   29M   41M  41% /boot
/dev/sda2            1008M   43M  915M   5% /tmp
vmshare               1.7T  161G  1.5T  10% /media/sf_vmshare
[oracle@orcl12c ~]$ su -
Password:
[root@orcl12c ~]# umount tmpfs
umount: /dev/shm: device is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
[root@orcl12c ~]# umount -l tmpfs
[root@orcl12c ~]# mount -t tmpfs  tmpfs -o size=10240m /dev/shm
[root@orcl12c ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda5              13G   10G  2.2G  83% /
/dev/sda1              73M   29M   41M  41% /boot
/dev/sda2            1008M   43M  915M   5% /tmp
vmshare               1.7T  161G  1.5T  10% /media/sf_vmshare
tmpfs                  10G     0   10G   0% /dev/shm
[root@orcl12c ~]# cat /etc/fstab
.
.
tmpfs                   /dev/shm                tmpfs   defaults        0 0
.
.

[root@orcl12c ~]# vi /etc/fstab
[root@orcl12c ~]# cat /etc/fstab
.
.
#tmpfs                   /dev/shm                tmpfs   defaults        0 0
tmpfs                   /dev/shm                tmpfs   size=1024m      0 0
.
.
[root@orcl12c ~]# exit
logout
[oracle@orcl12c ~]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.1.0 Production on Sun Aug 4 15:35:40 2013
Copyright (c) 1982, 2013, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup
ORACLE instance started.
Total System Global Area 1068937216 bytes
Fixed Size                  2296576 bytes
Variable Size            1010828544 bytes
Database Buffers           50331648 bytes
Redo Buffers                5480448 bytes
Database mounted.
Database opened.
SQL> show parameter memory_max_target
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_max_target                    big integer 1G
SQL> show parameter sga_max_size
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
sga_max_size                         big integer 1G
SQL> rem sga_max_size is set automatically equal to the memory_max_target
SQL> show parameter memory_target
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
memory_target                        big integer 0
SQL> alter system set memory_target=1024m;
System altered.
SQL>REM now automatic memory management has been configured

shutdown active processes prevent shutdown operation

When shutdown immediate hanged for long I found in user dump a trace file containing messgaes

ksukia: Attempt 1 to re-kill process OS PID=7427.
ksukia: Attempt 2 to re-kill process OS PID=7427..
.
.
.
$ ps -ef grep 7427
...
oracle 7426 11402 0 09:34 pts/1 00:00:00 sqlplus @a.sql
oracle 7427 7426 0 09:34 ? 00:00:00 [oracle] defunct

This process 7427 was defunct (dead) as it appeared defunct. So this could not be killed directly with kill -9, instead its parent process sqlplus was killed and shutdown immediate proceeded

Many guys on interent suggested shutdown abort but that must be avoided.

Oracle 12c installation on Linux (RHEL 6.4)

I first installed Oracle 11gR2 on RHEL 6 on Oracle virtual box then I Installed oracle 12c (12.0.1) successfully on its clone.  Requirement of RPMSs was same as for Oracle 11gR2.

to simplyfy rpm installations if you want to use YUM then see post
http://orababy.blogspot.in/2013/08/create-yum-local-repository-in-rhel-64.html

Oracle 12c  does not come for 32 bit OS any more and Oracle installation on 64 bit requires that certain 32bit packages also needs to be installed on system. So first of all you should be able to distinguish between a 32-bit or 64-bit package. Following rpm command can be used for this purpose.

#rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep
For example, rpm -qa --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" | grep glibc-devel
See the oracle installation documentation that comes with oracle 12c installation disks.
Negative side of 12c is it does not come with Oracle enterprize manager/DB Control. But another good option came that you can specify the 12C grid control OMS server at the time of creating DB from dbca.

While installing only error  I got came at the start of installer it was "Could not retrieve local nodename" fix was to add host name in /etc/host this is expected error in 10g also.

12c installer is more intelligent in terms of space calculation for DB creation but it is not correct for RDBMS software calculation . Installer said enterprize edition would need more than 6GB disk space but it took only 4.9GB space to install.

Another point, I did not create DB at the time of installation at the DBCA options it said it would create listener but the option did not let you specify the listener port number etc.

I was lacking in disk space but I reduced  the size of fast recovery area and I could create custome template based container database as well as one pluggable database in  4G.  xdb port was disabled at end so in order to use apex set EXEC DBMS_XDB.SETHTTPPORT(8080);

DB express port can be found from lsnrctl status command.

Note: there is no Oracle edition of 12c available for 32bit except its OMS agent.