Wednesday, June 7, 2017

10 Steps to Analyze AWR Report in Oracle



As you have Generated AWR Report in Oracle, Next task is to Analyze AWR Report in Oracle. By Reading AWR Report  you can easily solve issues like Slow database, high wait events, Slow query and many more. Though It's a lengthy report but Analyzing or Reading relevant part of AWR Report can help to troubleshoot issues in easy and fast manner.

AWR stands for Automatically workload repository, Though there could be many types of database performance issues, but when whole database is slow, then there are two possibilities.

1. Issue with Database Machine.OS Watcher is the best tool to start.
2. If Database performance issue, Then AWR Report is the place to look at
.

In case if a particular query is not performing well, i would suggest to look at execution plan of the query, stats of underlying table etc. In this case AWR won't help much.

Recommendations before getting an AWR Report.

1. Collect Multiple AWR Reports: It's always good to have two AWR Reports, one for good time (when database was performing well), second when performance is poor. This way Remote DBA can easily compare good and bad report to find out the culprit.

2. Stick to Particular Time: "Database is performing slow" will not help anymore to resolve performace issues. We have to have a specific time like Database was slow yesterday at 1 Pm and continue till 4Pm. Here, DBA will get a report for these three hours. 

3. Split Large AWR Report into Smaller Reports: Instead of having one report for long time like one report for 4hrs. it's is better to have four reports each for one hour. This will help to isolate the problem.

In case of RAC env. generate one report for each instance. Once, you have generated AWR report. Now, it's time of analyze the report. Since, AWR report is a huge report and area to look into AWR is also depends on problem to problem. Here, I am list most common area for a DBA to look into which will give a clear picture of the issue. 

Steps to Analyze AWR Report

1. Database Details:

After getting an AWR Report This is first and Top part of the report. In this part cross check for database and instance and and database version with the Database having performance issue.This report also show RAC=YES if it's an RAC database.

 

2. Host Configuration:

This will give you name, platform CUP, socket and RAM etc. Important thing to notice is number of cores into the system. In this example there are 12 CUP's in Cores



3. Snap Shot Detail:

This are the detail about snap shot taken, Snap start time and end time. Difference between them is as "Elapsed". Here is a new term "DB Time"

DB Time= session time spent in database.

DB Time= CPU Time + Non IDLE wait time.

You can find, DB time is very large as compared to Elapse time, which is not a concern. Check if you have taken a report for the time having performance problem. If yes fine, other wise take a report for performance problem time. 

Next is Cache Sizes, which is just detail about SGA components.

4. Load Profile:

Here are few important stats for a DBA to look into. Fist is "DB CPU(s)" per second. Before that let's understand how DB CUP's work. Suppose you have 12 cores into the system. So, per wall clock second you have 12 seconds to work on CPU. 


 So, if "DB CPU(s)" per second in this report > cores in (Host Configuration (#2)).

means env is CPU bound and either need more CPU's or need to further check is this happening all the time or just for a fraction of time. As per my experience there are very few cases, when system is CPU bound.

In this case, machine has 12 cores and DB CPU(s) per second is 6.8. So, this is not a CPU bound case

Next stat to look at are Parses and Hard parses. If the ratio of hard parse to parse is high, this means Database is performing more hard parse. So, needs to look at parameters like cursor_sharing and application level for bind variables etc.

In the above we have no issue with parses.
Parse issues usually occur as a result of:
           Bad Bind variables usage
           Insufficient memory
           Will also be co-indicated by low percentage of memory for multiple SQL execution.
5. Instance Efficiency Percentages:

In these statistics, you have to look at "% Non-Parse CPU". If this value is near 100% means most of the CPU resources are used into operations other than parsing, which is good for database health.


Note: DB Tuning must never be driven by the Hit ratios. These only provide additional information for help to understand how the instance is operating.

Meanings of particular Hit Ratios:

a.       Buffer Nowait%: Shows the % of times when data buffers were accessed directly without any wait time.

b.       Buffer Hit Ratio : Measures how many times a  required block was found in memory rather than having to execute an expensive read operation on disk to get the block.

c.        Library Hit%: Shows the % of times when SQL statements and PL/SQL packages were found in the shared pool.

d.       Execute to Parse %: Shows how often parsed SQL statements are reused without re-parsing.

e.        Parse CPU to Parse Elapsed %: Gives the ratio of CPU time spent to parse SQL statements.

f.        Redo NoWait %: Shows whether the redo log buffer has sufficient size.

g.        In-memory Sort %: Shows the percentage of times when sorts are performed in memory instead of using temporary tablespaces.

h.       Soft Parse % : Shows how often sessions issued a SQL statements that is already in the shared pool and how it can use an existing version of the statement.

i.         Latch Hit %: Shows how often latches were acquired without having to wait.

j.         % Non-Parse CPU : Shows the percentage of how much CPU resources were spent on the actual SQL execution.

In these statistics, you have to look at

Soft Parse % ~ 100% indicates that the SQL statements are actively re-used.

"% Non-Parse CPU". ~ 100% means most of the CPU resources are used into operations other than parsing, which is good for database health.

Parse CPU to parse Elapsed % : It is very low, it reveals that oracle waits for some resources during parsing of SQL statements. To be investigated further.

Rule of thumb: Always minimize the number of Hard parses. This reduction yields the benefits of minimizing CPU overhead spent performing costly parse work.

6. Top 10 Foreground Events by Total Wait Time:

This is another most important stats to consider while looking at AWR Report for any database performance related issue.  This report iscritical because it shows those database events that might constitute the bottleneck for the system. This has a list of top 10 foreground events arranged by Total Wait Time.


Here, first of all check for wait class if wait class is  User I/O , System I/O,  Others etc this could be fine but if wait class has value "Concurrency" then there could be some serious problem. Next to look at is Total Wait Time (sec) which show how many times DB was waiting in this class and then Wait Avg (ms). If Total Wait Time(sec) are high but Wait Avg(ms) is low then you can ignore this. If both are high or Wait Avg(ms) is high then this has to further investigate.

In the above screen shot, most of the resource are taken by Log file sync = 91.9% DB time and the wait class is “Commit”

In this report the event is "log file sync" which has high Waits, huge % DB time and large values in Total Wait Time (sec)  and Wait  Avg(ms) with wait class as commit. So, here you have to investigate further. 

Note : Use highest Total Wait Time(sec) to guide investigation.

Next are Wait Classes by Total Wait Time, Host CPU, Instance CPU, IO Profile, Memory Statistics, Cache Sizes are self-explanatory.
Point to remember while checking these results :
Ø    Watch for number of CPUs
Ø     Pay attention to changes in Memory size
Ø    An idle CPU can be a bad thing
Ø    Always look at Wait IO verses CPU usage
Ø    If the system is IO bound CPU will be idle!

7. Shared Pool Statistics:


In general, Memory usage % statistics should be ~70% after the DB has been running a long time. If its quite low, memory is being wasted. Here in our report we have Memory Usage % ~50% which is good.

In case if we have this to ~90% or above as shared pool consumed. This could indicate that the system experiences some overhead while aging out old shared memory structures like cursors, PL/SQL programs, and so on. This will place additional overhead on the CPU to perform reparsing aging-out. The size of the shared pool should be increased appropriately to eliminate such overhead.


The % SQL with executions >1 statistics indicate how many SQL statements are executed more than one time. This measures how well the applications are tuned and how well they make use of Bind variables.


8. Time Model Statistics:

This is a detailed explanations of system resource consumptions. Stats are order by Time (s) and % of DB Time.
 
A noticeable result Sum of all  % of DB time is > 100%. why is this ?

Because this is cumulative time i.e. In this case SQL execute elapsed time is taking 89% of DB time, which includes it sub parts like parse time elapsed, hard parse elapsed time etc. So, if you find Hard parse time elapsed is taking more %. So investigate further so on and so forth.

DBA has to look for stat which is taking abnormal % of DB time.  

9. Operating System Statistics - Detail:

This is the information related to OS, what is the load status on System shown here.


This report shows, system is 62 and 70% idle at time of report taken, So, there is no resource crunch at system level. But if, you found very high busy, user or sys % and indeed this will led to low idle %. Investigate what is causing this. OS Watcher is the tool which can help in this direction. 

Next, very crucial part of AWR report for a DBA is SQL Statistics. Which has all sql query details executed during report time interval. 


We will explore few of them, To understand, how to analyzed these reports. Let's start with

10. SQL Ordered by Elapsed Time:

As explained by name itself, this lists SQL queries ordered by Elapsed time into reported time interval.


In this report, look for query has low executions and high Elapsed time per Exec (s) and this query could be a candidate for troubleshooting or optimizations. In above report, you can see first query has maximum Elapsed time but no execution. So you have to investigate this.

In Important point, if executions is 0, it doesn't means query is not executing, this might be the case when query was still executing and you took AWR report. That's why query completion was not covered in Report. 

11. SQL Ordered by CUP Time:

In this report, SQL queries are listed on the basis of CPU taken by the query i.e. queries causing high load on the system. The top few queries could be the candidate query for optimization.









From above stat, look for queries using highest CPU Times, If a query shows executions 0, this doesn't means query is not executing. It might be same case as in SQL queries ordered by Elapsed time. The query is still executing and you have taken the snapshot.

However, There are so many other stats in AWR Report which a DBA needs to consider, I have listed only ten of them but these are the most commonly used stats for any performance related information.

Please share you view about this article, Does it helps you to understand, How to analyze AWR Report.


Refrence-http://www.dbas-oracle.com

Wednesday, May 24, 2017

 ORA-25153 error is because attempt was made to use space in a temporary tablespace with no files (no datafiles defined).
To solve the problem, the solution is just by adding files (datafiles) to the TEMP tablespace by using ADD TEMPFILE command, or by using Add Datafiles in Oracle Enterprise Manager.
If you check and found that TEMP tablespace already has data files, check the default temporary tablespace for all users and your database and set the default temporary tablespace to a valid temporarary tablespace.
To check the default temporary tablespace of the database:
SQL> select property_name, property_value from database_properties;
The SQL will return the following results, look for DEFAULT_TEMP_TABLESPACE for the setting:
PROPERTY_NAMEPROPERTY_VALUE
————————————————————
DICT.BASE2
DEFAULT_TEMP_TABLESPACETEMP
DBTIMEZONE+01:00
NLS_NCHAR_CHARACTERSETAL16UTF16
GLOBAL_DB_NAMEARON.GENERALI.CH
EXPORT_VIEWS_VERSION8
NLS_LANGUAGEAMERICAN
NLS_TERRITORYAMERICA
NLS_CURRENCY$
NLS_ISO_CURRENCYAMERICA
NLS_NUMERIC_CHARACTERS.,
NLS_CHARACTERSETWE8ISO8859P1
NLS_CALENDARGREGORIAN
NLS_DATE_FORMATDD-MON-RR
NLS_DATE_LANGUAGEAMERICAN
NLS_SORTBINARY
NLS_TIME_FORMATHH.MI.SSXFF AM
NLS_TIMESTAMP_FORMATDD-MON-RR HH.MI.SSXFF AM
NLS_TIME_TZ_FORMATHH.MI.SSXFF AM TZR
NLS_TIMESTAMP_TZ_FORMATDD-MON-RR HH.MI.SSXFF AM TZR
NLS_DUAL_CURRENCY$
NLS_COMPBINARY
NLS_LENGTH_SEMANTICSBYTE
NLS_NCHAR_CONV_EXCPFALSE
NLS_RDBMS_VERSION9.2.0.6.0
If default temporary tablespace is wrong the alter it with the following command:
SQL> alter database default temporary tablespace temp;
To check default temporary tablespace for all users of the database:
SQL> select username, temporary_tablespace, account_status from dba_users;
The query will return the following result, check if all users TEMPORARY_TABLESPACE is set to correct settings:
USERNAMETEMPORARY_TABLESPACEACCOUNT_STATUS
——————————————————————————————–
SYSTEMPRYOPEN
SYSTEMTEMPOPEN
OUTLNTEMPOPEN
DBSNMPTEMPOPEN
DBMONITORTEMPOPEN
TESTTEMPOPEN
WMSYSTEMPEXPIRED & LOCKED
If wrong temporary tablespace is found, alter it with the correct tablespace name (for example, sys) with the following SQL:
SQL> alter user sys temporary tablespace temp;
Alternatively, recreate or add a datafile to your temporary tablespace and change the default temporary tablespace for your database;
SQL> drop tablespace temp including contents and datafiles;
SQL> create temporary tablespace temp tempfile '/db/temp01.dbf' size 100m autoextend off extent management local uniform size 1m;


SQL> alter database default temporary tablespace temp;
reference site:-https://techjourney.net/ora-25153-temporary-tablespace-is-empty-error-in-oracle/

Tuesday, May 16, 2017

ORA-04024: self-deadlock detected while trying to mutex pin cursor 0xC1F2B19A8, ORA-01092: ORACLE instance terminated. Disconnection forced


I was performing a clone from a cold backup in a 12c (12.1.0.2) environment and was stuck with the errors (ORA-04024 and ORA-01092). The database restore was completed but while performing the ‘alter database open resetlogs’ it was giving the errors mentioned above. Below are the steps followed and resolution for the same.
Issue:
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-04024: self-deadlock detected while trying to mutex pin cursor 0x3BBEFC1C8
Process ID: 10432
Session ID: 441 Serial number: 38548




Cause:
It is a bug in 12.1.0.2 version of DB
Resolution:
1. Shutdown the database (from mount state)
2. Recreated the controlfile
SQL> @recreate_control_file.trc
Control file created.
3. Recovered the database using the redolog
RECOVER DATABASE USING BACKUP CONTROLFILE until cancel ;
SQL> RECOVER DATABASE USING BACKUP CONTROLFILE until cancel ;
ORA-00279: change 10493059653949 generated at 10/25/2016 22:38:24 needed for
thread 1
ORA-00289: suggestion : +DATA_SIT
ORA-15173: entry ‘ARCHIVELOG’ does not exist in directory ‘SIT’
ORA-00280: change 10493059653949 for thread 1 is in sequence #1
Applied the redolog to do the recovery
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA_SIT/sit/onlinelog/group_8.264.903647441
ORA-00308: cannot open archived log
‘+DATA_SIT/SIT/onlinelog/group_8.264.903647441’
ORA-17503: ksfdopn:2 Failed to open file
+DATA_SIT/SIT/onlinelog/group_8.264.903647441
ORA-15012: ASM file ‘+DATA_SIT/SIT/onlinelog/group_8.264.903647441’ does not
exist
Media recovery was completed
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
+DATA_SIT/SIT/ONLINELOG/group_1.460.926203011
Log applied.
Media recovery complete.
4. Tried to open the database using ‘resetlogs’, but stuck with the same issue
SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01092: ORACLE instance terminated. Disconnection forced
ORA-04024: self-deadlock detected while trying to mutex pin cursor 0xC1F2B19A8
Process ID: 25670
Session ID: 1895 Serial number: 9147
5. So, after investigating in Oracle site, performed the below fix and it went fine,
SQL> shut immediate
Database dismounted.
ORACLE instance shut down.
SQL> startup mount
ORACLE instance started.
Total System Global Area 5.1540E+10 bytes
Fixed Size 3725224 bytes
Variable Size 6845106264 bytes
Database Buffers 4.4426E+10 bytes
Redo Buffers 264708096 bytes
Database mounted.




So finally we can fix control it. and solve the issue.
 SQL> alter system set “_fix_control”=’9550277:ON’;

System altered.
SQL> alter database open;
Database altered.

Wednesday, May 3, 2017

Listener Fails To Start With IPC Permission Errors (TNS-12546 TNS-12555 TNS-00516 TNS-00525 Solaris Error: 13: Permission denied)

Ran into this issue where multiple Oracle Homes resides with listeners running from under different OS users or we are/have been running the same listener under different OS user(s).
Trying  to start the listener it is failing with permissions errors:
LSNRCTL> startStarting /app/oracle/product/102/bin/tnslsnr: please wait...
TNSLSNR for Solaris: Version 10.2.0.3.0 - ProductionSystem parameter file is /app/oracle/product/102/network/admin/listener.oraLog messages written to /app/oracle/product/102/network/log/listener.logListening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=meta4)))Error listening on: (DESCRIPTION=(address=(protocol=ipc)(key=meta3)))
TNS-12546: TNS:permission denied
TNS-12560: TNS:protocol adapter error 
TNS-00516: Permission denied  
Solaris Error: 13: Permission denied
— OR —
$ lsnrctl start LISTENERLSNRCTL for Linux: Version 10.2.0.1.0 - Production on 11-JUN-2007 11:05:52Copyright (c) 1991, 2005, Oracle.  All rights reserved.Starting /u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr: please wait...TNSLSNR for Linux: Version 10.2.0.1.0 - ProductionSystem parameter file is /u01/app/oracle/product/10.2.0/db_1/network/admin/listener.oraError listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC2)))
TNS-12555: TNS:permission denied
TNS-12560: TNS:protocol adapter error 
TNS-00525: Insufficient privilege for operation  
Linux Error: 1: Operation not permittedListener failed to start. See the error message(s) above...
Some of the changes associated with such errors are:
  • Some IPC keys have been reused from older listener(s)
  • Same listener has previously been started under different OS users
  • Listeners running under different OS users are using the same IPC keys
  • Permissions in /var/tmp have been previously reset.
Cause
Older IPC socket file(s) were left in /var/tmp/.oracle and they cannot be deleted by the listener upon startup.

Solution
Please be aware that this step should ONLY be taken as a last resort and ONLY when there are no  Net connections to the database(s).  This procedure could cause active sessions to be terminated.  It is NOT recommended to take this action in a RAC environment. 
·         Stop all listeners
·         Remove the /var/tmp/.oracle directory, e.g. run the following from the root account:

rm -rf /var/tmp/.oracle

Clearing the /var/tmp/.oracle directory will force the listeners to recreate the socket files from fresh, each with its own correct ownership.