Showing posts with label Financial Data Quality Management. Show all posts
Showing posts with label Financial Data Quality Management. Show all posts

Thursday, July 23, 2020

// // Leave a Comment

Automation of 'Maintain ODI Session Data' System Maintenance Task

In 'FDMEE System Maintenance Tasks' blog series, we have already covered the following topics :

Manual implementation:

FDMEE: System Maintenance Tasks: Maintain FDMEE Data Tables
FDMEE: System Maintenance Tasks: Maintain ODI Session Data
FDMEE: System Maintenance Tasks: Maintain Application Folder
FDMEE: System Maintenance Tasks: Maintain Process Tables
FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

Automation:

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

If you have not already gone through the above articles, I would suggest you read them first to get a comprehensive understanding of all the Hyperion Financial Data Quality Management (FDMEE) System Maintenance Tasks.

Note:
  • The demonstrating Hyperion environment has the 'Windows Server 2012 R2 Standard' operating system. 
  • The demonstrating Hyperion environment has 'Oracle database server 12.2.0.2 (18c)' as backend database. 
  • This post has been written and associated activities have been demonstrated on Oracle FDMEE version 11.1.2.4.220.
  • There is no outage required on the FDMEE application to execute these scripts. But it is advisable not to run any data loads during the activity.
  • Don't forget to take complete FDMEE schema backup before attempting to run these scripts. 
  • Different paths and folder names mentioned in this post may slightly vary for different Hyperion setups but at large it should be the same.  

Command syntax of FDMEE batch utility executescript.bat:
----------------------------------------------------------------------------------
executescript <username> <password> "<script name>" "<parameters>" <execution mode>

Where:
  • <username>: is your FDMEE user running the script
  • <password>: can be either the hard-coded password or a file with encoded password (-f:filename)
  • <script name>: is the name of the script registered in Script Registration page (not the python script filename). You find the script name here: Navigate-Administer-Data Management-Setup tab-Script Registration
  • <parameters>: are the script parameters in the form "Display Name=Parameter Value"
  • <execution mode>: is either SYNC (synchronous) or ASYNC (asynchronous). SYNC—Process runs immediately and control returns when process completes. ASYNC—When the ODI job is submitted, control returns. The load process continues to execute in ODI.
Now the command syntax in more detailed form will look like below:

executescript <username> <password> "<script name>" "<parameterdisplay1=value1>" "<parameterdisplay2=value2>" <SYNC | ASYNC>

If you are going to use an encrypted password (which is the recommended best practice), following will be the command syntax:

executescript <username> <password | -f:password.txt> "<script name>" "<parameterdisplay1=value1>" "<parameterdisplay2=value2>" <SYNC | ASYNC>

You will find the configuration steps for password encryption in the FDMEE admin guide as shown below:

Automation of 'Maintain ODI Session Data' System Maintenance Task

You need to ensure that your encrypted password.txt file is placed in the location defined in Navigate-Administer-Data Management-Setup tab-System Settings-Encrypted Password Folder.

Automating 'Maintain ODI Session Data' System Maintenance Task:
-------------------------------------------------------------------------------------------------
In this post, we will see how to automate 'Maintain ODI Session Data' System Maintenance Task. I suggest you to first read this post FDMEE: System Maintenance Tasks: Maintain ODI Session Data for theoretical understanding, manual implementation, and post-execution activities of 'Maintain ODI Session Data' System Maintenance Task.

When you execute ‘Maintain ODI Session Data’ purge script manually from Data Management console, it seeks one input parameter: Days to keep ODI Sessions

To run ‘Maintain ODI Session Data’ purge script using executescript.bat command-line utility below is the command syntax:

executescript username password "MaintainODISessionData" "Days to keep sessions=EnterDaysInNumber" SYNC

You need to enter the number of days you want to retain data for in ODI tables.

Note: Maintain ODI Session Data scrip takes some time to execute so its advisable not to purge too much in one go.

Suppose using FDMEE Admin user, you want to run 'Maintain ODI Session Data' purge script for 90 days (3 months) i.e. you want to delete all the ODI Sessions details keeping only last 90 days (3 months) details.

In order to achieve the above goal, create a batch script name 'MaintainODISessionData.bat' under folder E:\Admin\scripts having the following content: 

MaintainODISessionData.bat:
--------------------------------------------------
@echo off

REM Set the value for the variables  
set "username=Admin"
set "password=YourAdminPassword"
set "DaysToKeepODISessions=90"
set "logfile=E:\Admin\scripts\MaintainODISessionData.log"

REM Clear the content of existing logfile
rem.> %logfile%

REM Navigate the executescript.bat utility folder
cd "E:\apps\OracleEPM\Middleware\user_projects\epmsystem_fdm\FinancialDataQuality"

REM Run executescript to perform "MaintainODISessionData" for the mentioned days to keep ODI Sessions 


> %logfile% (
echo call executescript %username% %password% "MaintainODISessions" "Days to keep sessions=%DaysToKeepODISessions%" SYNC
call executescript %username% %password% "MaintainODISessions" "Days to keep sessions=%DaysToKeepODISessions%" SYNC
)

cd E:\Admin\scripts

Change the paths and variables mentioned in the script as per your requirement and FDMEE application setup.

Now login to your FDMEE server, open an Administrator Command prompt and run MaintainODISessionData.bat.

After completion of the script your MaintainODISessionData.log will be generated in below format:

Automation of 'Maintain ODI Session Data' System Maintenance Task
To verify whether your process ID has successfully completed or not, you can navigate to Process Details page (In Data Management console, click on the Process Details link under the Workflow tab—Monitor—Process Details):

Automation of 'Maintain ODI Session Data' System Maintenance Task

Click on the 'Show' button against the Process ID to open the corresponding log file to see more details. 

That's all for this post.

I hope this article has helped you.
Your suggestions/feedback are most welcome.
Keep learning and Have a great day!!!
Read More

Monday, July 20, 2020

// // 4 comments

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

For those who are new to my blog, we have already covered all the Oracle Hyperion Financial Data Quality Management (FDMEE) System Maintenance Tasks with the step-by-step process to manually execute them. If you have not already gone through those articles, I would suggest you click on the below links and read them first to get a comprehensive understanding of all the Hyperion Financial Data Quality Management (FDMEE) System Maintenance Tasks.

FDMEE: System Maintenance Tasks: Maintain FDMEE Data Tables
FDMEE: System Maintenance Tasks: Maintain ODI Session Data
FDMEE: System Maintenance Tasks: Maintain Application Folder
FDMEE: System Maintenance Tasks: Maintain Process Tables
FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

Running 'FDMEE System Maintenance Tasks' manually for multiple combinations of Target Applications and Categories requires a lot of manual effort causing it to be a tiresome process because for each maintenance task you need to manually select all the combinations of target applications and categories one by one in Data Management (FDMEE) console for a particular set of start and end period. So why not have a script which does all of these tasks for us and make life easier.

But now using below given automated script you no more need to do all these activities manually These automated jobs can be scheduled in Windows Task Scheduler to run at any chosen day/time. Also, these automated scripts don’t require any outage on FDMEE application and run smoothly using only the inbuilt functionality of FDMEE.

Note:
  • The demonstrating Hyperion environment has the 'Windows Server 2012 R2 Standard' operating system. 
  • The demonstrating Hyperion environment has 'Oracle database server 12.2.0.2 (18c)' as backend database. 
  • This post has been written and associated activities have been demonstrated on Oracle FDMEE version 11.1.2.4.220.
  • There is no outage required on the FDMEE application to execute these scripts. But it is advisable not to run any data loads during the activity.
  • Don't forget to take complete FDMEE schema backup before attempting to run these scripts. 
  • Different paths and folder names mentioned in this post may slightly vary for different Hyperion setups but at large it should be the same.  
In this blog series, we are going to learn how can we automate these 'FDMEE System Maintenance Tasks' one-by-one and schedule it via Windows Task Scheduler to run whenever we want to do FDMEE System maintenance activities.

FDMEE batch utility executescript.bat
------------------------------------------------------------
Oracle provides a utility called executescript.bat to run each of these System Maintenance Tasks as well as other custom scripts from the command line. This command-line utility allows us to execute custom scripts either from Windows or Unix/Linux systems. Unfortunately, this topic is not well documented in the FDMEE admin guide.

You will find the executescript.bat (among others) in the folder E:\apps\OracleEPM\Middleware\user_projects\epmsystem_fdm\FinancialDataQuality.

Using utility executescript.bat you can automate all the FDMEE System Maintenance Tasks.

Command syntax of FDMEE batch utility executescript.bat:
----------------------------------------------------------------------------------
executescript <username> <password> "<script name>" "<parameters>" <execution mode>

Where:
  • <username>: is your FDMEE user running the script
  • <password>: can be either the hard-coded password or a file with encoded password (-f:filename)
  • <script name>: is the name of the script registered in Script Registration page (not the python script filename). You find the script name here: Navigate-Administer-Data Management-Setup tab-Script Registration
  • <parameters>: are the script parameters in the form "Display Name=Parameter Value"
  • <execution mode>: is either SYNC (synchronous) or ASYNC (asynchronous). SYNC—Process runs immediately and control returns when process completes. ASYNC—When the ODI job is submitted, control returns. The load process continues to execute in ODI.
Now the command syntax in more detailed form will look like below:

executescript <username> <password> "<script name>" "<parameterdisplay1=value1>" "<parameterdisplay2=value2>" <SYNC | ASYNC>

If you are going to use an encrypted password (which is the recommended best practice), following will be the command syntax:

executescript <username> <password | -f:password.txt> "<script name>" "<parameterdisplay1=value1>" "<parameterdisplay2=value2>" <SYNC | ASYNC>

You will find the configuration steps for password encryption in the FDMEE admin guide as shown below:

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

You need to ensure that your encrypted password.txt file is placed in the location defined in Navigate-Administer-Data Management-Setup tab-System Settings-Encrypted Password Folder.

Automating 'Maintain FDMEE Data Tables' System Maintenance Task:
--------------------------------------------------------------------------------------------
In this post, we will see how to automate 'Maintain FDMEE Data Tables' System Maintenance Task. I suggest you to first read this post FDMEE: System Maintenance Tasks: Maintain FDMEE Data Tables for theoretical understanding, manual implementation, and post-execution activities of 'Maintain FDMEE Data Tables' System Maintenance Task.

When you execute ‘Maintain FDMEE Data Tables’ purge script manually from Data Management console, it seeks the following input parameters:
  • Target Application
  • Start Period
  • End Period
  • Category
To run ‘Maintain FDMEE Data Tables’ purge script using executescript.bat command-line utility below is the command syntax:

executescript username password "MaintainFDMEEDataTables" "Target Application=TargetAppName" "Start Period=YYYY-MM-DD" "End Period=YYYY-MM-DD" "Select Category=CategoryKey" SYNC

Description:
  • TargetAppName should be entered the same as what you get in the output by running below query against FDMEE schema:
          SELECT APPLICATION_NAME FROM AIF_TARGET_APPLICATIONS; 

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

Above query-output is same as what you see in Data Management console Target Applications list values as shown below:

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task
  • Start Period and End Period can be chosen from the values what you get in the output by running below query against FDMEE schema:
          SELECT PERIODDESC, PERIODKEY FROM TPOVPERIOD;

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

Above query-output is same as what you see in Data Management console Period list values as shown below:

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

If you want to quickly check what is the min and max value of period to help you choose your Start Period and End Period, you can run below query against FDMEE schema:

           SELECT MIN(PERIODKEY), MAX(PERIODKEY) FROM TPOVPERIOD; 

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

Note: Start Period and End Period values must be entered in the 'YYYY-MM-DD' format only.
  • In Category segment, you need to enter the 'Category Key (number)' what you get in the output by running below query against FDMEE schema:
          SELECT CATNAME, CATKEY FROM TPOVCATEGORY;

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

Above query-output is same as what you see in Data Management console Category list values as shown below:

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

Most of the time, we need to run 'MaintainFDMEEDataTables' purge script for many combinations of various Target applications and Categories for a selected Start Period and End Period range.

Let's understand the automation script for 'MaintainFDMEEDataTables' with the help of an example.

Suppose using FDMEE Admin user, you want to run 'MaintainFDMEEDataTables' purge script for 4 Target applications name TargetApp1, TargetApp2, TargetApp3, TargetApp4, and 3 Categories keys name CategoryKey1, CategoryKey2, CategoryKey3 for Start Period Oct 2017 to End Period Apr 2018.

In order to achieve the above goal, create a batch script name 'MaintainFDMEEDataTables.bat' under folder E:\Admin\scripts having the following content:

MaintainFDMEEDataTables.bat:
--------------------------------------------------
@echo off

REM Set the value for the variables 
set "username=Admin"
set "password=YourAdminPassword"
set "startperiod=2017-10-01"
set "endperiod=2018-04-01"
set "logfile=E:\Admin\scripts\output.log"

REM Clear the content of existing logfile
rem.> %logfile%

REM Navigate the executescript.bat utility folder
cd "E:\apps\OracleEPM\Middleware\user_projects\epmsystem_fdm\FinancialDataQuality"

REM Run executescript to trigger "MaintainFDMEEDataTables" for all combinations of 4 Target applications and 3 Category Keys for the mentioned Start and End Period range

for %%f in (TargetApp1 TargetApp2 TargetApp3 TargetApp4) do (
    for %%i in (CategoryKey1 CategoryKey2 CategoryKey3) do (
      echo call executescript %username% %password% "MaintainFDMEEDataTables" "Target Application=%%f" "Start Period=%startperiod%" "End Period=%endperiod%" "Select  Category=%%i" SYNC
call executescript %username% %password% "MaintainFDMEEDataTables" "Target Application=%%f" "Start Period=%startperiod%" "End Period=%endperiod%" "Select Category=%%i" SYNC
)
echo ------------------------------------------------------------------------------
echo ------------------------------------------------------------------------------
) >>%logfile%

cd E:\Admin\scripts

Change the paths and variables mentioned in the script as per your requirement and FDMEE application setup.

You can put your Target Application names in the segment (TargetApp1 TargetApp2 TargetApp3 TargetApp4) and Category Keys in (CategoryKey1 CategoryKey2 CategoryKey3) separated by single space. 

Now login to your FDMEE server, open an Administrator Command prompt, and run 'MaintainFDMEEDataTables.bat'.

After completion of the script your output.log will be generated in below format:

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

To verify whether your process IDs 8237, 8238... have successfully completed or not, we can navigate to Process Details page (In Data Management console, click on the Process Details link under the Workflow tab—Monitor—Process Details):

Automation of 'Maintain FDMEE Data Tables' System Maintenance Task

We can see our process IDs 8237, 8238 have executed successfully. Click on the 'Show' button for Process IDs 8237 to open the corresponding log file to see more details. Similarly, you can verify other process IDs recorded in output.log file for various combinations of Target Applications and Categories.

That's all for this post.

I hope this article has helped you.
Your suggestions/feedback are most welcome.
Keep learning and Have a great day!!!
Read More

Wednesday, May 20, 2020

// // 3 comments

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

Topic: How to manually run FDMEE System Maintenance Tasks: Maintain EBS GL Balances Table

Hi Friends,

This is the fifth post under Oracle Hyperion Financial Data Quality Management (FDMEE): System Maintenance Tasks blog series where we are going to cover the Maintain EBS GL Balances Table script.

If you have not yet read the previous four posts of this blog series, I would suggest you to go through them on below links to get a comprehensive picture of Oracle Hyperion Financial Data Quality Management (FDMEE) System Maintenance Tasks:

FDMEE: System Maintenance Tasks: Maintain FDMEE Data Tables
FDMEE: System Maintenance Tasks: Maintain ODI Session Data

In this fifth post, we will focus on the ‘Maintain EBS GL Balances Table’ task and will see how to manually execute ‘Maintain EBS GL Balances Table’ purge script.  

Important Note:
  • This post has been written and associated activities have been demonstrated on Oracle Hyperion Financial Data Quality Management (FDMEE) version 11.1.2.4.210.
  • The demonstrating Hyperion environment has Oracle database server 12.2.0.2 (18c) as backend database.
  • There is no outage required on the Oracle Hyperion Financial Data Quality Management (FDMEE) application to execute these purge scripts. But it is advisable not to run any data loads during the purge activity.
  • Based on the data retention policy of your organization, you should pre-decide how many days of data you want to retain for Oracle Hyperion Financial Data Quality Management (FDMEE) EBS GL Balances Table.
  • Don't forget to take complete Oracle Hyperion Financial Data Quality Management (FDMEE) schema backup before attempting to run these purge scripts
Maintain EBS GL Balances Table

FDMEE is the EPM tool for data integration. Using FDMEE, ERP applications like Oracle EBS, PeopleSoft, SAP, etc. can be directly connected to your target Oracle Hyperion applications for data loading.

You would have seen FDMEE integration system having Oracle EBS General Ledger as data source system and HFM or Hyperion Planning as target applications. Data (GL_Balances) is extracted from Oracle EBS source system using ODI (FDMEE), moved to staging tables (AIF_EBS_GL_BALANCES_STG), and then eventually loaded to your target applications after transformation.

Maintain EBS GL Balances Table’ purge script maintains the AIF_EBS_GL_BALANCES_STG table in the FDMEE schema. EBS General Ledger balances contain a snapshot of the General Ledger balances and are deleted based on the General Ledger period. Maintain EBS GL Balances Table’ purge process determines the list of General Ledger periods between the start and end periods. 

Following are the input parameters which you need to enter while running ‘Maintain EBS GL Balances Table’ purge script:
  • Source System
  • Start Period
  • End Period
Before proceeding further, with purging steps, I would suggest you to check and note down the pre-purge FDMEE schema size to see how much space has been freed up post purge activity.

Step-by-Step process to execute ‘Maintain EBS GL Balances Tables’ purge script

1- Log in to Workspace with Admin credentials.

2- Open Oracle Hyperion Financial Data Quality Management (FDMEE) console (Data Management) from Workspace and on the Workflow tab, under Other, select Script Execution:

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

3In Script Execution, and then in Custom Script Group, select System Maintenance Tasks.

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

4- From the Scripts grid, select Maintain EBS GL Balances Table script and Click Execute.

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

5It will prompt to enter the following parameter values on Execute Script screen.
  • Source System
  • Start Period
  • End Period
FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

6- Click on the search icon for Source System

Select any Source System name for which you want to purge data and click OK.

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

7- Source System will be populated. 

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table


8- Now enter Start Period and End Period. 

For example: If you want to purge all data from Nov-2016 to Oct-2017, then select Start Period and End Period as shown below:


FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table


9- Keep Execution mode selected as Online. The online method processes the report immediately.

Execution modes:

online  — ODI processes the data in sync mode (immediate processing).
offline — ODI processes the data in async mode (runs in background).

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

10- Optionally, If you want to schedule the purge script run rather than running it instantly, click on Schedule option and enter the scheduling details as per your requirement.

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

11- If you want to instantly run the purge script, then directly click on OK (not on the Schedule option).

You will see a message informing the process ID of your script run as shown below:

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

12- To see the status of your script execution, click on the Process Details link under the Workflow tab—Monitor—Process Details

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

13- Above we can see our script with Process ID 8184 executed successfully. 

Click on the Show button for Process ID 8184 to open the corresponding log file.

FDMEE: System Maintenance Tasks: Maintain EBS GL Balances Table

In the log file, look for the message "Maintain EBS GL Table: Completed" to confirm that the script has run successfully without any error. If you encounter any errors, troubleshoot them based on the cause mentioned in the log file. 

You need to repeat all of the above steps for other Source System from where your FDMEE data extractions are running to target Hyperion applications. 

Once you are done with running the Maintain EBS GL Balances Table purge script, you can check your Hyperion Financial Data Quality Management (FDMEE) schema size to see how much space has been cleared.

Here it’s important to note that after running the Maintain EBS GL Balances Table purge script, your DBA needs to do FDMEE tables reorganization activity in order to reclaim freed space. 

Actually, when you delete the data from the Oracle tables, the data blocks will go onto the free list for that table. That free space will be used for the subsequent insert and update operations into the table. You need to perform table reorganization to reclaim that free space.

Note: Reorganization of tables requires outage in your Hyperion environment so you have to stop all your application services prior to the reorganization activity to be performed by your application DBA.

You may also need to ask your DBA team to 'Rebuild Indexes' in your Hyperion Financial Data Quality Management (FDMEE) database after performing the 'Tables Reorganisation' activity.

If you miss rebuilding indexes after tables reorganization activity, your FDMEE data load job may take longer than normal to complete.

Index rebuild becomes absolutely necessary especially when you delete millions of records from FDMEE tables as part of running FDMEE purge scripts because the corresponding indexes of those deleted records go in the unusable state after tables reorganization activity. Due to this bulk deletion of records, indexes become fragmented causing indexes taking more time to give the output and hence delaying your FDMEE data load job.

You will observe the following type of error during your FDMEE data load run if you do not rebuild database indexes after tables reorganization activity:
ORA-01502: index 'FDMEE_ODI.PK_AIF_EBS_GL_BALANCES_STG' or partition of such index is in unusable state.
Once both the DBA tasks (Tables reorganization + Rebuilding Indexes) are completed, compare pre and post purge size of your Hyperion Financial Data Quality Management (FDMEE) schema to see how much has the schema size reduced. 

Maintain EBS GL Balances Table purge script can also be automated to run using Executescript batch file located in the folder: 

<drive>\Oracle\Middleware\user_projects\epmsystem1_fdm\FinancialDataQuality.

We will cover the automation part in a separate post of this blog series. 

That's all for this post.

I hope this article has helped you.
Your suggestions/feedback are most welcome.
Keep learning and Have a great day!!!
Read More