ROBOCOPY Backup files over Network paths and ERROR 5 (0x00000005) Copying NTFS Security to Destination Directory-Access denied

As part of the annual EPM DR Rehearsal, we got a requirement to move Prod Application backup files from PROD Network location to DR network path.

In this post, we will see how we can RoboCopy EPM backup files from one Network location to another network path.

Why ROBOCOPY?

If you are copying files from one network path to another and you have Windows Vista or later operating systems then Robocopy is a better choice than any other option. Because you don't need to bother with drive mappings, since it handles UNC paths just fine.

Robocopy is not a third party software. It's native (built-in) to all versions of Windows Vista and later.

It is usually far more reliable than xcopy command, and provides a lot more options.

Robocopy is tolerant of interrupts during copying i.e. it can pick up where it left off if it gets stopped for some reason. It has the ability to recover from certain types of network hiccups automatically.

Read Link1 and Link2 for more details about ROBOCOPY and its various options.

Script using ROBOCOPY to copy EPM backups from PROD to DR network path:

@echo off

for /f "delims=" %%a in ('wmic OS Get localdatetime ^| find "."') do set "dt=%%a"

::Format the WMIC command output in DDMMYYYY format

set "YY=%dt:~0,4%"

set "MM=%dt:~4,2%"

set "DD=%dt:~6,2%"

set "today_date=%DD%%MM%%YY%"

::Define Source path

set sourcepath1=\\PROD_Network_Share\data\EPM_Backups\%today_date%

set sourcepath2=\\PROD_Network_Share\data\EPM_Backups\DataZip

set sourcepath3=\\PROD_Network_Share\data\EPM_Backups\Scripts

::Define Destination path

set destinationpath1=\\DR_Network_Share\data\EPM_Backups_Copy\%today_date%

set destinationpath2=\\DR_Network_Share\data\EPM_Backups_Copy\DataZipCopy

set destinationpath3=\\DR_Network_Share\data\EPM_Backups_Copy\ScriptsCopy\%today_date%

::Define Log path

set logfile=E:\Admin\Prod_To_DR_Copy.log

::Run RoboCopy commands

REM Copy all the files, folders and sub-folders from source to destination

robocopy %sourcepath1% %destinationpath1% /E /COPY:DAT /NP /LOG+:"%logfile%"

REM Copy today'sdate(DDMMYY).zip file from source to destination

robocopy %sourcepath2% %destinationpath2% %today_date%.zip /COPY:DAT /NP /LOG+:"%logfile%"

REM Copy all the files with extensions .sh, .mxl, .ksh, .scr from source to destination

robocopy %sourcepath3% %destinationpath3% *.sh *.mxl *.ksh *.scr /COPY:DAT /NP /LOG+:"%logfile%"

Below is what each ROBOCOPY command options used above means:

• /E = Copy files including subfolders (even empty ones)
• /COPY:copyflag[s] = what to COPY for files. Here we have selected DAT: D=Data, A=Attributes, T=Timestamps 
• /NP = No Progress - don’t display % copied text in logfile; this keeps filesize down. 
• /LOG+:logfile = Output status to LOG file (+= append to existing log).

ERROR 5 (0x00000005) Copying NTFS Security to Destination Directory. Access denied

When I was trying to figure out the right set of options for ROBOCOPY command, I encountered this error multiple times.

This error is usually caused by RoboCopy trying to copy the security settings of the files, and this causes some mismatch regarding the file permissions. 

There is a /B switch in RoboCopy for copying in backup mode but Backup mode cannot circumvent explicit NTFS deny ACL’s if the copier isn’t the objects’ owner.

Solution: Use /COPY:DAT only

Option /COPY:copyflag[s] can take multiple values based on what you want to copy for files. To Copy ALL file info (equivalent to /COPY:DATSOU), there is an option /COPYALL.

To overcome the above-mentioned error, you should use /COPY:DAT instead of the /COPYALL option, because /COPY:DAT  ignores the NTFS access control lists (the COPY:S parameter) of the files you're copying. 

This works because /COPYALL is equivalent to /COPY:DATSOU, D=Data, A=Attributes, T=Timestamps, S=Security=NTFS ACLs, O=Owner info, U=aUditing info. While we mainly need Data and Timestamps of the files for EPM backups.

That's all for this post.

I hope this article has helped you. 

Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Read More

Send PowerShell email from Windows batch file without creating separate email.ps1 file

In this article, we will see how we can send a PowerShell email from the Windows batch file without creating any separate email.ps1 file.

Suppose you have a batch script to perform a certain task and once that task is completed you want to be notified through an email alert.

One way to achieve this is by creating a separate email.ps1 file and calling that email.ps1 from your batch file. You can find multiple solutions on Google for this approach. But if the purpose is just to send an email alert with an attachment like a log file or with some text in the email body then why to go for creating another email.ps1 file.

You can simply insert the below code at the end of your batch script or with some conditional statements to send an email alert within the batch script:

:: Send Email using PowerShell

Powershell.exe -command "& {Send-MailMessage -From Admin@company.com -To users@company.com -Subject 'Prod Hyperion Backup Copy to DR for DR Rehearsal' -SmtpServer 'smtpserver.company.com' -Body 'PFA log file consisting of details of PROD Hyperion Backup files copied to DR Network path' -Attachments 'E:\Data\Prod_DR_Copy.log'}"

Here:

• '&' is the call operator. The call operator (&) allows you to execute a command, script, or function.
• Change the parameter values (To, From, Subject, Body, SmtpServer, Attachments) as per your requirement.  

There are many other parameters that you can add to the above command as per your need. Read this Link1 and Link2 for more details.

Make sure you copy and paste the above code as it is. Because on some occasions, I have noticed that using double quotes ("") in place of single quotes ('') will throw PowerShell error:

Send-MailMessage :  A positional parameter cannot be found that accepts arguement "SomeWord"...

It happens because the PowerShell command processor strips your double quotes and your "SomeWord" appears as a parameter to Send-MailMessage. So to avoid this error, you need to escape double quotes in your command.

The best way is to use single quotes ('') in your command as given above but If you really want to use double quotes for whatsoever reason, then use it like below by escaping double quotes ("") in -Subject and -Body parameters:

Powershell.exe -command "& {Send-MailMessage -From Admin@company.com -To users@company.com -Subject \"Prod Hyperion Backup Copy to DR for DR Rehearsal\" -SmtpServer "smtpserver.company.com" -Body \"PFA log file consisting of details of PROD Hyperion Backup files copied to DR Network path\" -Attachments "E:\Data\Prod_DR_Copy.log"}"

That's all for this post.

I hope this article has helped you. 

Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Read More

Automation: Batch Script to delete Hyperion backup folders older than N months/days

Hi Friends,

Topic: How to delete Oracle Hyperion backup folders older than N months/days

Maintaining a backup of various applications is an important part of any Oracle Hyperion Production system. In most of the cases, we have a network drive connected and accessible from all the servers of a Hyperion Production environment where we save the backup of Hyperion application data, metadata, artifacts, LCM exports, etc. either manually or through automated scripts like Lifecycle Management (LCM) automation, FDMEE data load jobs, Maxl scripts, etc. on a daily or weekly basis.

Over the course of time, your network drive is filled with past many months’ backup folders consuming a lot of space and thus causing a requirement for network drive housekeeping. In most of the cases, maintaining last 3-6 months application backup folders actually serve the purpose.   

We too faced the similar case where we had many Hyperion application LCM backup folders over a network shared drive starting from the year 2017, which were consuming a huge amount of space raising critical space alerts. To housekeep the network drive when we tried to delete some of these LCM backup folders manually, we encountered the notorious ‘Source Path Too Long’ error to make matters worse.

Therefore, to fix this issue permanently, we have created a batch script called “Network_Housekeeper” which works perfectly fine and is based on the following logic:

1- We have a LCM automated scheduled job which adds two huge sized LCM backup folders every day to the network drive path: \\NetworkServerAddress\HyperionBackupFolder\AutomatedLCMExport with the following folder naming convention:

HYP_PROD_YYYYMMDDhhmm

Note: To learn how you can add current date and time in your backup folders' name, kindly refer this link: Batch script to add today's date and time in file/folder name

2- The script maps the network drive path as a local drive on the server from where we trigger this script.

3- It identifies the date 4 months (120 days) before today’s date.

4- It lists out all the LCM backup folders with the name HYP_PROD_* having the oldest date folder first and compares the date of each folder with the date 4 months before the current date.

5- If the folder date is less than the date 4 months before the current date, it passes that folder name to ROBOCOPY purge command section as an input to delete that folder.

Note: ROBOCOPY purge command is used to troubleshoot ‘Source Path Too Long’ error while deleting the folder.

6- This loop continues until all the folders older than 4 months (120 days) are deleted.

7- The script saves the names of the deleted folders in a log file for your record: D:\Data\Scripts\NETWORK_HOUSEKEEPER\Deleted_Folder.log 

Network_Housekeeper Batch Script:

@echo off

REM Connect to the network drive path

pushd \\NetworkServerAddress\HyperionBackupFolder\AutomatedLCMExport

REM Set the number of days to retain the folders for. You need to change the day=-120 to the relevant number of days you want. 

set day=-120

echo >"%temp%\%~n0.vbs" s=DateAdd("d",%day%,now) : d=weekday(s)

echo>>"%temp%\%~n0.vbs" WScript.Echo year(s)^& right(100+month(s),2)^& right(100+day(s),2)

for /f %%a in ('cscript /nologo "%temp%\%~n0.vbs"') do set "result=%%a"

del "%temp%\*%~n0.vbs"

set "yyyy=%result:~0,4%"

set "mm=%result:~4,2%"

set "dd=%result:~6,2%"

set "final=%yyyy%%mm%%dd%"

setlocal enabledelayedexpansion

REM List out all the folders with name HYP_PROD_* having the oldest folder first and then extract the date from each folder name 

for /f "delims=" %%a in ('dir "HYP_PROD_*" /a:d /o:d /b') do (

set "folder=%%a"

set folddate=!folder:~9,8!

REM If folder date is less than the date 4 months before the current date, trigger the 'work' section of the script

if !folddate! LSS !final! call :work

)

goto :EOF

REM Delete the folder using ROBOCOPY command route 

:work

echo !folder! >> D:\Data\Scripts\NETWORK_HOUSEKEEPER\Deleted_Folder.log

rmdir emptyfolder

mkdir emptyfolder

robocopy emptyfolder "!folder!" /purge                  

rmdir !folder!

rmdir emptyfolder

You can schedule this batch script through Windows Task Scheduler to run at any time of the day. I have tested this script on my system and it takes 80 seconds to delete a 2 GB LCM backup folder. That is quite ok! 

That's all for this post.

I hope this article has helped you. 

Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Read More

Batch Command/Script to extract Oracle EPM/Hyperion already applied patches in Windows server

Hi Friends,

Topic: How to extract Oracle Hyperion patches currently applied in your Hyperion environment using a batch script

In this post, we will see how you can extract all the Oracle Hyperion patches currently applied in your Hyperion environment Windows servers. 

When you plan to apply any newly released Oracle Hyperion patches to upgrade your Hyperion applications patch level, you may need to first know the currently applied Oracle patches for that particular Hyperion application. 

For the Linux/Unix Servers, to know how to extract Oracle Hyperion patches currently applied in your Hyperion environment using a shell/bash script, you may like to read this post: Shell script to extract Oracle Hyperion patches currently applied

Note: 

• This post has been written and associated activities have been demonstrated on Oracle Hyperion version 11.1.2.4.
• The demonstrating Hyperion environment has the 'Windows Server 2012 R2 Standard' operating system. 

Concept:

As you know, we all use opatch lsinventory command in order to see all the details of the patches that are currently installed on a particular Oracle Hyperion server.

The lsinventory command reports what has been installed on the system for a particular Oracle home directory, or for all installations. 

The following syntax is used for this command (on Windows machine):

opatch.bat lsinventory -oh E:\apps\OracleEPM\Middleware\EPMSystem11R1

where -oh option is your Oracle Home location. 

The result of opatch lsinventory command contains a lot of data like various versions, paths, bugs fixed, overlaying patch details, etc as shown in the below image. So in order to list out only the Oracle patch numbers and when they were applied you need to go through all the details generated as a result of opatch lsinventory command then extract the required details manually.

The below-mentioned batch script does all these things for you automatically by extracting out only the Oracle patch numbers with the date when they were applied. It saves the output into a log file for your record along with saving your manual effort. 

On Command prompt:

If you want to see all the applied patch list directly on your EPM/Hyperion Windows server's command prompt, run the below commands:

To see all installed patches:

----------------------------------------

cd E:\apps\OracleEPM\Middleware\EPMSystem11R1\OPatch

For EPMSystem11R1 patches:

opatch.bat lsinventory -oh E:\apps\OracleEPM\Middleware\EPMSystem11R1 -jdk E:\apps\OracleEPM\Middleware\jdk160_35 | findstr -i applied

For oracle_common patches:

opatch.bat lsinventory -oh E:\apps\OracleEPM\Middleware\oracle_common -jdk E:\apps\OracleEPM\Middleware\jdk160_35 | findstr -i applied

To see a particular installed patch:

---------------------------------------------

cd E:\apps\OracleEPM\Middleware\EPMSystem11R1\OPatch

For EPMSystem11R1 patches:

opatch.bat lsinventory -oh E:\apps\OracleEPM\Middleware\EPMSystem11R1 -jdk E:\apps\OracleEPM\Middleware\jdk160_35 | findstr "applied" | findstr "24738130"

For oracle_common patches:

opatch.bat lsinventory -oh E:\apps\OracleEPM\Middleware\oracle_common -jdk E:\apps\OracleEPM\Middleware\jdk160_35 | findstr "applied" | findstr "16964825"

Through Batch Script:

Create a folder (E:\Admin) in the server where you want to extract the installed Oracle patch list and then create a batch script (E:\Admin\test.bat) with the below codes under that folder.

@echo off

cd E:\apps\OracleEPM\Middleware\EPMSystem11R1\OPatch

call opatch.bat lsinventory -oh E:\apps\OracleEPM\Middleware\EPMSystem11R1 >"E:\Admin\Patch_History.txt"

cd E:\Admin

rem. > "E:\Admin\Installed_Patches_%COMPUTERNAME%.txt"

findstr /L /C:"Interim patches" "E:\Admin\Patch_History.txt" >>"E:\Admin\Installed_Patches_%COMPUTERNAME%.txt"

echo. >>"E:\Admin\Installed_Patches_%COMPUTERNAME%.txt"

FOR /F "tokens=*" %%A IN (
  'findstr /L "applied" "E:\Admin\Patch_History.txt"'
) DO (
  echo %%A >>"E:\Admin\Installed_Patches_%COMPUTERNAME%.txt"
)

When you run this test.bat script via command prompt, it will extract out the following things for you:

1. Total number of Oracle patches
2. Patch numbers of all the currently applied Oracle Hyperion patches
3. Date of applying these patches

It will also save the output in a log file with your SERVERNAME appended in the log file name.

Below is the folder structure and the output format of the above-given script:

You can run this script across all the Windows servers of your Hyperion environment to extract the list of applied Oracle Hyperion patches. 

In order to see which Oracle Hyperion applications your extracted patch numbers belong to, simply do the following:  

• Go to https://support.oracle.com.
• Select the Patches & Updates tab.
• In Patch Search, enter the patch number. 
• Make sure you are searching for a Patch Name or Number and select Search.

The corresponding Hyperion application name with patch details will be displayed. 

That's all for this post.

I hope this article has helped you. 

Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Read More

Automation: Batch script to add today's date and time in file/folder name

Topic: Batch script to create files/folders with current (today) date and time in their name

Hi Friends,

In this post, we will see a batch script to add the current date and time in Windows files or folders' names.

This batch script is very useful especially when you need to take periodic backups of Hyperion applications, LCM exports, Essbase Level 0 data exports, Oracle database schemas, creating log files, etc. This backup process is part and parcel of any administration work including Hyperion Administration.

In Windows system, manipulating date and time value is not that straight-forward compared to Linux/Unix systems.

Concept:

As you know, most of the time following commands are used to display the current date and time in Windows system when you quickly want to check the same:

date /t
time /t

But when it comes to batch scripting where you want to use date and time values, there is a problem with these commands. 

date /t command returns the current date using the windows local system settings for the "short date format" so it is Windows-version and region-dependent. This local system 'date and time setting' is fully customizable. Each language/region has its own settings and the users can change this too. One user may configure its system to show the short date as Tue060719; while another user (even in the same Windows system) may choose 07/06/2019. So it is not really a reliable way to get the date. It becomes difficult for you to script using date /t command as the output of this command varies with Windows machines with a different regional/country settings.

To check the above, I randomly tested these commands on two different Windows systems.

On Windows Server 2012 R2 server:

On Windows 10 system:

Alright! here is the solution.

In order to overcome this issue, you should use the WMIC localdatetime command to get the date and time as this will work independent of the region setting in your Windows system/server. WMIC is the WMI command-line interface to WMI. WMIC localdatetime command provides you regionally independent date-time parsing.

I tested WMIC localdatetime command on the same set of Windows systems and got the output as shown below where we can see the output format is exactly the same in both:

On Windows Server 2012 R2 server:

On Windows 10 system:

From the above output, you can easily see that this WMIC command output needs some formatting to put the current date and time in more readable format.

Below is the batch script that formats the WMIC command output for you. You can directly use this script to add current/today date and time in your Windows files/folders.

Batch Script: Version-1

@echo off

for /f "delims=" %%a in ('wmic OS Get localdatetime  ^| find "."') do set "dt=%%a"

:: Format the WMIC command output in YY_MM_DD_hr_mn format

set "YY=%dt:~0,4%"

set "MM=%dt:~4,2%"

set "DD=%dt:~6,2%"

set "hr=%dt:~8,2%"

set "mn=%dt:~10,2%"

set "today_date_time=%YY%_%MM%_%DD%_%hr%_%mn%"

echo %today_date_time%

:: Create a folder in the current directory with name as today’s date and time as shown below

mkdir .\%today_date_time%

:: Append today’s date and time in your log file name created in the current directory as shown below

echo This information will be recorded in the following log file >> %today_date_time%_Output.log

Save the above code in a batch file (test.bat) as shown below:

Now let’s run this batch script (either from the command line or directly from the folder itself by double-clicking on test.bat).

Running this batch script will create the following two things:

1. A log file in current directory with name: %today_date_time%_Output.log
2. A folder in current directory with name: %today_date_time%

Batch Script: Version-2

If you want to display the month-name (with the first 3 characters like 'Feb') and not the month-number, use the below batch script code:

@echo off

for /f "delims=" %%a in ('wmic OS Get localdatetime  ^| find "."') do set "dt=%%a"

:: Format the WMIC command output in YY_MM_DD_hr_mn format

set "YY=%dt:~0,4%"

set "MM=%dt:~4,2%"

set "DD=%dt:~6,2%"

set "hr=%dt:~8,2%"

set "mn=%dt:~10,2%"

:: Format the MM (month-number) to display the month-name

if %MM%==01 set MM=Jan

if %MM%==02 set MM=Feb

if %MM%==03 set MM=Mar

if %MM%==04 set MM=Apr

if %MM%==05 set MM=May

if %MM%==06 set MM=Jun

if %MM%==07 set MM=Jul

if %MM%==08 set MM=Aug

if %MM%==09 set MM=Sep

if %MM%==10 set MM=Oct

if %MM%==11 set MM=Nov

if %MM%==12 set MM=Dec

set "today_date_time=%YY%_%MM%_%DD%_%hr%_%mn%"

echo %today_date_time%

:: Create a folder in the current directory with name as today’s date and time as shown below

mkdir .\%today_date_time%

:: Append today’s date and time in your log file name created in the current directory as shown below

echo This information will be recorded in the following log file >> %today_date_time%_Output.log

Running above batch script will generate your output in below format (with month-name in place of month-number compared to the first batch script):

The purpose here is that you can use the above two batch script codes in any of your batch files where you either want to create a backup folder or a log file with the current date and timestamp added in the folder or file name.

WMIC localdatetime command can be run on any Windows platform and the result will be the same. You don’t need to worry about Windows system regional settings and the variation in date and time formats. The returned value of WMIC localdatetime is always in the same format and with time in the 24-hour format.

WMIC localdatetime command works universally on all locales or date-time formats that's why it is the most reliable way to get the date in batch scripting.

That's all for this post.

I hope this article has helped you. 

Your suggestions/feedback are most welcome.

Keep learning and Have a great day!!!

Read More