Maximizing Excel / VBA Automation Performance

Overview

When automating Excel operations with VBA, there are many ‘tricks’ which can be implemented to significantly improve performance. As the automations run in a non-interactive fashion, features that primarily exist for the benefit of an end user can be temporarily disabled resulting in performance gains. The following Excel settings fall into this category:

Excel Application Properties

  • Calculation
  • EnableEvents
  • ScreenUpdating
  • EnableAnimations
  • DisplayStatusBar

Calculation

One of the best features of a spreadsheet is its magical ability to instantly update data contained in the worksheet cells. While these real-time updates are useful when users are interactively working in the Excel document, this feature robs performance and isn’t necessary when executing automation processes. By changing the Application.Calculation property, we can toggle the calculation logic between Automatic, Manual, and SemiAutomatic.

Toggling between Manual and Automatic during your VBA code execution will typically result in significant performance gains.

Figure 1 – Disable Automatic Calculations in Excel

Figure 2 – Enable Automatic Calculations

To illustrate the potential performance difference, I created a simple VBA routine to generate a 22×22 grid of numbers. Formulas are contained on the bottom of the sheet which sum up the numbers in the grid. The numbers on the grid are updated multiple times based on a user supplied value. When this grid is generated with / without automatic calculations enabled / disabled, there is a significant performance impact. *

Figure 3 – Calculation Manual vs Automatic

As the size of the grid/number of calculations required increases, the performance will continue to decline. Temporarily disabling the calculation method will significantly boost performance on sheets that make use of formulas. (e.g. just about every spreadsheet on the planet)

*IMPORTANT NOTE – Performance metrics are going to vary on a machine by machine basis depending on the system’s hardware. As a general rule, these settings will almost always improve performance. The amount of improvement will be inversely proportional to the machine’s hardware. (e.g. the slower the machine, the larger the improvement) Even if the improvement isn’t the greatest on your machine, you should still consider implementing these steps if other users will work with the excel files.

EnableEvents

In Excel, events can trigger actions and/or programming logic to be executed. For instance, when you change a value in a cell, a “changed” event triggers. Events are one of the ways in which Excel “automagically” knows to update formula based cells when the underlying data is altered. Disabling events is just as easy as adjusting the calculation mode:

Figure 4 – Disable / Enable Excel Events

Using the same 22×22 grid of numbers from the previous example, testing was performed contrasting the difference between enabling / disabling events. Disabling events resulted in a 50% improvement in processing time on a modern quad core i7 machine!

ScreenUpdating

The ScreenUpdating setting in Excel is relatively self-explanatory. When this is enabled, the worksheet display area will be refreshed after any cell values are altered. When users are directly interfacing with Excel, this absolutely needs to be enabled; however, when executing scripts, this is not needed. Updating this setting is also straight forward:

Figure 5 – Disable / Enable ScreenUpdating

Using the same 22×22 grid of numbers from the previous example, testing was performed contrasting the difference between enabling / disabling screen updating. Disabling ScreenUpdating resulted in a modest 10% improvement in processing time on a modern quad core i7 machine with a high end graphics card.

EnableAnimations / DisplayStatusBar

The final two items, EnableAnimations and DisplayStatusBar, typically do not result in significant gains; however, they can help if you need to squeeze every possible ounce of performance out of your Excel/VBA.

EnableAnimations is another user interface related performance setting. This setting only disables certain visual elements as opposed to all screen updates. (e.g. user interface animations for certain actions) Typically, this setting will not make a significant impact; however, it is better than nothing in situations where you cannot leverage the ScreenUpdating setting. Like many of the previously discussed settings, this is another On/Off (True/False) setting:

Figure 6 – Disabled / Enable Application Animations

The DisplayStatusBar is yet another user interface setting that may result in minimal gains. Unless your Excel application relies heavily on the status bar, it is not likely this is worth the effort. This is also an On / Off (True / False) setting:

Figure 7 – Disable / Enable Status Bar

 

Additional Tips / Suggestions

When leveraging these settings, keep the following in mind:

  • Cumulative Effects –The performance settings can be combined for cumulative improvements! In the previously discussed grid test, enabling all performance settings results in an amazing improvement. On a modern quad core i7 computer, the test went from 4 seconds to less than 1 second!
  • Global Scope – Be cognizant that these settings are global settings and will persist across workbooks, instances of Excel, and will even persist after you close and restart Excel! Be sure to consider this when designing your solutions.
  • Restoring original settings – While these settings are great for performance, they are not great for end users! If your code fails to restore the original values, end users will be negatively impacted! (End users and IT teams alike may remove you from their Christmas mailing lists!)

As some of the settings, such as Calculation State, are more than an On/Off setting, it is recommended that you read the current setting and store it in a variable to ensure that you can correctly reset the application setting to its original state.

  • Code Robustness – To ensure that the settings are consistently reset, it is imperative that you follow good coding practice and account for errors/exceptions in your code. If you do not implement proper error handling in your code, an end user will eventually run into an unhandled exception resulting in the code ending in an incomplete state. (e.g. your VBA code tries to delete a file and encounters an error, such as ‘permission denied’) The ‘On Error’ statement should be utilized in your code to provide a way for your code to handle errors gracefully.

The attached Excel workbook contains sample code as well as the Sample Grid test which has been referenced in this write-up:

Figure 8 – Excel Performance Demo Application

http://www.charlescbeyer.com/SampleCode/2_PerformanceOptimizations.xlsm

Resolving Frozen HFM Applications

Overview

While Hyperion Financial Management (HFM) applications do not regularly suffer from technical issues, there are certain situations that can cause your application to become unresponsive. This scenario usually is caused by poorly designed or corrupted HFM member list or calculation rules which run in a loop. (e.g. self-referencing member lists) When this happens, the application becomes frozen and even a restart of the servers/application will fail to solve the issue since the rules are executed again on startup.

One way to resolve this issue is to revert to a backup; however, this could result in the loss of artifacts and/or data. Fortunately, there is a better way to solve this problem that will minimize downtime and prevent the loss of information!

The information provided below will cover key details and complete recovery steps using the HFM COMMA application as an example.

HFM Database Concepts (Rule File Storage)

While the HFM relational database contains a plethora of tables, one table (<appname>_BINARYFILES) contains the key to this solution. HFM Rules, Metadata, Member Lists, and other Application Settings are all stored in this table. (see my other post about altering HFM Starting years via this table for more information!) When you use the HFM user interface to import Rules / Member lists / Metadata to the application, the information is placed in the appropriate instance of the table. (e.g. COMMA_BINARYFILES) The LABEL column indicates what type of information is being stored (AppSettings, CalcRules, MemberListRules, SharedAppData, etc.) while the BINARYFILE column contains the data.

Since the BINARYFILE column can only contain 2,000 bytes of data, you will find multiple rows of for a given type. (e.g. Rules files will typically have numerous rows since rule files are typically much larger than 2,000 bytes) When the data needs to be used in the application, it is read in ascending order based on the ENTRYIDX column.

.

While the BINARYFILE data does not appear to be readable, it is simply the ASCII values stored as a string as opposed to the actual human readable characters. The beginning of the 2nd row has been decoded for reference:

 

Resolving HFM App Hang Up

While the HFM database table ‘history lesson’ was fun, how exactly does this solve the problem? Since the HFM application is looking for specific LABEL values for Rules / Member Lists, the easiest way to resolve the issue is to update the LABEL for the relevant rows so that HFM will no longer “see” them as rules / member lists. This will effectively give HFM a clear slate and it will startup as expected. (Before doing anything with the application you should re-load a good set of rules / members though!)

IMPORTANT – You should not make these changes while the application is running! Be sure to stop the application first!

The following steps should be followed to fully recover the application:

  1. ​Stop HFM Services
  2. Connect to the database and execute a query to change the Label value to a dummy value
    • Member Lists – UPDATE <APPNAME>_BINARYFILES SET LABEL = ‘BadMemberListRules’ WHERE LABEL = ‘MemberListRules’
    • HFM Rules – UPDATE <APPNAME>_BINARYFILES SET LABEL = ‘BadCalcRules’ WHERE LABEL = ‘CalcRules’
  3. Restart HFM Services
  4. Connect to Application
  5. Load new Member List or HFM Rules
    • NOTE – If this is an EPMA application or you are using Calculation Manager, be sure to correct the rule issue and redeploy the application

 

Oracle Inventory User Maintenance Scenarios

Overview

The Oracle Inventory is a series of registry entries and files that keep track of all Oracle products / patches that are installed on a machine. As certain portions of Oracle Inventory are stored in a user specific manner, product and patching problems can occur if multiple user accounts are utilized for product installation and/or patching operations. (e.g. User 1 installs Product A, and User 2 attempts to patch Product A)

Common Oracle Inventory / User related scenarios / solutions are:

  1. A user account is used to install software / apply patches and this account is disabled / deleted at a later date – Assuming no additional Oracle software has been installed, this is easy to fix via a combination of file copy / registry updates.
  2. Multiple user accounts have been used to install Oracle Software – In this scenario, the inventory needs to be recreated under one user regardless as to whether the accounts exist or not.

[IMPORTANT – It is strongly recommended to install/patch all software with a non-user specific account, where possible.  While certain security policies may not allow this, using a global account will ensure you do not have to perform the steps in this document.  ]

Prior install account has been disabled / deleted

In this scenario, an account used to install the Oracle EPM Software has been disabled / deleted and we are unable to patch the environment as the Oracle Inventory for the current user does not have knowledge of the prior installed products. The recommended fix is to reassign the previous user account’s Oracle Inventory to the new user account.

To move the Oracle Inventory, perform the following steps:

  1. Identify service account and setup with local administrator access on all Hyperion machines
  2. Copy .oracle.instances file from the user account which was used previously to the new account. In the example below, the user was epm_admin making the path to the file C:\Users\epm_admin\.oracle.instances.NOTE: This file may not be visible depending on your file folder view settings.

  3. Modify Windows Registry Hive & Key values – Replace the old install user with the new one
    1. Rename Computer\HKEY_LOCAL_MACHINE\SOFTWARE\ORACLE\KEY_EpmSystem_<old user>
    2. Update the values for ORACLE_GROUP_NAME & ORACLE_HOME_NAME keys so that they refer to the new user account.

 

  1. Modify file C:\Program Files\Oracle\Inventory\ContentsXML\inventory.xml

Replace any reference to the old install account with the new account

  1. Confirm OPatch software shows inventory properly by:
    1. Opening a Windows Administrator Command Prompt
    2. Navigate to the Oracle\Middleware\EPMSystem11R1\OPatch folder
    3. Execute the Opatch utility:Opatch lsinventory –oh <Install Drive>:\Oracle\Middleware\EPMSystem11R1 –jdk <Install Drive>:\Oracle\Middleware\jdk160_21
    4. Visually verify that the installed patch list corresponds with what has been installed

Oracle products have been installed to multiple accounts

In this scenario, multiple accounts have been used to install Oracle software. Patching attempts may fail or appear to operate successfully, though not all products are truly updated. To resolve this issue, the Oracle Inventory will need to be rebuilt targeting one user account.

NOTE: It is strongly recommended that a generic/service account is used for Oracle installation / patching to prevent these issues.

To rebuild the Oracle Inventory, perform the following steps:

  1. Log onto the server as the account which will contain the Oracle Inventory
  2. Execute the createInventory.bat script located in the <Install Drive>:\Oracle\Middleware\EPMSystem11R1\Opatch

Screen Clipping

  1. From the main menu, click on the Install menuScreen Clipping
  2. Scroll through the contents list to confirm that all installed products are reflectedScreen Clipping
  3. Monitor the Installation progress.NOTE: This may run for a while, > 10 minutes

    Screen Clipping

  4. On the Specify Source Location screen, provide the location of the products.xml file.NOTE: Should be <Install Drive>:\Oracle\Middleware\EPMSystem11R1\common\epmstage\Disk1\stage\products.xml

    Screen Clipping

  5. Specify the Oracle Inventory Home DetailsNOTE: By default this is OUIHome / C:\OraHome, use OUIHome1 & C:\OraHome1

    Screen Clipping

  6. Scroll through the provided list and ensure your products are shown and then click NextScreen Clipping
  7. Review items on the summary screen and click InstallScreen Clipping
  8. Upon completion of steps above, re-run the Opatch utility to confirm installed products now appearScreen Clipping

Oracle EPM / Windows Prerequisite Tuning [Part 1]

If you are installing Oracle EPM in a Windows environment, there are a plethora of settings you need to tweak per Oracle guidance.  To greatly simplify this process, I made a script that performs all of the steps for you, saving you a pile of time / effort.  There are a few steps that you’ll need to do manually; however, this will allow you to perform these steps in a fraction of the time *and* ensure that your servers are configured in a consistent manner.

NOTE:  Part 2 of this post will actually go into what these settings do.  For now, just enjoy a free script.

http://charlescbeyer.com/Infra_Win_Prereq.txt

@echo off

@REM ----------------------------------------------------------------------------------------------------
@REM - Name: Infra_Win_Prereq.bat
@REM - Author : Charles Beyer
@REM - Creation Date : 11/2/2015
@REM - Last Modified DAte : 7/17/2017
@REM - DESC: The purpose of this script is to configure Oracle required default settings for Oracle
@REM -       Hyperion Enterprise Performance Management products running on Windows 2012, 2008, 2003
@REM -
@REM - Inputs :
@REM            %username% - Windows system variable containing the currently logged on user
@REM            %COMPUTERNAME% - Windows system variable containing the current machine name
@REM
@REM - Outputs : Command line success/fail responses
@REM -
@REM - NOTES: There are a few steps that cannot be performed via command line
@REM -         Those items are in the script and will be called out.
@REM -         User should perform the manually
@REM -         The logged on user while running this script should be the EPM Service Accout as some of the
@REM -         config items are a "per user" setting.
@REM -
@REM - Items Configured in this script are:
@REM -       DHCP Check (Should be using Static IPs)
@REM -       Time Sync w/ Domain Controller
@REM -       UAC Disable
@REM -       Local Security Policy Permission Settings
@REM -           (*NOTE* You need ntrights.exe for these steps. Available on Windows Resource CD of online download)
@REM -       Disable DEP
@REM -       Windows Firewall Disable
@REM -       Group Policy Updates (Disable force registry unload)
@REM -       Windows Power - Set to Max performance
@REM -       Adjust Advanced Display Settings to 'Best Performance'
@REM -       TCP Tuning Settings (Part 1)
@REM -       Worker Threads
@REM -       Disable Delete TEMP on logoff / Disable 'per user' temp folders
@REM -       TCP Tuning (Part 2)
@REM -       Remove TMP / TEMP User Environment variables
@REM -       Disable Anti-Virus reminder (I can't disable anti-virus software, just display message to add exceptions, etc.)
@REM -       Disable Internet Explorer Enhanced Security

@REM -----------------------------------------------------------------------------------------------------

@REM -------------------------- DHCP Check --------------------------------------
@REM Check for any Dynamic IP addresses as we should be used STATIC
@ECHO Checking for Dynamic IPs.
@ECHO If DHCP is shown as enabled, please investigate further.
@ECHO Static IPs are expected.
ipconfig /all | find "DHCP Enabled."

@REM Add space between command output
@ECHO.

@REM -------------------------- Time Sync w/ Domain ---------------------------------
@REM Sync time to Domain Server
@ECHO Configuring Date/Time to be synchronized with Domain Controller
w32tm /config /syncfromflags:domhier /update
net stop w32time
net start w32time

@REM Add space between command output
@ECHO.

@REM -------------------------- UAC Disable ---------------------------------
@REM Disable UAC
@ECHO Disabling UAC
reg add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f

@REM Add space between command output
@ECHO.

@REM -------------------- Local Security Policy Permissions --------------------
@ECHO Please review Location Security Policy Permissions and assign the service/install account
@ECHO 'Adding Logon as service' right
ntrights +r SeServiceLogonRight -u %username% -m \\%COMPUTERNAME%

@ECHO Adding 'Logon as batch job' right
ntrights +r SeBatchLogonRight -u %username% -m \\%COMPUTERNAME%

@ECHO Adding 'Act as part of the OS' right
ntrights +r SeTcbPrivilege -u %username% -m \\%COMPUTERNAME%

@ECHO Adding 'Bypass Traverse Checking'
ntrights +r SeChangeNotifyPrivilege -u %username% -m \\%COMPUTERNAME%

@REM Add space between command output
@ECHO.

@REM -------------------- Disable DEP --------------------
@REM Disable Data Execution Prevention (DEP)
@ECHO Disabling Data Execution Prevention (DEP)
bcdedit /set nx AlwaysOff

@REM Add space between command output
@ECHO.

@REM -------------------- Disable Firewall --------------------
@REM Disable Firewall
@ECHO Disabling Firewall for EPM
netsh advfirewall set currentprofile state off

@REM Add space between command output
@ECHO.

@REM -------------------- Upgrade Group Policy --------------------
@REM Update Group policy to Disable force unloaded of user registry upon logoff
@ECHO Updating Group policy to Disable force unload of user registry upon logoff
reg add "HKLM\SOFTWARE\Policies\Microsoft\Windows\System" /v DisableForceUnload /t REG_DWORD /d 1 /f

@REM Add space between command output
@ECHO.

@REM -------------------- Set Power Option to High Perf --------------------
@REM Set Power Option to High Performance
@ECHO Setting Power Option to High Performance
powercfg /S SCHEME_MIN

@REM Add space between command output
@ECHO.

@REM -------------- Update Adv Settings for Best Performance --------------
@REM Update Advanced Settings / Adjust for Best Performance (UI)
@ECHO Updating Advanced Settings / Adjust for Best Performance (UI)

reg add "HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\VisualEffects" /v VisualFXSetting /t REG_DWORD /d 2 /f

@REM Add space between command output
@ECHO.

@REM ------------- Registry Updates (TCP / Session Mgr / Term Svr) -------------
@REM Add Windows Registry settings for TCP Performance, Additional CPU Thread settings, Disable Delete Temp Directories on Exit
@ECHO Updating Registry settings for TCP, CPU, Temp/User
reg add "HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters" /v TcpTimedWaitDelay /t REG_DWORD /d 30 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Executive" /v AdditionalCriticalWorkerThreads /t REG_DWORD /d 64 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Executive" /v AdditionalDelayedWorkerThreads /t REG_DWORD /d 20 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v DeleteTempDirsOnExit /t REG_DWORD /d 0 /f
reg add "HKLM\SYSTEM\CurrentControlSet\Control\Terminal Server" /v PerSessionTempDir /t REG_DWORD /d 0 /f

@REM Add space between command output
@ECHO.

@REM ------------------- Network Performance Settings -------------------
@REM TCP Performance Settings
@ECHO Apply TCP Performance Settings

netsh int tcp set global autotuninglevel=disabled
netsh int tcp set global chimney=disabled
netsh int ipv4 set dynamicport tcp start=1025 num=64000
netsh int ipv4 set dynamicport udp start=1025 num=64000
netsh int ipv6 set dynamicport tcp start=1025 num=64000
netsh int ipv6 set dynamicport udp start=1025 num=64000

@REM Add space between command output
@ECHO.

@REM ------------------- Remove TEMP USER Env Variables -------------------
@REM Delete User TEMP variables
@ECHO Deleting Current User defined TEMP/TMP Environment Variables
@ECHO NOTE: You may see ERROR / Unable to find key messages. These indicate the
@ECHO TMP / TEMP environment variables have not been set for the current user.
reg delete HKCU\Environment /F /V TEMP
reg delete HKCU\Environment /F /V TMP

@REM Add space between command output
@ECHO.

@REM ----------------- Display Virus Exception Reminder ----------------
@REM Virus software warning
@ECHO IF virus scanning is active on this machine, be sure to do the following:
@ECHO - Disable Scanning on Hyperion servers during software installation
@ECHO - Enable Scanning after install, but exclude the \Oracle folders.

@ECHO.
@ECHO All steps complete, please review for any errors.

@REM ----------------- Disable IE Enhanced Security ----------------------
@REM This disables the IE enhanced security

@REM Disable for users (0 = disabled, 1 = enabled)
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A7-37EF-4b3f-8CFC-4F3A74704073}" /v IsInstalled /t REG_DWORD /d 0 /f

@REM - Disable for Admin Users (0 = disabled, 1 = enabled)
reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Active Setup\Installed Components\{A509B1A8-37EF-4b3f-8CFC-4F3A74704073}" /v IsInstalled /t REG_DWORD /d 0 /f

Security Devices, only as good as their implementation…

Security Devices, only as good as their implementation…

Recently, I needed to use an old program that is protected by a security device. The device, an M Activator hardware key, connects to your computer’s parallel port.

C:\Users\beyerch\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Outlook\K9R4HEY5\IMG_9355.JPG C:\Users\beyerch\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Outlook\K9R4HEY5\IMG_9356.JPG

Figure 1 – M Activator Security Key

If the security device is not attached to the PC, the application program will restrict your access to certain application functions or prevent you from using an application altogether.

C:\Users\beyerch\AppData\Local\Temp\SNAGHTML1dd2ee31.PNG

Figure 2 – Application Rejection due to no hardware key

Since I own the software and still have the security key, none of this should be a problem. Unfortunately, modern computers no longer have parallel ports! As the software isn’t maintained, I can’t call the original provider for an alternative leaving me with few choices. The first, and preferred, choice was to purchase a parallel port to USB adapter on-line. I purchased two highly rated units; however, the software failed to recognize the dongle when connected through either of the units.

C:\Users\beyerch\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.Outlook\K9R4HEY5\IMG_9358.JPG

Figure 3 – Parallel to USB Adapters (that didn’t work..)

As the USB adapter routed was unsuccessful, my remaining option is to …. hack the security key or its implementation in the application program..

A Ridiculously Brief Discussion on Security/Hacking

The first rule of hacking is that you don’t talk about hacking. Wait, or is that the first rule of Fight Club? The first rule of hacking is to accept the fact that nothing will be 100% secure.

When a product is developed, the security implementation is typically driven many factors such as:

  • What is the risk/damage of being compromised?
  • How likely is it that the product will be attacked?
  • What impacts to the development process will occur due to security?
  • How will the timeline be impacted?
  • How will users of the product be impacted?
  • Does the development team understand and have security experience?
  • How much will can we afford????

Because of all of the competing considerations, product security typically looks more like the Griswold family truckster than the shiny red Ferrari.

Figure 4 – Typical Security vs Assumed Security

From the hacker’s perspective, product security really boils down to how badly they want it. Do they have the time, resources, team/skills, and money to dedicate to their mission.

In the case of my ancient dongle security application, I’m willing to invest about 60 minutes into seeing if I can get anywhere. After that, I dig out one of my older computers and use it with this program. (and hope it doesn’t ever die…..)

With that said, let’s see just how secure this old dongle application is…

Hacking 101

Now that we’ve decided that we’re going to take a stab at working around the security device, the first thing we need to do is gather information about our target. Before we can formulate a plan, we need to know what we’re up against. After about 5 minutes of research, we know the following about our target application/security device:

  • Application Program
    • Windows 32 bit executable
    • Written in C++
    • Program appears to leverage multiple external libraries, some of which are known/some are not
      • ZIP/PKZIP – File Compression
      • W32SSI.dll/.lib – ? Not sure. (yet)
  • M Activator Green Key
    • Made by Sentinel
    • The W32SSI files are related to this dongle

NOTE: Researching this scenario finds a lot of “hits” to people with similar scenarios. There are emulators and other products made to solve this problem; however, I’d rather try to figure it out myself first.

Given what has been found, it seems likely that the application program is going to use the W32SSI files to talk to the dongle. Depending on how this is done in the application, we may be able to update the application program and simply bypass the dongle. All we need to do is take a peek at the application software to see what is going on, no biggie.

Source Code, Assembly Code, Machine Code, Oh My!

If this were our application program, we could simply open it in our editor, make our desired changes to the source code, recompile the code, and be on our way. Since we didn’t write this program and the original company is no longer in existence, this isn’t an option. While we could look at the executable binary (e.g. Machine Code) unless you have a photographic memory, know low-level Windows modules by heart, and Intel OpCodes like the back of your hand, it’s going to be impossible to directly analyze the chain of files.

Figure 5 – Machine Code, no problem…..

While it might be cool to rattle off machine code instructions on trivia night, it would take us forever to try and analyze an application in this manner. Fortunately, there are many programs that we can leverage which will translate the machine code into something slightly easier to deal with, assembly code.

Figure 6 – Assembly Code

While assembly code is not nearly as friendly as actual source code, it is a 1 to 1 representation of the machine code in a somewhat human readable format. If you have an appropriate tool, such as the IDA Pro disassembler, you can convert the machine code into the assembly. This tool also allows us to map out the program flow and find text and object file references.

Using the IDA Interactive Disassembler

As mentioned previously, we can use IDA to do a quick search to see if our security device program is called. Since we know that the program uses the security key, we should be able to find one or more references to the W32SSI library files. Depending on how many and what type of references we find, we may be able to easily alter the program so that we can bypass the security hardware.

After opening the program in IDA, we can easily see that the W32SSI libraries are being used by checking the Imports section of IDA.

Figure 7- IDA Imports

In addition to verifying the presence of the libraries via the Imports screen, we can use the Functions / IDA view to find the code references:

Figure 8 – Locating code references to W32SSI

Somewhat surprisingly, the only two functions imported from the security program are referenced once!

Figure 9 – Code section using W32SSI functions

While we do not know what those routines do entirely, since they are only called once, it is safe to assume that they attempt to validate that a security key, of the right type, is connected. To help understand what we’re seeing, we can use the Graph View feature to get a visual representation of the code:

Figure 10 – Graph View of W32SSI logic

Looking at the Graph View of the code leveraging the W32SSI routines, we see that there are two main code branches. The branch on the left performs secondary checks and ultimate ends up with failure messages relating to a security key not being found. The code branch on the right simply returns a value of 1, which presumably is a “TRUE” response.

The Quick and Easy Fix

Looking at the code structure, it appears that the second W32SSI call is performing a check as to whether the security dongle is present or not. If the security dongle is found, a “TRUE” (1) is returned; otherwise, secondary tests are performed. (e.g. serial port instead of LPT, etc.)

Because of this, there appears to be a very easy way to “fix” the program. If we force the initial check to always return TRUE (or flip flop the PASS / FAIL check) then the application program will behave as if the key was present.

The following logic needs to be tweaked from:

call wSSIMIni
cmp eax, 0FFFFFFFFh
jz loc_409FBA

to:

call wSSIMIni
cmp eax, 0FFFFFFFFh
jnz loc_409FBA

JZ and JNZ are machine code instructions that are used in conjunction with comparison checks. If the result of a compare (CMP) instruction is ZERO, a Jump if Zero (JZ) instruction will result in a jump to another portion of the application. Jump if Not Zero (JNZ), on the other hand, results in a jump if the compare (CMP) instruction is non-zero.

To make the change, switch to the Hex View, right click on the highlight value and change the 84 to 85.

Figure 11 – Switching JZ to JNZ

After committing the change, you will see the code switch from

to

After starting the program, we no longer receive an error about the missing security key and the program operates as expected.

Well That Easy…..

While it may be hard to believe that changing one byte of data, by one digit, entirely bypassed an application’s security, this is a surprisingly common scenario. The security dongle used by this application could have been utilized much differently preventing this type of scenario, though. (e.g. the dongle could have stored a required piece of information that the application would need to operated properly)

FDMEE Essbase/Planning Script Execution Glitch

FDMEE Essbase/Planning Script Execution Glitch

When targeting Essbase/Planning applications through FDMEE, a particularly useful feature is the ability to trigger Calculation scripts before and after the Load process as well as before and after the Check process. Not only can you execute scripts, but you can control the execution order and pass script parameters. This functionality is quite useful for executing fixed scope data clear operations before the data load and executing targeting aggregation/calculations after the data load has been completed.

Figure 1 – FDMEE Target Application Calculation Script Editor

While this feature works great when the scripts are working, what happens when there is a script failure? If a script executed during the Load process fails, should the FDMEE Load step report a failure? (even if the data loaded?) How about a script failure during the Check step? Would you be surprised to know that currently in 11.1.2.4.210, this is not the case?

In the event of a successful load process, even if a script error occurred, all FDMEE “fish” steps will return gold and the Process Monitor report will reflect the same. (e.g. no issues) Your only indication of a failure will be in the job log for the data load process!

If you are currently leveraging this functionality, please be aware of this quirk until this is corrected!

NOTE(s):

  • 11.1.2.4.100 (Patch 20648390) advertises this as being fixed; however, it still presents itself in 11.1.2.4.200+ [See Defect: 20631385]
  • An enhancement request, 25217240, was created for this issue when submitted in 11.1.2.4.200. I do not know if this has been implemented yet; however, no fix is listed in the Defects Fixed Finders Tool through 11.1.2.4.210

Steps to Recreate [target a non-existent script]

#1 – Create FDM Target Planning Application and associate a Calculation script that does not exist in the target application


Figure 2 – Create Calculation Script references for our sample Target Application

#2 – Create your Location / Data Load Maps / Data Load Rule / etc.

#3 – Load a data file through and confirm that the Process Monitor reflects success.


Figure 3 – Perform a data load and confirm that the Process Monitor User Interface shows no errors

#4 – Pull Process Monitor report and confirm no errors reflected.


Figure 4 – Run the Process Monitor Report to verify it also does not reflect any errors

#5 – Review the log file

Since the script doesn’t exist, the FDMEE log will reflect an Essbase error due to the non-existent script.


Figure 5 – Review the Job log to verify an error is reflected in the logs

Version Information:


Figure 6 – Confirm version of FDMEE

Workaround

If your load process is being performed manually, the easiest recommendation is to have users review the log to confirm successful script execution.

If you are performing automated processing where it is not feasible to manually review logs, considering implementing an Event script to scan the log file for script success/failure and using that to trigger a failure in FDMEE / Error Log / Email Notification / etc.

PBCS Data Loading Glitch

PBCS Data Loading Glitch

While building out data load automations for PBCS and FCCS recently, I ran into a somewhat annoying issue when loading data to PBCS. Even more surprising is that Oracle claims it works as expected….. This post will explain the issue and provide a really simple work around until this gets properly addressed.

The Problem

Anyone who has worked with FDM Classic, FDMEE, or Data Management for any amount of time has run into at least one failed data load due to invalid dimension member(s) in their exported data. While the data load process should ensure that the target application metadata is current before loading data, this doesn’t always happen. (e.g. last minute source ERP updates that do not get properly communicated to the EPM team)

While a data load failure isn’t optimal, typically this failure is easy enough to identify and fix based off of the feedback provided from FDM/FDMEE/Data Management and the target application.

FDMEE / PBCS Merge Load Failure

FDMEE / Planning Merge Load Failure

cid:image010.png@01D1E6E3.8124EAE0

Data Management / FCCS Merge Load Failure

Unfortunately, when you perform a Replace export to a PBCS target application, under some circumstances you will get a quite unhelpful response:

Data Management / PBCS Replace Load Failure

What Happened?

When performing a Merge export, FDMEE/Data Management simply passes the generated data file to the target application for loading without any pre-processing. When performing a Replace export, a data clear script is dynamically generated based on the members contained in the export data.

The script will clear data for every intersection of Scenario, Year, Period, Entity, and Value dimensions. In the event that one of those members is missing in the target application, the dynamic clear script will contain an invalid member resulting in the error shown above. While the error shown above is completely legitimate, it isn’t meaningful enough to allow you to locate the member easily.

Oddly, if you recreate this scenario when loading to FCCS, you will have a much different result! When performing a replace load to FCCS, invalid members will result in meaningful errors that include the invalid member name(s)! Even more odd, Oracle Support will tell you that both are working as designed. It would seem that PBCS should behave similarly to FCCS to simplify troubleshooting efforts. (Perhaps if enough people raise this as an issue, they will address it?)

Work Around

The easiest way to work around this issue is to perform two data exports to PBCS for Replace operations. The first export should be a Merge load. As the dynamic clear script will not be generated for a Merge, this will result allow you to receive specific errors in the event that there is a data load failure. After have a successful Merge load, then perform the Replace load to ensure that all data gets cleared, etc.

Job Output from Merge/Replace Load Automation Process

Oracle Hyperion Planning Application Migration Fun!

EPMA Planning Application w/ Workforce Module First Time Deployment (Post Migration)

If you have one or more Workforce enabled Planning applications and you have multiple EPM environments, you may be in for a surprise the first time you migrate those applications to another one of your environments!

After LCM’ing in your EPMA application definition and performing the initial Application deployment, you may be greeted with the following error:

C:\Users\cbeye002\AppData\Local\Temp\SNAGHTML2568c15f.PNG

Since the application works perfectly fine in the source environment *and* Planning supports shared members in the Time Period dimension, you would be correct to suspect ‘shenanigans’ are at play. The trick is that when the Workforce Planning module is first initialized, there cannot be shared members in this dimension! Once the module is initialized, you can add duplicate member instances. Unfortunately, when you copy an application to a new environment and deploy it for the first time, part of this process includes module initialization!

To work around this issue, keep the default Time Period members and remove any duplicates, deploy the application for the first time, use LCM to restore the full Time Period dimension, and then perform another deployment.

Remove shared members. (Use SHIFT+click to highlight multiple)

Click Yes on the confirmation window that appears after clicking on Remove Member.

Return to application library and attempt the deployment again.

Confirm that this Deployment completes without any errors.

Return to Shared Services and reimport the Period dimension for the application via the EPMA portion of the LCM file

After the LCM import completes successfully, return to Application Library. Since we changed the Period dimension back to its original state in EPMA, this will signal that the application is Out of Sync and will need to be deployed again.

Perform another deployment and confirm that the Status of the application is

 

NOTE:  This issue appears to impact Planning through 11.1.2.4.

Oracle Hyperion EPM Environment Branding Made Easy!

Oracle EPM Branding Made Easy

For IIS and OHS

Overview

A common pain point when working in multiple EPM environments is ensuring that you are working in the right one.  “Out of the box”, each environment visually looks exactly the same.  As no one wants to be the person that accidentally makes a change in the wrong environment, people have tried all sorts of ways to remedy this issue.

Typically, people physically swap out image files for the individual web applications; however, the solutions are problematic for multiple reasons:

  • require manual file system changes
  • require rework since patching / reinstallation / reconfiguration will wipe out the changes
  • updating requires modification to Java WAR/EAR files which could lead to unintended issues
  • does not work properly in all versions of EPM due to content-length limitations
  • does not scale well if branding multiple products

The solution below resolves all of these issues by intercepting image requests at the webserver level via URL Rewriting.   Utilizing URL Rewriting allows us to:

  • use a small number of images, in one central location, for multiple products
  • easily scale and allow for multiple branding options
  • significantly reduces the likelihood of our branding changes being lost due to patching, redeployment, installation, configuration
  • avoids manual manipulation of EPM application files..

The following walk-through will show you have to use URL rewriting to replace the Oracle logo contained in the upper corner of most EPM applications with one shared image.

NOTE: URL Rewriting could also be leveraged for other uses cases such as globally redirecting all EPM users to a maintenance page while allowing admins to access the system via a special URL.

Oracle logo swapping

There are a few images that lend themselves nicely to branding replacement as the images are used globally among all the EPM products. The red Oracle logo (oracleLogo.png) is one such example. The oracleLogo.png file appears in the title bar on many of the pages, such as the initial log on page:

A quick search for this file, on a webserver running most EPM products, reveals how common it is:

As the same file is virtually used in every EPM product, the best way to replace this image is through URL Rewriting. URL Rewriting instructs the web server to replace requests for a given URL with a different URL of our choosing.

For instance, if a user requests http://EPM.COM/EPMA/oracleLogo.png, we can tell the webserver to re-reroute that request to http://EPM.COM/MyCentralLocation/myCoolerLogo.png. Since this functionality supports regular expressions, we can create one rule to replace requests for almost all of the EPM products. (DRM & FDM need additional rules)

Implementing URL Rewriting on IIS

To implement URL Rewriting for the oracleLogo.png file, perform the following steps:
[NOTE: The screen shots below depict IIS 7/7.5; however, this process works for all currently supported versions of IIS]

  1. Create a replacement oracleLogo.png file. As the original file has a height of 25 pixels and a width of 119 pixels, it is imperative that your image is the same size. If you attempt to use an image with a different size, it will be scaled to fit and it may not look how you want it to.Sample images are shown below.

    (NOTE: We will use the QA file for the rest of the IIS walkthrough)
  2. Copy your replacement logo to a location accessible by the web server and the end users. (HINT: The IIS WWWROOT folder is typically available to all users and is a good common spot)
  3. Confirm that you can access this file via Web Browser
  4. Confirm that IIS Rewrite is installed on the Web Server.
    (If it is not, follow the steps in Appendix A)
  5. Start Internet Information Services (IIS) Manager
  6. In the connections panel (on the left), expand the Server, Sites, and then Default Web Site.
  7. In the right window, click on Features View and then double click on the URL Rewrite button.
  8. In the Actions panel (on the right), click on Add Rule(s)
  9. Click on Blank rule
  10. Complete the Inbound Rule Screen as follows
    1. Name: oracleLogo Replace
    2. Match URL
      1. Requested URL: Matches Pattern
      2. Using: Regular Expressions
      3. Pattern: (.*)/oracleLogo.png
      4. Ignore Case: [Checked]
    3. ConditionsSkip, No changes required.
    4. Server Variables Skip, No changes required.
    5. Action
      1. Action Type: Redirect
      2. Redirect URL: http://<Web Server Name Here>/oracleLogo_qa.png
      3. Append query string: Checked
      4. Redirect type: 302 Found

  1. Click Apply
  2. Confirm changes were saved successfully
  3. Test a page
  4. For FDM add a rule as follows:
    1. Name: FDM Logo
    2. Match URL
      1. Requested URL: Matches Pattern
      2. Using: Regular Expressions
      3. Pattern: (.*)/logo.gif
      4. Ignore Case: [Checked]
    3. ConditionsSkip, No changes required.
    4. Server Variables Skip, No changes required.
    5. Action
      1. Action Type: Redirect
      2. Redirect URL: http://<Web Server Name Here>/logo_qa.gif
      3. Append query string: Checked
      4. Redirect type: 302 Found

 


 

Implementing URL Rewriting on OHS

URL Rewriting in OHS is relatively simple as the capability is activated out of the box in the version that is installed with EPM products. To redirect oracleLogo.png in OHS, perform the following steps:

  1. Copy your replace image to the OHS Root folder.
    cp /oracleLogo-TRN.png /Oracle/Middleware/user_projects/epmsystem1/httpConfig/ohs/config/OHS/ohs_component/htdocs/
  2. Update the epm_rewrite_rules.conf configuration file for the redirect actions
    NOTE: For each file above, create a RedirectMatch entry similar to below:RedirectMatch (.*)\oracleLogo.png$ https://<ServerNameHere>/oracleLogo-TRN.png

  1. Restart OHS
    cid:image004.png@01CFB70C.BCE74F40
    cid:image005.png@01CFB70C.BCE74F40
  2. Open a Web Browser (after clearing all caches / temporary files) to confirm update has taken place

Appendix A – Install URL Rewrite on IIS

If your IIS server does not already have URL Rewrite installed, perform the following steps to acquire / install it from Microsoft.

  • Download URL Rewrite

 

HFM – Removing Years

One of the most important steps during the creation of your application creation is determining how many years should exist in the application.  Specifying too many years could result in having wasted space and poor application performance while too few years could result in running out of room in your application when you hit the upper limit.

HFM Application Profile Editor (Pre-11.1.2.4 shown, though same applies to 11.1.2.4)

When HFM first shipped, you had no means of updating the Start Year or the End Year of your application; therefore, a mistake during the application creation process could lead to major headaches in the future.  As customer’s HFM applications matured and those upper year limits were getting close, Hyperion/Oracle rolled out the capability to add more years to the applications.  (with 11.1.2.4, there is a built in function to do this)

When it comes to removing years from the application, you can clear data from years; however, there is not a way to actually remove the years in the application.  Unfortunately, Oracle has never released guidance or functionality for altering the Start Year and will tell you that it isn’t possible to change this.  I’m here to tell you that, you can absolutely change this!  Before we get into how to make those updates, some quick words on why you’d want to do this and some considerations around it.

Reasons to remove years from your application:

  • Database has grown significantly due to continued use of the same application over many years
  • Metadata is cluttered due to the need to ensure historical data remains consistent
  • Prior years are no longer used for reporting  and are not necessary
  • Minimize the years displayed in the U/I to only valid years
  • Reduce the number of database objects (data tables may exist for each year/period/scenario combination)

Considerations when removing years from your application:

  • Beginning Balances for Balance Sheet accounts – (e.g. If our Start Year changes to 2015, we will still need prior year ending balances)
  • HFM Rules/Calc Mgr – (e.g. ensure that rules are not impacted by beginning year change

Process to change the Start Year:

  • “Easy Steps”
    • Make a backup.  (LCM / Database Backup / etc.)
    • Create an archive application – Consider creating an archive HFM application for the historical data.  Storing the information in a separate application gives you the ability to still view the data if needed without having to make metadata trade-offs in your day to day/primary application.  Use HFM Application Copy Utility or LCM/Application Snapshot to accomplish this.  (If you already have an archive app, you can export/import data, though Journals may be slightly annoying in this instance)
    • Update Beginning Year variables / values in the application rules / data for the new Start Year
    • Stop the HFM services
  • “Not so Easy Steps”  (explained later in more detail, don’t panic yet)
    • Connect to your HFM database via your database tool of choice  (e.g. SQL Server Management Studio, SQLPlus, etc.)
    • Query the <APP_NAME>_BINARYFILES table by ENTRYIDX for Labels of type SharedAppData
    • Update bytes 74 & 75 of the first row to reflect your new Start Year
  • Start HFM Services
  • Verify application data / functionality
  • “Post Validation Cleanup”
    • Stop HFM Services
    • Remove Year / Period specific database tables (e.g. <APP_NAME>_DCE_<PeriodNum>_<Year> )

HFM Binary Files Table

For each HFM application, you will find a Binary Files table in the format <APP_NAME>_BINARYFILES.  This table is used for multiple purposes, including storing application configuration information, application metadata, HFM rules, memberlists, etc.  The LABEL column defines the type of file the data is related to.  The BINARYFILE column contains the raw information from the corresponding file.  (e.g. Application Profile .PER file, Rules .RUL file, Metadata, etc.)  Since this column has a character limit that is smaller than most of these files, you will find multiple rows for each file type.  The ENTRYIDX column is used to order the rows in the proper order.  (Ascending order)

As the Start and End years are defined in the profile, we only want to view rows containing a LABEL of SharedAppData:

SELECT
 ENTRYIDX, LABEL, BINARYFILE
FROM
 &lt;APP_NAME&gt;_BINARYFILES
WHERE
 LABEL = 'SharedAppData'
ORDER BY
 ENTRYIDX ASC

 

SharedAppData 

SharedAppData ENTRYIDX 1 as viewed in a Hex Editor

As mentioned above, the SharedAppData rows correspond to the application profile (.PER) that defines the HFM application.  Analysis of this data indicates that both the Start and End years are stored in the first row of this information.  The Start Year is stored in bytes 74 and 75 while, the End Year is stored in bytes 78 and 79.  The data is stored in Least Significant Byte Order.  (e.g. Byte 75 is first half of year, 74 is the second)

In the Shared App Data above, the Start Year is 2006 while the End Year is 2015.

Start Year (Hexadecimal) = D6 07 = 07 D6 (flipped) = 2006 (decimal)
End Year (Hexadecimal) = DF 07 = 07 DF (flipped) = 2015 (decimal)

To change the Start Year, convert the decimal year to hexadecimal and then reverse the byte order.  For instance, if your new start year is 1999, you would end up with:  1999 (dec) = 07CF = CE 07

HINT:  You can use Windows built-in Calculator program in Programmer mode to make these conversions.

Update SharedAppData

After you’ve calculated your new Start Year value, create an Update query to alter the stored binary data

UPDATE query to alter Start Year

 

Verify Results

After you have updated your settings and restarted HFM, you can verify that you now have the appropriate years

Original Years

 

Updated Years

NOTE – While the screen shots are from an HFM 11.1.2.1 application, this has been tested through 11.1.2.4, though I won’t guarantee this works with every patch version, etc.  Always test in a non-production environment.