DataStage tip: using job parameters without losing your mind

A job parameter in the ETL environment is much like a parameter in
other products, it lets you change the way your programs behave at
run-time by tweaking or changing parameters to alter the way the job
behaves. This constant changing can either ease your support burden or
drive your support staff mad.

In this blog we will be looking at
project specific environment variables and how they become happy job
parameters. This is also my first blog with diagrams to show steps.

This
is a happy parameter method because an administrator or support person
can modify the value of a parameter, such as a database password, and
every job in production will run with the new value. Yippee!

If
you have trouble with the pictures below you can read the alternative
text (which isn't nearly as good) or go straight to the photo album.

Objectives
A
job parameter is a way to change a property within a job without having
to alter and recompile it. The trick to job parameters is to manage
them so the values can be easily maintained and there are not too many
manual maintenance steps that can result in incorrect job execution.

Creating Project Specific Environment Variables
Finding
the place to set these variables is a bit of a Minataurs Maze, we can
only hope that under DataStage Hawk we get a short cut to this screen
for the benefit of admin staff, here is the sequence of buttons:



Type
in all the required job parameters that are going to be shared between
jobs, note the passwords are set to a type of encrypted and will only
show up as asterisks here and in any job logs:



*
Database login values are job parameters as they change when you move
from development through to production and database passwords should be
modified on a regular timetable (though some sites never change them!).
* File directories can change when you move from development to production and they can change if new disk partitions are added.
*
DB2 settings such as transaction size can be changed by support to test
different performance settings or to match modifications to the
database setup.

The problem with these settings is that they all
sit in the one folder, this makes them a bit cumberson to administer
and find from within a job. Folder names can be added by hacking the
DSParams file found in the project home directory:



This
looks a lot neater. Note how now only the value field is modifiable,
moving these values into folders prevents the entries from accidentally
being deleted by administrators, they become protected entries. This is
a risky modification if you attempt it directly in production as you
may corrupt your DSParams file, always take a backup. The mofication is
done by adding a folder name into the file entry:
[EnvVarDefns]
DW_DB_INST/User Defined/-1/String//0/Project/DW DB Client Instance

Changed to:
[EnvVarDefns]
DW_DB_INST/User Defined/DB DW/-1/String//0/Project/DW DB Client Instance

The
text in bold shows the hack, the folder name has been inserted into the
second field on that line adding the sub directory "DB DW" under the
folder called "User Defined". I do not think IBM support will be too
thrilled with this hack, would really help if the next version had
folder support built in.

Adding to a job
Now we have
our variables for the project we can add them to our jobs. They are
added via the standard job parameters form via the verbose but slightly
misleading "Add Environment Variable.." button which doesn't add an
environment variable but rather brings an existing environment variable
into your job as a job parameter:



The
job parameters need to be set to the magic word "$PROJDEF" to ensure
they are dynamically set each time the job is run. Think of it like the
DataStage equivelent of abra cadabra. When the job runs and it finds a
job parameter value of $PROJDEF it goes looking for the value within
the DSParams file and sets it. Even the password is set to $PROJDEF,
use cut and paste into the encrypted password entry field instead of
trying to type it to avoid mistakes:



These job properties can now be used in a DB2 database stage:



Low Administration
So
now the job will dynamically pick up job parameter values using
whatever has been entered by the Administrator for that parameter. The
parameter values on the individual jobs never need to be changed.

The View Data Drawback
In
the current version you cannot easily use the database or sequential
file "View Data" option as it fails with the magic word $PROJDEF, it
treats it as a normal word and does not do the magic substitution. It
becomes a pain as you find yourself repeatedly switching the magic
world for real values manually each time you want to View Data. Since I
use View Data all the time in development (fast way to make sure a
stage is setup correctly) I use normal job parameters at the
parallel/server job level and use environment variable job parameters
at the sequence job level.

The sequence job maps project specific environment variables into normal job parameters:



In
this example the database login values required for View Data to work
have been turned into normal job parameters within the parallel job.
The two DB2 properties are still $ parameters within the job because
they do not affect view data. The PROCESS_ID and PROCESS_DATE are both
fully dynamic parameters set by some BASIC code within the Sequence job
User Variables stage. They are set to a counter ID and the current
processing date.

The restriction on this design is that users
should only run jobs via a Sequence job, if they try to run the job
directly the normal job parameters may have incorrect default values.

Disclaimer: The opinions expressed herein are my own personal opinions and do not represent my employer's view in any way.