Last week, I read this article by Craig Mullins: http://www.sswug.org/articlesection/default.aspx?TargetID=70585. Go read it, this blog entry will still be here when you get back.

I have to say that in large part, I agree. I don’t like having differing naming standards for tables and views. And certainly not for aliases.

I do like to have a naming standard for constraints and indexes though. For constraints, I’m never going to change one constraint to another – a primary key to a foreign key for example. So I like to include PK and the table name in the constraint name for primary keys, and FK and the table name in the constraint name for foreign keys. This makes troubleshooting easier, because when an insert or update fails due to a FK violation, many applications will give the foreign key name. I like it when that name means something to me. And again, I’m not going to change the meaning of an existing FK constraint, so there’s no reason not to include that information in my mind.

I get Craig’s contention about keeping the meta data out of the object name, but with indexes, there’s very little that can be done to alter the index – significant alterations require dropping and re-creating anyway, where re-naming is easy. I agree in not including PK or FK in an index name, as the primary or foreign keys can change without regards to the index name. But I do like to have ‘IX’ in there for indexes and a ‘U’ if it’s a unique index. I’m not going to change an index from unique to non-unique without the opportunity to rename it. One of the reasons I have IX in there for my index names is to differentiate them from vendor indexes or to meet with vendor naming standards.

In IBM Websphere Commerce databases, there are very specific naming standards that custom objects should conform to. They are documented on a page in the IBM WebSphere Commerce Information Center.

Like it or not, if you have a vended database, you have to conform to vendor standards. Using those standards and a knowledge of IBM WebSphere Commerce databases, I have a pretty good idea just from the index name whether it is a primary key index, a base vendor index or a custom added index, which can be useful when reading explain plans.

I also have my own naming standards for both DB2 instances and databases. Instances, I like to have very generic names because I see them re-used for various purposes in both test and production environments. But databases themselves I like to have identifiers in for the app (WCS, Sterling, WebSphere Portal or whatever), for the client (several numbers or letters) and an identifier for the environment (Prod, stage, QA, dev, etc). That’s a lot to fit into 8 characters, but I find having these really makes a difference in preventing busy dbas from accidentally doing something to the wrong client, environment, or database. I also tend to have a database nearly exclusively dedicated to one application – a luxury that I know not everyone has.

A pet peeve of mine is also in column naming within tables. I get so frustrated when developers come to me with a table with columns named “field1″, “field2″, and “field3″. In today’s RDBMS it’s not hard to add a column later if needed, and I get a lot of developers submitting tables to me with these “just in case I need them” columns. Each field should have a descriptive name, and columns should not exist just in case they are needed at some future time – talk about a waste of space and a potential performance issue if all of those fields are suddenly populated and create overflow records.

And of course, I strongly object to anything named in mixed case or named with a space in it in a DB2 database. Those make scripting maintenance difficult. Most annoying thing I’ve seen along these lines is a Tivoli database that I supported for a while that had a trailing space on the schema name. That was insanely difficult to deal with in my runstats and reorg scripts.

So what naming standards do you have? What are your naming standard pet peeves?

Transaction log files are one of the core strengths of a DBMS, if you ask me. The ‘D’ in ACID refers to ‘Durability’. What Durability means is that once a transaction is committed, it is not lost, even if the database crashes or the server it is on has to be restarted.

Transaction Logging in DB2

DB2 writes transactions to the tranacation log files while they are still in progress. Every logged action is written to the log buffer. The log buffer is then written out to disk whenever either it becomes full or a transaction is committed (or a couple of other special situations). With many transactions in progress, this means that the active transaction logs on disk contain both committed and uncommitted changes. Part of the information in the transaction logs is what transaction various changes are a part of and whether or not they are committed.

Active Logs

Any uncommitted transactions can only be in the active log files. The active log files are defined in number by the database configuration parameters LOGPRIMARY and LOGSECOND, and in size by the database configuration parameter LOGFILSIZ. On database activation (or first connect), the number of log files defined by LOGPRIMARY are created. If DB2 runs out of room in these log files, then it will allocate additional logfiles up to the number specified by LOGSECOND. This is true for either circular or archive logging.

It is generally a good idea to keep LOGPRIMARY on the smaller side to speed up database activation time, but to also have room there for your normal day-in and day-out transaction logging purposes. Usually LOGPRIMARY on my databases is 14 or fewer, with the LOGFILSZ for midsize IBM Websphere Commerce database working well somewhere between 10000 and 60000. Obviously larger, more active databases need more. I like LOGSECOND somewhere between 50 and 240 to handle those unexpected spikes. Of course all of this has to take into account the size of your active log filesystem, too.

A single transaction cannot use more log space than LOGFILSZ(LOGPRIMARY + LOGSECOND). Large deletes fail when they fill up the available log files. Commit frequently to avoid this. You can also end up with a long idle connection causing logfile saturation – causing the database to be unavailable. See “Managing db2 transaction log files” for more information on log file saturation.

Archive Logs

All OLTP and many other databases should use archive logging.

When archive logging is enabled, you should always choose a location for LOGARCHMETH1, even if it is just a different directory. A different filesystem and even a vendor archive system such as TSM is vastly preferred. But even if you are only using a single filesystem, set it to a different directory. What this does is give you a location where you know you can compress, archive, or delete archived transaction log files without crashing the database. Because if you (or anyone) mess with any active log file, you are pretty likely to crash the database, probably in a pretty nasty way.

Side note: LOGARCHMETH1 was a great parameter when it was added. Before that we had to compile and manage userexit programs, and that was a pain.

An active log is archived whenever it is full, even if it contains transactions that are still being used. However, db2 does not ‘remove’ it from the active path until it contains no active transactions. I say ‘remove’ because it looks to us like it is removed, but actually the file is renamed and reused because that performs better than actually removing a files and creating a new one. Because of this scenario, you can see the same log file in the active and archive paths at the same time.

There are some scenarios, such as data replication or HADR that can cause an archived log file to be archived. Scenarios like this make it generally a bad idea to compress or move/remove every log file as soon as it it archived to disk. They can also require you to make retrieval of logs from a tape library or other location possible from more than one database server.

Log files, whether full or not, may be archived with an ARCHIVE LOG command, a DEACTIVATE DATABASE command or a BACKUP.

LOGBUFSZ

The log buffer size frequently needs to be tuned. The larger it is, the more partial transactions you may lose in a failure – though in most failures partial transactions are rolled back anyway. In a database snapshot, two numbers are reported like this:

$ db2 get snapshot for db on sample |grep  "Log pages" 
Log pages read                             = 23108
Log pages written                          = 20513325

If you want to get that through an administrative snapshot view, you can use:

db2 "select log_reads, log_writes from sysibmadm.snapdb with ur"

LOG_READS            LOG_WRITES          
-------------------- --------------------
               23108             20513325

  1 record(s) selected.

Interestingly enough, I’m not finding this in the mon_get table functions.

Generally, you want Log pages read to be at or near 0, and Log pages written to be a large number. If Log pages read is not 0, you may want to consider increasing your log buffer, but generally you don’t want it much over 1096 or so.

Basically during normal operations we want to mostly avoid log reads.

So, readers, please share with me other tuning that you do around log file size or log buffer size. Is there anything else on transaction log files that you want to learn about?

It doesn’t happen as often as it used to, but it is still possible that you might miss installing a custom install component when you install DB2 and need to go back and install it later. I recently had to do this with the APPLICATION_DEVELOPMENT_TOOLS feature, and thought I’d share the experience and how to do it. This is all on a Linux system on which I have a typical root install of DB2 ESE 10.1 fixpack 3.

The Problem

My problem was that I needed the application development tools. I’m working on changing one of my build scripts from Perl to Ruby, and I’m attempting to make use of the IBM_DB gem to connect to the database. I’ll be blogging about that in the near future. A problem I turned out to have was that there were files I needed in /opt/IBM/db2/V10.1/include that were not there. Research and friends led me both to this conclusion and to the conclusion that to get those files, I needed to install the application development tools component.

Methods for Installing a Single Component

It has been years since I’ve had to do this, and in working through the options, I found there were three.

The Hack-it Method

On Linux, one of my options was simply to untar the file from where I have the fixpack code into the right location. This was a bit of a hacky way to go about it, and I wouldn’t recommend it unless you have a driving need, an ability to really thoroughly test it, and an ability to revert the changes. I did not choose this method, and I do not recommend it. This would involve the file universal/db2/linuxamd64/FILES/APPLICATION_DEVELOPMENT_TOOLS_10.1.0.3_linuxamd64_x86_64.tar.gz and a command that would look something like this:

cat APPLICATION_DEVELOPMENT_TOOLS_10.1.0.3_linuxamd64_x86_64.tar.gz | ( cd /opt/ibm/db2/V10.1 ; tar xzvfBp - )

Again, that is an un-tested method, and not recommended, but may work.

The GUI Method

If you’ve read other blog entries that I have written, you have probably learned that I really don’t like GUIs for most things. But I’m pretty sure you could set up x-windows or whatever and run db2setup in interactive method to do this. I don’t think you can use db2_install to add a single component. I also suspect that db2_install includes this component if you used it to install DB2 in the first place.

The db2setup Method With a Response File

The method I settled on was using db2setup with a tailored response file. This fit my command-line sensibilities and also avoided any tweaking outside of the normal supported stuff.

Using db2setup

In order to use db2setup to install a single component, you’ll have to create a tailored response file. Response file options can change with versions so make sure you’re using one for the correct version. A sample response file is included in the DB2 base code, but not in Fixpack code. It can be found in ese/db2/linuxamd64/samples/db2ese.rsp for DB2 ESE – with the ‘linuxamd64′ changing depending on your platform.

The file I used for this tailored purpose was quite short:

PROD                       = ENTERPRISE_SERVER_EDITION
FILE                        = /opt/IBM/db2/V10.1
LIC_AGREEMENT               = ACCEPT 
INSTALL_TYPE                = CUSTOM    
COMP                     = APPLICATION_DEVELOPMENT_TOOLS       ** Base application development tools

The FILE value must be changed to match wherever you have the code installed.

After you have that, running db2setup with it (as root) is simple:

cd /path/where/fixpack/code/is/universal
./db2setup -r /path/where/rsp/file/is/db2ese_10_1.app_dev.rsp

It is important that this be executed from the same fixpack code that the rest of DB2 is using. If you try to execute it from the base code or wrong fixpack, you’ll get an error like this:

DBI1191I  db2setup is installing and configuring DB2 according to the
      response file provided. Please wait.


The install location "/opt/IBM/db2/V10.1" contains an installed DB2 product
that is not at the same level as the DB2 product you are attempting to install.
Specify another location.

When it runs successfully, it will look like this:

Requirement not matched for DB2 database "Server" "". Version: "10.1.0.3". 
Summary of prerequisites that are not met on the current system: 
   DBT3514W  The db2prereqcheck utility failed to find the following 32-bit library file: "/lib/libpam.so*". 


DBI1191I  db2setup is installing and configuring DB2 according to the
      response file provided. Please wait.


A minor error occurred while installing "DB2 Enterprise Server Edition " on
this computer. Some features may not function correctly.

For more information see the DB2 installation log at "/tmp/db2setup.log".

Some Interesting Notes

I executed the above on a server with one database that was actively serving an application (development database for IBM WebSphere Commerce). I noticed this in the setup log (always read your setup logs):

The instance "db2inst2" has been updated successfully.

WARNING: The install is unable to update the instance "db2inst1" because it is
currently in use.  Ensure the instance is not active and update the instance
post install using the command "db2iupdt".

WARNING: One or more errors occurred while updating the instances on the
system. This usually occurs because some instances are currently active.

The following instances were successfully updated:  db2inst2

An error occurred while updating the instances. The following instances were
not updated successfully:  db2inst1   You must update the instances manually.
If you are performing a root installation, use the db2iupdt command. If you are
performing a non-root installation, use the db2nrupdt command. If the manual
instance update also failed, contact DB2 Customer Service.

Updating existing DB2 instances :.......Failure 

The interesting thing to me here, is that using db2setup, DB2 took it upon itself to update db2inst2 for me. db2inst2 was started, but all the databases were inactive. I did not expect this. I expected to have to update both instances myself.

After you’ve installed a component, you’ll want to run a db2iupdt on any instances to make sure they are using the new code. This requires you to stop the instance and run the db2iupdt command as root, which causes a database outage. That looks something like this:

[root@server instance]# su - db2inst1
[db2inst1@server ~]$ db2stop force
02/06/2014 21:32:30     0   0   SQL1064N  DB2STOP processing was successful.
SQL1064N  DB2STOP processing was successful.
[db2inst1@server ~]$ ipclean
Application ipclean: Removing DB2 engine and client IPC resources for db2inst1.
[db2inst1@server ~]$ exit
logout
[root@server instance]# cd /opt/IBM/db2/V10.1/instance
[root@server instance]# ./db2iupdt -k db2inst1
DBI1446I  The db2iupdt command is running.


DB2 installation is being initialized.

 Total number of tasks to be performed: 4 
Total estimated time for all tasks to be performed: 309 second(s) 

Task #1 start
Description: Setting default global profile registry variables 
Estimated time 1 second(s) 
Task #1 end 

Task #2 start
Description: Initializing instance list 
Estimated time 5 second(s) 
Task #2 end 

Task #3 start
Description: Configuring DB2 instances 
Estimated time 300 second(s) 
Task #3 end 

Task #4 start
Description: Updating global profile registry 
Estimated time 3 second(s) 
Task #4 end 

The execution completed successfully.

For more information see the DB2 installation log at "/tmp/db2iupdt.log.18853".
Required: Review the following log file also for warnings or errors:
"/tmp/db2iupdt_local.log.*"
DBI1070I  Program db2iupdt completed successfully.


[root@server instance]# su - db2inst1
[db2inst1@server ~]$ db2start
02/06/2014 21:33:24     0   0   SQL1063N  DB2START processing was successful.
SQL1063N  DB2START processing was successful.

Looking at the log file from db2iupdt, it appears to be calling db2setup to do the instance update. The file starts out:

DB2 Setup log file started at:  Thu Feb  6 21:32:45 2014 EST
============================================================

Just like a db2setup log. I find it interesting that IBM seems to be merging instance updating and installing of code into the same commands.

And my problem that required the application development tools was solved.

I am currently working on learning Ruby. I have always been a Perl fanatic, but a new build framework that my company is using requires me to use Ruby. Whee, a chance to learn something new! I am a geek.

Background

In Perl, I never used the DBI due to the issues that I had with getting it installed in a couple of AIX environments, and the inability in some client environments to install the DBI. However, with Ruby I don’t see that being an issue at this point, so I’m trying to make use of the tools designed for interacting with the database.

Little did I suspect that this decision would lead me to a 6-hour oddessy of simply trying to connect to the database. Granted, part of that time was due to my own ignorance of Ruby, and it won’t take a tenth as much time for future setups.

Installing the IBM_DB Gem

I started out wrong by simply installing the gem without looking for instructions. I found: sqllib/ruby64/ibm_db-2.5.11.gem, and simply used: gem install ~db2inst2/sqllib/ruby64/ibm_db-2.5.11.gem as root. Don’t make my mistake – use the proper instructions on installing the gem. That also happens to be by one of my favorite authors on technical blogging topics.

After I fixed my install mistake, a friend helped me figure out that version of the driver that comes with DB2 10.1 Fixpack 3 is not compatible with the version of Ruby I was using (2.0.0). The gem that was packaged with db2 was 2.5.11, and I needed 2.5.12. Of course, it did not tell me this, it simply did this when I tried to connect:

-bash-4.1$ irb
2.0.0-p353 :001 > require 'ibm_db'
 => true 
2.0.0-p353 :002 > conn = IBM_DB.connect("WC042D02", "", "")
TypeError: no implicit conversion from nil to integer
from (irb):2:in `connect'
from (irb):2
from /usr/local/rvm/rubies/ruby-2.0.0-p353/bin/irb:12:in `
'

Nice useful error message, that.

After determining that I needed to update the gem, I tried it and got this error:

[root@server ~]# gem update ibm_db
Updating installed gems
Updating ibm_db
Building native extensions.  This could take a while...
ERROR:  Error installing ibm_db:
ERROR: Failed to build gem native extension.

   /usr/local/rvm/rubies/ruby-2.0.0-p353/bin/ruby extconf.rb
Detected 64-bit Ruby

checking for SQLConnect() in -ldb2... yes
checking for gil_release_version... no
checking for unicode_support_version... no
creating Makefile
extconf.rb:114:in `libpathflag': Use RbConfig instead of obsolete and deprecated Config.
extconf.rb:114:in `libpathflag': Use RbConfig instead of obsolete and deprecated Config.

make "DESTDIR="
compiling ibm_db.c
In file included from ibm_db.c:27:
ruby_ibm_db_cli.h:52:21: error: sqlcli1.h: No such file or directory
In file included from ibm_db.c:27:
ruby_ibm_db_cli.h:56: error: expected specifier-qualifier-list before ‘SQLSMALLINT’

After much digging around with the help of my friend, we found that /opt/IBM/db2/V10.1/include was nearly empty. The reason it was empty was because I had not installed the application development tools component when I installed db2. I believe db2_install includes this component, but a typical install with db2setup does not. See my blog entry on how to install a component after the rest of db2 is installed if you, too, need to add this component in.

After installing that component and updating my instances, I tried again to update the gem and got what I needed this time:

[root@server]# . /db2home/db2inst2/sqllib/db2profile
[root@server]# echo $IBM_DB_INCLUDE 
/db2home/db2inst2/sqllib/include
[root@server]# echo $IBM_DB_LIB
/db2home/db2inst2/sqllib/lib
[root@server]# gem update ibm_db
Updating installed gems
Updating ibm_db
Building native extensions.  This could take a while...
Successfully installed ibm_db-2.5.12
Parsing documentation for ibm_db-2.5.12
unable to convert "\xB0" from ASCII-8BIT to UTF-8 for lib/ibm_db.so, skipping
Installing ri documentation for ibm_db-2.5.12
Installing darkfish documentation for ibm_db-2.5.12
Gems updated: ibm_db

Finally, I again tested a simple database connection:

[db2inst2@server]$ irb
2.0.0p353 :001 > require 'ibm_db'
 => true 
2.0.0p353 :002 > conn = IBM_DB.connect("SAMPLE","","")
 => # 
2.0.0p353 :003 > quit

That is what a successful connection looks like.

Executing a Statement

A week later, when I finally got around to attempting to execute a statement, I got this error:

2.0.0-p353 :003 > stmt=IBM_DB.exec(conn, "select * from syscat.bufferpools")
irb: symbol lookup error: /usr/local/rvm/gems/ruby-2.0.0-p353/gems/ibm_db-2.5.12/lib/ibm_db.so: undefined symbol: rb_str2cstr

With some research on http://rubyforge.org/), I figured out that in the week between getting the connection working and trying the statement, a new version of the gem had been released, that fixed this exact issue. I updated again:

[root@server ~]# . /db2home/db2inst2/sqllib/db2profile
[root@server ~]# export IBM_DB_DIR=/db2home/db2inst2/sqllib
[root@server ~]# gem update ibm_db
Updating installed gems
Updating ibm_db
Building native extensions.  This could take a while...
Successfully installed ibm_db-2.5.14
Parsing documentation for ibm_db-2.5.14
unable to convert "\xD1" from ASCII-8BIT to UTF-8 for lib/ibm_db.so, skipping
Installing ri documentation for ibm_db-2.5.14
Installing darkfish documentation for ibm_db-2.5.14
Gems updated: ibm_db

After that, I was able to execute a statement. Phew, lot of work just to get to that point.

A big thanks to @idbjorh for his extensive help with this issue. If you don’t already, you should be listening to his podcast – The Whole Package Cache

Look for more Ruby/DB2 articles from me. I write about what I do, and I expect to be working on this pretty intensively.

What is a Storage Group?

A storage group is a layer of abstraction between the data in your database and disk. It is only used with Automatic Storage Tablespaces (AST). It allows us to group tablespaces together to live in similar places. Storage groups were first introduced in a roundabout way with automatic storage databases in DB2 8.2. These databases allowed you to specify one or more paths for the entire database, and DB2 would manage spreading the data across them. It soon became clear that a level between the old school “control where everything goes” and the newer “I don’t care, just spread it across as many read/write heads as possible” was needed. Personally, I’m just fine with having only two locations for my data. I could manage that just fine in with the old methodology with DMS tablespaces, and I manage it just fine in my more recent databases with storage groups.

With DB2 10.1, IBM introduced this middle level of control. We can now create a number of storage groups. This was introduced as a way to handle multi-temperature data and disks of varying speeds. But it’s clear that we can use it in ways beyond that. I use it to separate my administrative, event monitor, and performance data to a separate filesystem from the regular database data – mostly so that if that data gets out of control, it doesn’t affect the rest of my database. If you do have SSD or different speeds of disk, the multi-temperature approach sure makes sense.

This image represents a standard DMS tablespace with file containers:

This image represents an AST tablespaces within a storage group:

You can see the additional level of abstraction represented by a storage group in gray. Assuming a tablespace is added after both storage paths are added to the storage group, DB2 will create tablespace containers on each storage path. Interestingly, all the old DMS standards like keeping all the tablespace containers for a tablespace the same size still apply with AST tablespaces and storage groups. DB2 will continue to stripe things across the tablespace containers in the same way that it does for DMS tablespaces.

Automatic Storage Tablespaces

Storage groups can only be used with automatic storage tablespaces. Automatic storage tablespaces are essentially DMS under the covers, with mechanisms for automatically extending them. They combine the best of both SMS and DMS tablespaces in that they can have the performance of DMS tablespaces, but the ease of administration like SMS tablespaces. IBM had actually deprecated both SMS and DMS tablespace types (for regular data) in favor of AST tablespaces. This means that in the future, our ability to use these tablespace types may be removed.

How to Create a Storage Group

Unless you specified AUTOMATIC STORAGE NO on the CREATE DATABASE command or have upgraded a database all the way from DB2 8.2 or earlier, you likely already have a default storage group in your database, even if you have not worked with storage groups at all. You can look at the storage groups in a datatbase with this SQL:

select  substr(sgname,1,20) as sgname, 
        sgid, 
        defaultsg, 
        overhead, 
        devicereadrate, 
        writeoverhead, 
        devicewriterate, 
        create_time 
    from syscat.stogroups 
    with ur

SGNAME               SGID        DEFAULTSG OVERHEAD                 DEVICEREADRATE           WRITEOVERHEAD            DEVICEWRITERATE          CREATE_TIME
-------------------- ----------- --------- ------------------------ ------------------------ ------------------------ ------------------------ --------------------------
IBMSTOGROUP                    0 Y           +6.72500000000000E+000   +1.00000000000000E+002                        -                        - 2014-05-07-17.44.21.791318
DB_ADM_STOGRP                  1 N           +6.72500000000000E+000   +1.00000000000000E+002                        -                        - 2014-05-08-18.49.16.108712

  2 record(s) selected.

Notice that a lot of disk characteristics that you may be used to seeing at the tablespace level are now available at the storage group level. Tablespaces can be created or altered to inherit disk settings from the storage group. Assuming each storage group is associated with similar kinds of disks, it makes sense to do things this way. To alter an existing AST tablespace to inherit from the storage group, use this syntax:

alter tablespace TEMPSPACE1 overhead inherit transferrate inherit
DB20000I  The SQL command completed successfully.

Creating a Storage Group and AST Tablespace

Creating a new storage group is easy if you know what filesystems you want associated with it:

db2 "create stogroup DB_ADM_STOGRP on '/db_data_adm/SAMPLE'"
DB20000I  The SQL command completed successfully.

Then creating an automatic storage tablespace using that storage group can be done simply as well:

db2 "create large tablespace DBA32K pagesize 32 K managed by automatic storage using stogroup DB_ADM_STOGRP autoresize yes maxsize 4500 M bufferpool BUFF32K overhead inherit transferrate inherit dropped table recovery on"
DB20000I  The SQL command completed successfully.

Since storing permanent data in DMS and SMS tablespaces has been deprecated, it is clear that IBM’s direction is to eliminate the use of these in favor of AST and storage groups.

See these blog entries for more detailed information on Automatic Storage Tablespaces:
(AST) Automatic Storage – Tips and Tricks
Automatic Storage Tablespaces (AST): Compare and Contrast to DMS

Today we are going to talk about some random DB2 features that can’t stand in a blog of their own, but are worth discussing nonetheless. These are tidbits I had discovered during “DB2’s Got Talent” presentations, IDUG conferences, or “Hey, look what I discovered” moments.

CPUSPEED (Database Management Configuration)

You blow past this setting every time you execute “db2 get dbm cfg”. It’s located at the very top of your output and is one of the more important settings that is overlooked. The result for this parameter is set after DB2 looks at the CPU and determines how fast instructions process through (millisec/instruction).

Optimizer is influenced greatly by this setting. CPUSPEED is automatically set upon instance creation and is often never examined again. The setting will stay static unless you ask DB2 to re-examine the CPU.

So, why do we care? Well there could be a few reasons.

• An additional CPU was added or subtracted from an LPAR.
• An image of your old server was taken and placed on a new, faster server, with a different type or number of processors.

If for some reason your CPU configuration was altered or changed the new processing speed is not taken into account until it is re-evaluated. So go ahead and add on that additional CPU to handle your black Friday workload. It won’t help much unless DB2 knows to take it into account.

To re-evaluate:
db2 "update dbm cfg using CPUSPEED -1"

Once done, you should see a new CPUSPEED displayed for your DBM configuration.

If you are comparing apples to apples you want to see the number get smaller to show a speed increase. If you are changing architecture (P6 to P7 for example) the number could theoretically go up. Put that in context though, the higher number could be the setting DB2 needs to account for multi-threading or some other hardware change. So it may look worse when it really isn’t.

Once done, DB2 will use the new number to determine proper access paths. So make sure to issue a REBIND once done so your SQL can take advantage of the speed increase.

(Special thanks to Robert Goethel who introduced this topic during the DB2 Night Show competition this year).

RESTRICTIVE (Create Database …. RESTRICTIVE)

I picked this up in Roger Sander’s DB2 Crammer Course at IDUG this year. I had just spent the past two months auditing our database authorities and was frustrated with the amount of PUBLIC access. I even created a separate SQL script to run after new database creation to revoke some of the default PUBLIC authority.

Apparently I reinvented the wheel. If you use the RESTRICTIVE clause in the CREATE DATABASE command no privileges or authorities will automatically be granted to PUBLIC.

For example:
db2 “create database warehouse on /data1 dbpath on /home/db2inst1 restrictive”

DB2_WORKLOAD (System Environment Variable)

This db2set parameter has been available for a while but I know a new option (ANALYTICS) became available with v10.5. Essentially, DB2_WORKLOAD will preset a group of environment variables for your needs. Set it once and go – no need to look up various configurations or develop scripts. This is valuable for  various application configurations such as BLU or Cognos.

To activate:
db2set DB2_WORKLOAD <option>

If you are a Websphere Commerce nerd like Ember, make sure to read her blog on DB2_Workload and how it can be used for Websphere Commerce.

Michael Krafick is an occasional contributor to db2commerce.com. He has been a production support DBA for over 12 years in data warehousing and highly transactional OLTP environments. He was acknowledged as a top ten session speaker for “10 Minute Triage” at the 2012 IDUG Technical Conference. Michael also has extensive experience in setting up monitoring configurations for DB2 Databases as well as preparing for high availability failover, backup, and recovery. He can be reached at “Michael.Krafick (at) icloud (dot) com”. Linked-in Profile: http://www.linkedin.com/in/michaelkrafick. Twitter: mkrafick

Mike’s blog posts include:
10 Minute Triage: Assessing Problems Quickly (Part I)
10 Minute Triage: Assessing Problems Quickly (Part II)
Now, now you two play nice … DB2 and HACMP failover
Technical Conference – It’s a skill builder, not a trip to Vegas.
Why won’t you just die?! (Cleaning DB2 Process in Memory)
Attack of the Blob: Blobs in a Transaction Processing Environment
Automatic Storage Tablespaces (AST): Compare and Contrast to DMS
DB2 v10.1 Column Masking
Automatic Storage (AST) and DMS
Reloacting the Instance Home Directory
Informational Constraints: Benifits and Drawbacks

I have not made extensive use of the HADR Tools that IBM offers in the past. Most of my HADR setups to date have either been same-data-center using NEARSYNC or have used ASYNC to copy data between data centers. I haven’t had much cause to tweak my network settings or change my SYNCMODE settings based on hardware/networking.

However, I have a chance to make use of these tools in several scenarios now, so I thought I would share what I’m finding. I do not claim to be the foremost expert on these tools. And there is an incredible amount of details on them available from IBM. For the full technical specifications and details on using the HADR tools, see:
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/DB2HADR/
http://www.ibm.com/developerworks/data/library/techarticle/dm-1310db2luwhadr/index.html?ca=dat-

I thought I would share my own journey with these tools to help others. Comments, corrections, additions, are all welcome in the comments form below.

What are the HADR tools?

IBM provides three major HADR tools on a devloperWorks wiki site.

The HADR Simulator is used to look both at disk speed and network details around HADR. It can be used in several different ways, including helping you to troubleshoot the way HADR does name resolution.

The DB2 Log Scanner is used to look at log files and report details about your DB2 Workload. The output is a bit cryptic, and this tool is best used in conjunction with the HADR Calculator. This does require real log files from a real workload, so if you’re setting up a new system, you will need to have actual work on the system before you can use it. Also, IBM will not provide the tool they use internally to uncompress automatically compressed log files, so if you want to use it, you’ll have to turn automatic log compression off. I tried to get the tool, they would not give it to me.

The HADR Calculator takes input from the DB2 Log Scanner, and values that you can compute using the HADR Simulator, and tells you which HADR SYNCMODEs make the most sense for you.

These three tools do NOT require that you have DB2 on a server to run – they are fully standalone. There are versions of the first two for each operating system. The third requires that you have perl, but can be run anywhere, including on a laptop or personal computer. This allows you flexibility in considering details of a network or server you are thinking of using before actually using it. And allows you to analyze log files without adding workload to a server.

Using the HADR Simulator

In this post, I’m going to focus on the HADR simulator.

First of all, the download and details can be found at: https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/DB2HADR/page/HADR%20simulator. Note there are some child pages and links there with good detail there.

The HADR Simulator is a stand-alone tool. This means that you do not need DB2 on the servers in question. It is a binary executable. To use it, you simply download it from the link above to one or more servers. You can simulate primary-standby network interaction by running it on two servers at the same time. You can also run it on one server alone to look at things like disk performance.

Simulating HADR with the HADR Simulator

To use it in the main intended way, you download the right version for your OS, place it on each of the severs in question, make sure you have execute permission on it and execute it like this:
Primary:

 simhadr_aix -lhost host1.domain -lport 18821 -rhost host2.domain -rport 18822 -role primary -syncmode NEARSYNC -t 60

Standby:

simhadr_aix -lhost host2.domain -lport 18822 -rhost host1.domain -rport 18821 -role standby

The ports in the above should be the ports you plan to use for HADR. However, you cannot use the same ports that HADR is currently running on if you happen to already be running HADR on the servers. If you try that, you will get output like this:

+ simhadr -lhost host1 -lport 18819 -rhost host2 -rport 18820 -role primary -syncmode NEARSYNC -t 60

Measured sleep overhead: 0.000004 second, using spin time 0.000004 second.
flushSize = 16 pages

Resolving local host host1 via gethostbyname()
hostname=host1
alias: host1.domain
address_type=2 address_length=4
address: 000.000.000.000

Resolving remote host host2 via gethostbyname()
hostname=host2
alias: host2.domain
address_type=2 address_length=4
address: 000.000.000.000

Socket property upon creation
BlockingIO=true
NAGLE=true
TCP_WINDOW_SCALING=32
SO_SNDBUF=262144
SO_RCVBUF=262144
SO_LINGER: onoff=0, length=0

Binding socket to local address.
bind() failed on local address. errno=67, Address already in use

You should be passing in the host names as you would use them with HADR. This allows the tool to show you how the names are resolving. The HADR calculator can be used for that purpose alone if you’re having name resolution issues. The ports that you pass in must be numbers – /etc/services or its equivalent is not consulted for port names if you’re using that.

The output from the HADR Simulator, invoked using the syntax above looks something like this:
Primary:

+ simhadr -lhost host1.domain -lport 18821 -rhost host2.domain -rport 18822 -role primary -syncmode NEARSYNC -t 60

Measured sleep overhead: 0.000004 second, using spin time 0.000004 second.
flushSize = 16 pages

Resolving local host host1.domain via gethostbyname()
hostname=host1.domain
alias: host1.domain.local
address_type=2 address_length=4
address: 000.000.000.000

Resolving remote host host2.domain via gethostbyname()
hostname=host2.domain
alias: host2.domain.local
address_type=2 address_length=4
address: 000.000.000.000

Socket property upon creation
BlockingIO=true
NAGLE=true
TCP_WINDOW_SCALING=32
SO_SNDBUF=262144
SO_RCVBUF=262144
SO_LINGER: onoff=0, length=0

Binding socket to local address.
Listening on local host TCP port 18821

Connected.

Calling fcntl(O_NONBLOCK)
Calling setsockopt(TCP_NODELAY)
Socket property upon connection
BlockingIO=false
NAGLE=false
TCP_WINDOW_SCALING=32
SO_SNDBUF=262088
SO_RCVBUF=262088
SO_LINGER: onoff=0, length=0

Sending handshake message:
syncMode=NEARSYNC
flushSize=16
connTime=2014-06-15_18:24:42_UTC

Sending log flushes. Press Ctrl-C to stop.

NEARSYNC: Total 18163171328 bytes in 60.000131 seconds, 302.718861 MBytes/sec
Total 277148 flushes, 0.000216 sec/flush, 16 pages (65536 bytes)/flush

Total 18163171328 bytes sent in 60.000131 seconds. 302.718861 MBytes/sec
Total 277148 send calls, 65.536 KBytes/send,
Total 0 congestions, 0.000000 seconds, 0.000000 second/congestion

Total 4434368 bytes recv in 60.000131 seconds. 0.073906 MBytes/sec
Total 277148 recv calls, 0.016 KBytes/recv

Distribution of log write size (unit is byte):
Total 277148 numbers, Sum 18163171328, Min 65536, Max 65536, Avg 65536
Exactly      65536      277148 numbers

Distribution of log shipping time (unit is microsecond):
Total 277148 numbers, Sum 59711258, Min 175, Max 3184, Avg 215
From 128 to 255               263774 numbers
From 256 to 511                13335 numbers
From 512 to 1023                  23 numbers
From 1024 to 2047                 15 numbers
From 2048 to 4095                  1 numbers

Distribution of send size (unit is byte):
Total 277148 numbers, Sum 18163171328, Min 65536, Max 65536, Avg 65536
Exactly      65536      277148 numbers

Distribution of recv size (unit is byte):
Total 277148 numbers, Sum 4434368, Min 16, Max 16, Avg 16
Exactly         16      277148 numbers

Standby:

+ simhadr -lhost host2.domain -lport 18822 -rhost host1.domain -rport 18821 -role standby

Measured sleep overhead: 0.000004 second, using spin time 0.000004 second.

Resolving local host host2.domain via gethostbyname()
hostname=host2.domain
alias: host2.domain.local
address_type=2 address_length=4
address: 000.000.000.000

Resolving remote host host1.domain via gethostbyname()
hostname=host1.domain
alias: host1.domain.local
address_type=2 address_length=4
address: 000.000.000.000

Socket property upon creation
BlockingIO=true
NAGLE=true
TCP_WINDOW_SCALING=32
SO_SNDBUF=262144
SO_RCVBUF=262144
SO_LINGER: onoff=0, length=0

Connecting to remote host TCP port 18821
connect() failed. errno=79, Connection refused
Retrying.

Connected.

Calling fcntl(O_NONBLOCK)
Calling setsockopt(TCP_NODELAY)
Socket property upon connection
BlockingIO=false
NAGLE=false
TCP_WINDOW_SCALING=32
SO_SNDBUF=262088
SO_RCVBUF=262088
SO_LINGER: onoff=0, length=0

Received handshake message:
syncMode=NEARSYNC
flushSize=16
connTime=2014-06-15_18:24:42_UTC

Standby receive buffer size 64 pages (262144 bytes)
Receiving log flushes. Press Ctrl-C on primary to stop.
Zero byte received. Remote end closed connection.

NEARSYNC: Total 18163171328 bytes in 59.998903 seconds, 302.725057 MBytes/sec
Total 277148 flushes, 0.000216 sec/flush, 16 pages (65536 bytes)/flush

Total 4434368 bytes sent in 59.998903 seconds. 0.073907 MBytes/sec
Total 277148 send calls, 0.016 KBytes/send,
Total 0 congestions, 0.000000 seconds, 0.000000 second/congestion

Total 18163171328 bytes recv in 59.998903 seconds. 302.725057 MBytes/sec
Total 613860 recv calls, 29.588 KBytes/recv

Distribution of log write size (unit is byte):
Total 277148 numbers, Sum 18163171328, Min 65536, Max 65536, Avg 65536
Exactly      65536      277148 numbers

Distribution of send size (unit is byte):
Total 277148 numbers, Sum 4434368, Min 16, Max 16, Avg 16
Exactly         16      277148 numbers

Distribution of recv size (unit is byte):
Total 613860 numbers, Sum 18163171328, Min 376, Max 65536, Avg 29588
From 256 to 511                  166 numbers
From 1024 to 2047              55614 numbers
From 2048 to 4095               8845 numbers
From 4096 to 8191              18028 numbers
From 8192 to 16383             34458 numbers
From 16384 to 32767           227758 numbers
From 32768 to 65535           264416 numbers
From 65536 to 131071            4575 numbers

Ok, that’s great, right, but what do I do with that?

Well, here’s one thing – you can tune your send and recieve buffers using this information. Run this process several times using different values for those like this:

./simhadr_aix -lhost host1.domain -lport 18821 -rhost host2.domain -rport 18822 -sockSndBuf 65536 -sockRcvBuf 65536 -role primary -syncmode NEARSYNC -t 60
./simhadr_aix -lhost host2.domain -lport 18822 -rhost host1.domain -rport 18821 -sockSndBuf 65536 -sockRcvBuf 65536 -role standby

./simhadr_aix -lhost host1.domain -lport 18821 -rhost host2.domain -rport 18822 -sockSndBuf 131072 -sockRcvBuf 131072 -role primary -syncmode NEARSYNC -t 60
./simhadr_aix -lhost host2.domain -lport 18822 -rhost host1.domain -rport 18821 -sockSndBuf 131072 -sockRcvBuf 131072 -role standby

./simhadr_aix -lhost host1.domain -lport 18821 -rhost host2.domain -rport 18822 -sockSndBuf 262144 -sockRcvBuf 262144 -role primary -syncmode NEARSYNC -t 60
./simhadr_aix -lhost host2.domain -lport 18822 -rhost host1.domain -rport 18821 -sockSndBuf 262144 -sockRcvBuf 262144 -role standby

./simhadr_aix -lhost host1.domain -lport 18821 -rhost host2.domain -rport 18822 -sockSndBuf 524288 -sockRcvBuf 524288 -role primary -syncmode NEARSYNC -t 60
./simhadr_aix -lhost host2.domain -lport 18822 -rhost host1.domain -rport 18821 -sockSndBuf 524288 -sockRcvBuf 524288 -role standby

In the line of output that looks like this:

NEARSYNC: Total 14220328960 bytes in 60.000083 seconds, 237.005155 MBytes/sec

Pull out the MBytes per second, and graph it like this:

In this example, it is clear that the throughput levels off at a buffer size of 128 K. Your results are likely to vary. To allow additional space, in this example, we would choose values of 256 KB, and set them using this syntax:

db2set DB2_HADR_SOSNDBUF=262144
db2set DB2_HADR_SORCVBUF=262144

This is the kind of thing I might never have gone into detail on if I didn’t blog. And yet it led to me changing parameters used and improving what I’m doing at work.

I am also interested in what I might do with some of the disk information supplied here. I sometimes have trouble getting disk information from hosting providers and, depending on the situation, there might be numbers here that I could use.

I’m really disappointed that IBM won’t share their internal log uncompression tool to use the log scanner – I’m not sure I can justify running with manually compressing logs just to run the logs scanner. Automatic log compressions is one of my favorite recent features. If I get the opportunity, I’ll play with that tool and blog about it too.

I was a bit shocked when I searched my own blog for entries on db2look and came up with nothing. While it’s not a complicated tool, db2look is an essential tool.

What db2look Can Do

db2look is a tool provided with DB2. db2look’s primary purposes are:

• Mimic Statistics – To generate the SQL to update statistics with the idea that they will be run in one database to match statistics in another database.
• Generate SQL – To generate all or part of the SQL required to re-create a database.

The tool is generally referred to as a mimicking tool – generating the SQL to mimic something from one database in another database. It can be used for a single table, a schema, or the whole database. There are a wide range of options to specify exactly how deep you want to go with the mimicking.

IBM DB2 Info Center page on db2look

Authorization Required

For much of what db2look does, you’ll just need select on the system catalog tables, which in many cases may be granted to PUBLIC by default. For some options, you may need one of the following:

• SYSADM
• SYSCTRL
• SYSMAINT
• SYSMON
• DBADM
• EXECUTE privilege on the ADMIN_GET_STORAGE_PATHS table function

You’ll also need connect authorization on the database, as db2look establishes its own database connection. If no one has ever run db2look on that databse before, you may have to also bind the db2look packages (which db2look may try to do automatically), and therefore may need bindadd permission.

db2look Syntax

The db2look syntax is not difficult. You can run db2look either for a local database or over a remote connection – though it doesn’t play nice across versions sometimes if the local and remote versions are significantly different. You always have to provide db2look with a database name using the -d option. If connecting to a remote database, you will also have to provide a user name and password.

In DB2 10.1 and DB2 10.5 a couple of new options have been added – the -createdb option will also generate the create database statement needed to create a new, identical database. The -printdbcfg option will print out the database configuration statements as well. These were added to 10.1 in a fixpack, so are only available if you either created the database on a more recent fixpack, or if you issued the db2updv10 command after a fixpack or version upgrade (and you should have).

The -e option is what you want if you’re extracting SQL to re-create objects or the database. You can optionally specify tables or schemas to limit the output. You may want to use the -x or -xd options to also generate grant statements for the objects you’re extracting the SQL for. If your intention is to mimick the whole database, the -l option is useful to also generate the SQL to create things like tablespaces and bufferpools. If part of what you’re extracting includes triggers or SQL stored procedures, you may want to use the -td <delimiter> option to specify a non-standard statement delimiter (something other than ;).

The -m option is what you want to generate the update statements to mimic statistics.

The -o option is used to specify an output file name.

There are certainly other options you may want to consider using in specific situations.

When to use db2look

There are a number of scenarios where db2look is useful.

• Generating the syntax to create a single table in one environment that already exists in another environment.
• Generating the syntax to compare an object in one environment to an object in another environment.
• As part of a disaster recovery plan. If you regularly generate a full db2look, you’re prepared to re-create objects that may have been incorrectly dropped or changed, or in a worst-case scenario to create an empty version of your database.
• Generating the syntax from a production environment that can be used to create a non-production or development environment.
• Generating the syntax to move all objects in a schema from one database to another (object definition only)
• Generating the syntax to re-create objects as a backout plan for object changes
• Along with db2move, db2look can be used to move a DB2 database from one environment to another when a backup/restore is not possible (such as between DB2 versions that are more than 2 versions apart or between differing OS platforms)
• Generating the syntax to recreate everything in the database, but then altering the output file before it is executed to make significant structure changes (though admin_move_table is now often used for these within the same database)

db2look Caveats and Warnings

db2look does not extract the data within the database in any way. You cannot use it as a replacement for backup/restore. Also be aware that db2look output generated on some newer versions may not directly work on older versions – clauses like organize by row work on DB2 10.5, but not on any older versions.

In some older versions, the options to specify a table or schema were not very reliable. This may have been as far back as DB2 version 7, but it is easy to make a mistake and not get the SQL you thought you were getting. Always review the output file before relying on it – especially in situations where you are relying on having the data.

db2look does not have a significant performance impact, especially considering the normally infrequent basis when it is used. It does not affect database availability to generate the file, but if you run the generated file somewhere, that can have drastic impacts on database structure – with the right set of options, the file output will drop objects.

DB2 doesn’t have any defaults for filesystems because that is an OS-level thing. However, there are a few sanity checks and some recommeded separation when installing DB2 on Linux and UNIX operating systems. These are my best practices for installations, in general terms.

Software Location

The default location for installing the DB2 code in /opt/ibm/db2/V10.5 (replacing V10.5 with whatever version). Yes, I typed that from memory without having to look it up. The default location works fine, though I would recommend that /opt is mounted as its own filesystem and not just part of root. There are reasons to put the code in other places, but I wouldn’t unless you have a technical reason to do so. If you’re using db2_install (my favorite installation method), it will prompt you for the install location.

Don’t Create Instances in / or in /home

After software installation, the next location that DB2 uses is the home directory of the instance owner. Before creating the DB2 instance (either using db2setup or using db2icrt), you need to verify the primary group and the home directory of the DB2 instance owner. DO NOT place the home directory of the db2 instance owner on / or /home. The reasoning for this is because those locations may quite easily be filled up by other users on the server, and if the instance home fills up, DB2 becomes unusable. We don’t want a random user transferring a giant file and crashing the database. The home directory of the DB2 instance owner cannot easily be changed after instance creation.

Other Recommended Filesystems

For a minimal usage system, I can live with only two filesystems as described above – /opt and some filesystem for the home directory of the DB2 instance owner. But if I’m setting up a production system to the best reasonable design, I also include separate filesystems for at least the following.

Two Data Filesystems

I place the database home directory in a data filesystem. I like to have two just because it makes sense to me to start with two, even if they are both part of the same storage group.

The easiest way to configure a new database to use these filesystems is on the create database command – specify ON /path1, /path2 DBPATH ON /path1. I will also frequently set the DFTDBPATH DBM CFG parameter to the first data path.

Active Transaction Logs

Active transaction logs are critical to database performance and also to database functioning. Active logs should be on the fastest storage you have, and you should train operations and other people who might touch your database server to NEVER touch the active transaction log files. I have seen a production database crashed by a more junior operations person compressing an active transaction log.

To direct active log files to a new path, you’ll have to set the database configuration parameter NEWLOGPATH and deactivate/activate your database for the change to take effect

Archive Transaction Logs

If you have TSM or another location for Archiving transaction logs that is best and safest. If you are archiving to disk, these MUST be in a different directory from active transaction logs, and preferably in a separate filesystem. The logic for a separate filesystem is that this gives you an extra layer of protection – if you’re monitoring for full filesystems, you will catch a logging issue when it fills up your archive log filesystem, and hopefully have time to address it before it also fills your active log filesystem and makes your database unavailable.

To archive transaction log files to a path, you have to set the LOGARCHMETH1 db cfg parameter to: DISK:/path. If you’re setting LOGARCHMETH1 for the first time, you may also be changing from circular logging to archive logging, which requires an offline database backup, so be cautious.

db2diag/db2dump

The db2 diagnostic log and other diagnostic files by default will be in $INSTHOME/sqllib/db2dump. I like to have them in another filesystem – this ensures that no matter what else fills up or what other filesystem level problems are encountered, I should still get the error messages from those issues.

The location for this data is changed using the DIAGPATH parameter in the DBM cfg. It can be updated with an UPDATE DBM CFG command, and changes to it take effect immediately.

Backups

I like to take backups to a different filesystem. If your filesystems are fully separate I/O paths, this can have some performance benefits. But the real reason is because the backup location is the location you’re most likely to fill up, and you don’t want a filesystem filling up because of a backup and causing an availability issue for your database.

Specify the backup filesystem on any BACKUP DATABASE commands you issue, including those from scripts.

Scripts

I have seen a script go a bit haywire and capture more data that it was thought it would, and fill up the filesystem it is writing data to. For this reason, I like to keep my administrative scripts and their output on a separate filesystem from everything. This makes it much harder for a scripting mistake to cause a database availability issue.

Others

Obviously there are other filesystems that particular designs may call for – such as a shared filesystem between two HADR servers when doing loads with copy yes. But the above are true to just about every DB2 installation.

Separating I/O

This post does not cover storage design in depth. Usually the storage I get is a black box. A client allocates the space locally on the server or on a SAN, and I have a very hard time finding out the storage details. If I actually get to specify anything about where the storage goes or what is separate, my first priority is to separate out my active transaction logs and put them on the fastest storage I have, separate from everything else. If I have more specificity available, I like to separate my two data filesystems from my backup filesystem. After that it’s all icing. In only a very few cases do I see indexes and data separated any more. When I started as a DBA 14 years ago, that was a core tenant of DB database storage design.

It is Only Temporary

I cannot tell you how many stupid things I have been asked to do in the name of “It’s only temporary!” or “It’s only a development environment”, only to have those things become permanent fixtures in a production environment. Just today, I installed DB2 on root, with no separate filesystems at all, after making it perfectly clear I thought it was a horrible idea. My exact words were:

If you want me to move forward with the understanding that a significant percentage of our clients come to us just because of this kind of mis-configuration and the stability issues it causes, then I can. But I want you to understand what a bad idea I think it is.

There really is no such thing as “It’s only temporary” – design correctly from the start and the problems that you encounter are much easier to deal with.

I have been working with multiple instances on Windows servers lately, and have learned a few things about them, so I thought an article about multiple instances in general would be a good thing.

The Basics

First, if you don’t understand what an instance is, check out my earlier blog entry: DB2 Basics: What is an Instance?

When Multiple Instances Work and When They Don’t

In general, for production systems, the one-server -> one-instance -> one database approach is best. This allows us to fully isolate the workload and is easiest for the intense workload related performance problems that may be encountered for production systems. There are exceptions to that, of course. I’ve seen an application that required 8 (EIGHT!) separate, tiny, and not very busy databases. I would certainly not recommend each of those go on a different server, and even put all eight databases on a single instance. But if you’re talking about databases of greater than 10 GB that are reasonably heavily used, the more isolated, the better.

On non-production database servers, multiple instances are much more common. If you have multiple development/test/QA/staging environments on the same non-production server, you need them to be on separate instances. Why? Well, you need to be able to move a change to an instance-level parameter through the enviornments one-by-one to test it. Perhaps more importantly, you need to be able to independently upgrade the instances so you can move an upgrade through the environments.

Sometimes I see HADR standbys for multiple different production databases servers on the same standby server. I’ve also seen Dev and HADR standbys the the same server. Either of these setups require not just separate instances, but the ability for the separate instances to run on different versions or Fixpacks of the DB2 code – you should be upgrading them independently and likely at different times.

Separate DB2 Base Installation Paths/Copies

We talk not just about multiple instances but multiple base installation paths (Linux/UNIX) or multiple copies (Windows). You can have two instances on the same server running on the same path/copy if they will be upgraded together. But if there is some need to upgrade or patch them separately, they need to run on separate paths/copies.

Multiple Instances on Linux/UNIX

The Instance Owner

On Linux and UNIX systems, there is a DB2 Instance Owner. This ID is the same exact name as the DB2 instance, and the DB2 Instance’s SQLLIB directory is always in the DB2 Instance Owner’s home directory. One of the pre-instance creation verification steps is to make sure that this user exists and has an appropriate home directory, so your Instance Home does not end up in someplace inappropriate like the root or /home filesystems.

The Base Installation Path

When I started working with DB2, it wasn’t possible to change the installation path. Then they realized that some servers require the ability to have different instances on different fixpacks, so they introduced “alternate” fixpacks, which were a bit of a pain to work with. Finally, they gave us the option to choose our own installation path, which makes it so that we can have as many different versions and fixpacks as we like on the same server. This also means that whenever you install a fixpack you have to specify the base installation path (using the -b option or as prompted). You can also change an instance from one base installation path to another using a db2iupdt command (offline).

Administering the Instance

DB2 instances are administered separately, as you can really only work on one instance at a time. A DB2 instance is administered either as the DB2 instance owner or as an ID in the SYSADM group for an instance. In order to take actions for the instance or the databases under it, you will execute the db2profile from the SQLLIB directory of the Instance Home directory. You cannot use the same ID to take actions on two different instances without switching from one to another.

Multiple Instances on Windows

Multiple instance support feels to me like it has come slower to Windows than it did on UNIX/Linux, but I don’t have the facts to support that, as I have only worked extensively with DB2 on Windows in recent years.

The Instance Owner

There isn’t a userid that is quite as tightly tied to an instance as it is on Linux/UNIX. You still have a name for each instance, and can add additional instances on a server.

The DB2 Copy

The equivalent concept to the base installation path on UNIX/Linux is the DB2 Copy on Windows. You’ll have a name for the DB2 Copy in addition to the instance name. By default, this name is DB2COPY1.

Administering the Instance

Many times on DB2 servers, the local administrators will all have SYSADM rights on the database. If not, every ID in the DB2ADMINS group will have SYSADM. You may use the same ID to administer two DB2 instances at once, but any given command window, PowerShell prompt, or DataStudio window only accesses one instance at a time. In a PowerShell window, you can set the environment variable DB2INSTANCE to switch between instances on the same DB2 Copy, and can set the PATH variable along with DB2INSTANCE to switch between instances on different DB2 copies.

Updated 6/21/2012 to provide complete syntax for db2iupdt.

So sometimes in Commerce 7 instance creation, we get this error:

SQL30082N Security processing failed with reason “42” (“ROOT CAPABILITY REQUIRED”). SQLSTATE=08001

It doesn’t happen every time, but significantly often. On our most recent build it occured on 1 out of 3 instances created.

The solution is not to run Commerce instance creation as root (please, don’t!). Instead, a db2iupdt is what you need. This does require a DB2 instance outage so be careful if you have other databases on the same DB2 instance.

After instance creation has failed:

1. Stop the DB2 instance on the database server:
   • (as DB2 instance owner) db2stop
   • (as DB2 instance owner) ipclean
   • IF AIX (as root) /usr/lib/slibclean
2. Run db2iupdt on the database server:
   • (as root) cd /opt/IBM/db2/V9.5/instance (may differ if you installed db2 on a different path)
   • (as root) ./db2iupdt -k -e (syntax may vary for multiple instances on a server – see http://publib.boulder.ibm.com/infocenter/db2luw/v9r7/topic/com.ibm.db2.luw.admin.cmd.doc/doc/r0002060.html)
3. Start the DB2 instance on the database server:
   • (as DB2 instance owner) db2start

Then re-do Commerce Instance creation, and you won’t get the SQL30082N RC 42 error message.

db2iupdt is usually only required for fixpacks. I’ve also seen it resove various issues including persistent SQL1042C errors. It basically goes out to sqllib and refreshes both copies and links between there and the base db2 code. I’m not entirely sure as to why it is required in this case. We do the DB2 installation and instance creation manually ourselves – perhaps the Commerce instance creation is attempting to do some of that work?

So one of the most common questions I get is about moving data between homogenous Commerce databases. Our standard setup includes 4 environments – Dev, QA, Stage, and Prod. Dev/QA are a stagingprop pair, Stage/Prod are a stagingprop pair, and Prod usually has HADR. So with 4 environments, they can get out of sync. For the most part, business users are making changes on staging, and stagingprop is used to move those changes to prod. Therefore the catalog between staging and prod is kept in sync, and we know right away if stagingprop fails and it gets out of sync. Some data we never expect in an environment other than prod – and in fact do not want it to end up in the lower environments where many developers have unrestricted access, and security is not as tight. That data includes orders, users, payment information, etc.

I hope to present a series of postings to address a number of scenarios and approaches for this kind of data movement. I’m not talking in this series about data movement as in loading catalog or user data into the database for the first time, or keeping things in sync with a separate system of record.

So the most common set of goals are:

1. Keep private data in production only
2. Synchronize catalog and other data across environments
3. Staging is the master for catalog and data that is needed in all environments

Data movement Scenarios:

Create a staging environment to match production

This is most commonly encountered during build, but can be needed after build as well to refresh the data in a staging environment to match production. Commerce intends that we create our production environment, develop there, and then move the database data back to staging using the stagincopy utility. Essentially the stagingcopy utility does these tasks:

1. Exports data from the production database
2. Deletes any data from the staging database
3. Imports data into the staging database
4. Creates the stagingprop triggers (and sequences) in the staging database

When running stagingcopy you can either have stagingcopy do all the work, or you can have it write the script and run the script yourself.

One of the biggest problems with stagingcopy is that it does all of the work in the staging database in ONE UNIT OF WORK! This means that you must have huge transaction logs to support it. We usually run with a LOGFILSIZ of 10000 and 12 primary and 50 secondary logs. For a newer database with a small or medium size, you can get by with bumping LOGSECOND up to 240 in this scenario (an online change). If you’ve got the default log file size of 1000, there is not an online change you can make to make that work. There is a limit of 255 for the combination of LOGPRIMARY and LOGSECOND. What you can do is change LOGFILSIZ and then recycle the database for the change to take effect, making sure that your filesystem has space for the log files. If you don’t have log files on a separate filesystem, you’ll also have to make sure that your filesystem can hold the files data is exported to, the data being loaded into, and the log files. Depending on the size of your catalog, you may not be able to have enough log space to support stagingcopy.

Stagingcopy also tends to end up with cryptic error messages, and I usually expect at least two iterations to get it to work – with an experienced commerce DB2 DBA sitting there waiting for it to succeed or fail. My knee-jerk estimate (when I don’t know database/catalog size) is that a stagingcopy will take 8 hours to complete, during which the target (staging) environment will be unavailable. In most situations, that allows for it to start over at least once. I’ll talk in a future post about some non-supported ways that stagingcopy can be useful.

Stagingcopy must be run from an application server (if you have application servers that are separate from your database servers). You also must use the specify the same id to stagingcopy that you use for the application to connect to your database – stagingcopy cannot handle a different schema name than the user you specify on execution. I’m not going to go through the syntax in detail – it is in the info center:

http://publib.boulder.ibm.com/infocenter/wchelp/v7r0m0/index.jsp?topic=/com.ibm.commerce.admin.doc/refs/rsscopy.htm

The one thing I will say is that in my experience the batchsize parameter does not work.

Stagingcopy is the supported way of doing this. But it’s not the only way we do it. IF the Commerce versions including FixPacks, Feature Packs, and Feature enablement are exactly identical(query the site table), and the Merchant Keys are also identical, then you can do a straight restore at the DB2 level from the production database to the staging database. I only recommend doing this pre-go-live or for environments where the database you’re backing up does not have any real customer or order data. If everything falls into line for this, it is immensely easier, taking usually less than 30 minutes, depending on your hardware and your catalog size. With Staging and Production on different servers, we keep the databases such that we can do a straight restore without a redirect (storage architecture is a separate topic), and do this kind of restore quickly.

Using this restore method, you also have to ensure that your staging triggers are in place after you have completed the restore. Conversely, you’ll need to make sure that the staging triggers DO NOT exist on your production database. You can query syscat.triggers like this to see if the staging triggers are there:

$db2 "select count(*) from syscat.triggers where trigname like 'STAG%' with ur"
1
-----------
        951
  1 record(s) selected.

The number of triggers will vary by version and with enablement of features like CMC, but you should certainly have over 500. If you don’t have them, you can find the syntax for them on the app servers, here(on Linux, anyway):

/opt/IBM/WebSphere/CommerceServer70/schema/db2/wcs.stage.trigger.sql

and for Commerce 7 you’ll also need to have the sequences in place:

/opt/IBM/WebSphere/CommerceServer70/schema/db2/wcs.stage.sequence.sql

These steps are not needed if you use stagingcopy because stagingcopy takes care of them for you.

Note that if you have different Feature Packs, FixPacks, or features enabled, using the restore method will succeed, and the app will come up, BUT you will have inconsistencies that will likely require you to rebuild your whole commerce instance from scratch at a later time.

Next post I’ll discuss keeping a dev or qa database in sync with your stage/prod pair.

So one of the most common questions I get is about moving data between homogenous Commerce databases. Our standard setup includes 4 environments – Dev, QA, Stage, and Prod. Dev/QA are a stagingprop pair, Stage/Prod are a stagingprop pair, and Prod usually has HADR. So with 4 environments, they can get out of sync. For the most part, business users are making changes on staging, and stagingprop is used to move those changes to prod. Therefore the catalog between staging and prod is kept in sync, and we know right away if stagingprop fails and it gets out of sync. Some data we never expect in an environment other than prod – and in fact do not want it to end up in the lower environments where many developers have unrestricted access, and security is not as tight. That data includes orders, users, payment information, etc.

I hope to present a series of postings to address a number of scenarios and approaches for this kind of data movement. I’m not talking in this series about data movement as in loading catalog or user data into the database for the first time, or keeping things in sync with a separate system of record.

So the most common set of goals are:

1. Keep private data in production only
2. Synchronize catalog and other data across environments
3. Staging is the master for catalog and data that is needed in all environments

Data movement Scenarios:

Synchronizing Data Between Commerce Environments

Ok, so the idea in this post is that you’ve got your Commerce Environments up and running – with a synchronized and staginproped stage/prod pair and one or more development/test/qa environments as well. I frequently get an email or ticket saying something like “Copy production to devint by 4pm today”. That is rarely doable or a complete request.

In all of the methods below, your target environment will need to be down (or will not work right) while the copy is in progress.

DB2 Database level backup/restore

One of the first concerns is private data. Whether you’re storing card data and it’s PCI you’re worried about or just plain protection of customer data such as email addresses and other data that must be kept confidential. Our security set up means extremely restricted permissions on who can query and change data in production. For that reason, after go-live and/or data load, I don’t recommend doing a full database backup and restore from production to any other environment. Most of the time, what is really needed is just the data that is part of stagingprop – what I refer to as ‘catalog’ data – not the database catalog, but the “product” catalog of the site. Because your staging environment is kept in sync with production on this data, frequently a full database backup and restore from staging can be done.

There are some considerations with a backup of the database for one environment and the restore into another:

1. The Commerce version and FixPack and FeaturePack and Feature Enablement must be exactly identical. If they are not, you will completely mess up your environments and frequently end up with an environment with a feature or FixPack only partly applied and no way to fully apply it or to back it out. I have ended up rebuilding environments from scratch when this happens – it’s not fun.
2. The Merchant Key must be the same. Whether you keep the Merchant Key in sync between two or more environments or you temporarily change it to be identical, if you restore with different Merchant Keys, any encrypted data will be completely unusable, and you may not realize it right away.
3. If you’re restoring a staging database to production (or to an environment that is a target of a stagingprop pair), you’ll want to drop the staging triggers after the restore on the target. If you don’t, you’ll be unable to delete members and performance may suffer
4. If you’re restoring from production or an environment that is the source of a stagingprop pair into a staging database or a database that is the target of a stagingprop pair, you’ll want to create the stagingprop triggers (and possibly sequences on commerce 7) after the restore
5. If you doing keysplitting or anything fancy there, you’ll want to get the keys table to the proper state – possibly by exporting it, restoring, and then importing the keys table back in.

 Other than those, backup/restore is a quick and easy way to move data, and is my preferred option when the items above can be dealt with. I do still backup the target database before doing the restore so I have a solid fall-back.

Non-standard use of stagingcopy

sshhh, don’t tell IBM. But you can use stagingcopy other than as intended. IBM intends that you use stagingcopy exactly once – when you first create your staging server. But what happens if I specify any databases I choose when executing it? Well, it still copies that critical data. As I mentioned previously, stagingcopy has drawbacks. It is slow – running deletes and imports on dozens of tables as a single unit of work. So you’ll have to increase the size of the log files on your target, bumping LOGSECOND up to 255-LOGPRIMARY and possibly increasing LOGFILSZ as well. Also stagingcopy deletes users and some other things you might have expected to keep so be sure to test the results to make sure they’re what you need. Stagincopy will not work for especially large databases – you just won’t be able to make the transaction log files large enough.

Stagingcopy does create the staging triggers (and maybe sequences in Commerce 7?) on the target database when it is done, so you would want to drop them if the target database is not a staging environment.

Stagingcopy can also be used to generate a script (with the -script_file option). If you need to or want to make changes to what stagingcopy is doing, that script is an excellent way to do it. That is actually my preferred method for developers moving data to their personal development environments – the scripts and export files are usually something that can be moved over to Windows implementations of Commerce from the Linux or Unix systems.

A short list of tips…

1. remember to take a backup of the target database before starting the stagingcopy – it is very likely stagingcopy will fail and you’ll have to start again. I can’t tell you how many times I’ve used that backup even with my extensive experience.
2. Run your process in the background (I like starting the command with ‘nohup’ and ending it with ‘&’), so that if you lose your connection, you won’t have to start over.
3. Make your logs as large as you possibly can (given disk limits), increasing LOGSECOND and/or LOGFILSZ db cfg parameters
4. Have a person actually watching/checking in every few minutes to catch failures as soon as possible
5. Set expectations that the process will take a while
6. Take a backup of the target database when you’re done so that you have a restore point moving forward

Traditional database methods: db2move, load, etc

Somewhere out there, a DBA is reading this, and is thinking “well, I’d just use db2move”. Ok, so I’ve done it that way – using db2move or a custom script of load statements. But you’d better have a script ready that will go through and determine the tables in check pending and take them out of it and report any violations directly to you, because a simple load on the member table for example will put approximately 300 tables into check pending. And in my humble opinion, that’s just too many to run set integrity statements on manually. Usually if I end up at that method, I spend 16 hours all of it working closely with a developer to work through all the exceptions. And believe me, even if you have the mentioned script, you’ll likely have exceptions.

The RI (Referential Integrity) in a Commerce database is simply insane (Why, oh why does each item on an order have to have a member id associated with it in addition to the member id associated with the order?). I bet I could find three tables that would put 90% of the database into check pending – yes, that’s over 600 tables. Until you’ve done it, you just won’t understand the scale. So go ahead and come up with a list of tables and the right order if you’re not using LOAD and the script for set integrity, and then make sure someone tests the heck out of it. And don’t lose those because if it’s needed once, it’ll likely be needed again.

This is not my favorite method, though I have used it and probably will in the future – there are some cases where apparently only it will do – you can finely control which tables are moved, if you can avoid causing exceptions to RI constraints. If you have to go this route, starting with a script file from stagingcopy(and altering it to meet your needs) is a good way to go, because it gets you the correct order and makes you think the way that stagincopy runs.

A short list of tips…

1. Take a backup of the target database before starting – having something to fall back to can be a lifesaver.
2. If possible, generate and start from the script file generated when using the -script_file option on stagingcopy. If you’re not using LOAD, you’ll need this to get tables done in the correct order
3. If you’re using LOAD be prepared with a script to help you set integrity and dump conflicts to an exception table. Set integrity statements will take longer than the load itself in all likelihood.
4. Have someone test thoroughly when you’re done – nothing worse than discovering weeks later that there was an issue and trying to get everything fixed then.
5. Take a backup of the target database when you’re done so that you have a restore point moving forward

I have generally been a bit disappointed on the information coming out of IBM on supporting Commerce databases. I also meet clients who don’t believe they need a dba or that they can hire someone out of college or with minimal database experience to fill such a role. In my opinion, any reasonably sized Commerce implementation needs a DBA, and unless that DBA has years of experience with DB2 (or Oracle as the case may be), and at least a year of experience supporting Commerce databases specifically, then it’s going to need to be a full-time dedicated DBA.

The basic message I get from IBM on database support – both in their intro readily available materials and in their selective customer experience and feedback sessions is that they see the DBA’s role as basically:

1. backup the database
2. do runstats and reorgs on the database
3. run dbclean on the database

Sure, that’s all I do, all day long. And my two other highly experienced colleagues together supporting over 100  databases.

As an example, here’s IBM’s tech-note (side-note, afterthought) on database support: http://www-01.ibm.com/support/docview.wss?uid=swg21224921

Heck, that one doesn’t even mention backups.

I could (and have) written over 50 pages of documentation on the items on that page alone. Not to mention thousands of lines of scripting. I’m hoping to do some nice details here on the blog on dbclean and data pruning, but that one section that they give so little mention to (basically they say “run dbclean”), I’ve put not less than 30 hours of effort into for every single client, not to mention the dozens if not hundreds of hours I and my colleagues have spent developing our overall strategy and scripts.

Supporting a WebSphere Commerce database is not rocket science. But it’s not 5th grade science either. Here’s a random list of things that I have some part in certainly monthly if not weekly:

1. Database-level backup and restore
   • Designing backup strategy based on resources and restore requirements
   • ad-hoc restores frequently needed during build, patching, etc
   • managing retention of backup images and transaction logs 
2. Data movement
   •  Between commerce databases on an ad-hoc basis
   • advising and assisting with data movement to/from other systems
   • Stagingcopies or restores in support of stagingprop
   • assisting in automation of regular data movement processes
3. Setting up and refining monitoring and alerting
4. Responding to alerts related to overall outages, database specific outages, filesystem utilization, and failures of database maintenance
5. Setting up and monitoring regular database maintenance including runstats, reorgs, and a few other niceties
6. DBCLEAN and data pruning
   • Designing initial implementation of data pruning including discussion of each of at least a dozen pruning areas with development and business people
   • Monitoring ongoing pruning, and correcting failures
   • Looking out for new pruning areas 
7. Stagingprop
   • Assisting with initial setup, possibly with use of stagingcopy
   • Heavy involvement in resolving errors which happen for every client at some time with some weeks spending 50% of my time on this 
   • Creating/dropping/altering stagingprop triggers as needed
8. Database object creation (working with developers on best practices and standards – sometimes writing the code myself)
   • Tables
   • Indexs
   • Stored procedures
   • Functions
   • Triggers
9. assisting in writing SQL (DML)
10. Executing SQL, particularly on Production environments where developers may not have access (per PCI standards!!)
11. Performance analysis, both proactive and reactive
    • Physical db2 parameters and resources – identifying problems and correcting them
    • Identifying and analyzing problem SQL and communicating about it and identifying and creating indexes to help resolve 
12. Implementing and supporting HADR and other High-Availability solutions
13. Managing security including:
    • Granting and revoking privileges and maintaining a consistent approach to doing so
    • Being aware of various items related to security and incorporating them into both new builds and existing databases
    • Assisting with DB related issues for PCI compliance 
14. Training new DBAs as well as non-DBAs on DB related topics
15. Resetting configadmin’s password
16. Assisting with Merchant Key Changes
17. Disaster recovery when needed from a wide variety of disaster situations
18. On an ad-hoc basis communicating with whoever asks on a  variety of these topics
19. Architecting strategy for all areas of database support

Whew. That’s  a mash-up of “System” or “Physical” DBA tasks and “Logical” or “Development” dba tasks – which some of the larger companies keep separate. I actually like to do them both so I really own MY DB2 databases and know them as well as possible.

The point is that there is a significant amount of database specific work in supporting a WebSphere Commerce implementation, and an experienced DBA is nice with a fast-learning DBA being a minimum.

I started my career with 7 years of “Physical” DB2 DBA work(the last year or two of that was mostly Commerce-specific), and am completing my 3rd playing a mixed role for Commerce databases specifically (with ESB, Portal and a few other misc databases thrown in for good measure). And now an interesting twist – I’m starting to learn Oracle too. So I’ll probably do some posts about that as well.

I truly love how there’s always something new to learn

Just a quick tip. When building Commerce environments, select a different database name for each environment (Stage, Prod, etc), even if they are on different database servers. This will help you ensure that you(or developers or whoever else accesses the databases) never do something in the wrong environment. It adds a bit of complexity when you restore between different environments, but nothing that can’t be cured in the restore command. It will also help you know what database backups come from what environments if you end up with backup images in the same directory.

Choose a database name that is descriptive and meaningful to you (as much as you can in 8 characters).

Schedule DB2 backups starting immediately after instance creation – the most common time that restores are needed is during the build process. Of course the most common cause for needing a restore is human error, but that’s something that is important to protect against during build.

So the easy thing to do on Commerce build is to use your DB2 instance owner for everything related to the database. But that’s not really the best choice. It’s almost always the choice I see when a DBA was not involved with the architecture or build of a Commerce system. You’ll notice in the instance XML that there is an option to specify a separate DBUser and DBAName. Generally, it is best to have different values for these. The DBAName is the instance owner, but the DBUser name is a dedicated user for Commerce to use to connect to the database.

The DBUser should have its own ID. This is the ID that commerce will use to connect to the database going forward. What it really needs is Select, Update, Insert, and Delete (SUID) on all tables. If set up when the system is built, then it will also be the primary schema name used. During upgrades, you can add it into the SYSADM group too, so it can create tables and other objects as needed. But remove it after that – it does not need that level of permissions all the time. Why separate this? Well, it is best to address potential security threats at every level. I frankly don’t know what protections Commerce puts in place against SQL injection, but if the id that Commerce is using doesn’t have the authority to drop anything, then that’s so much more protection against the potential threat. If PCI (DSS) compliance is a concern, this is also necessary.

What other IDs to use?

Do not share the password to the ID that commerce uses to access to the database with developers. That means that each developer has their own ID that belongs to a group that has SUID on all Commerce tables. Do not let developers create or drop objects – we expect them to work through a DBA for that so we can apply basic sanity checks. This also puts you on a good footing for PCI compliance.

Use a dedicated ID for stagingprop(and stagingcopy). The id needs SUID on all tables listed in STGMERTAB and STGSITETAB and STGMRSTAB along with the STAGLOG, KEYS, and CMFILE tables. We also create aliases on these tables – with the schema the same as the stagingprop user. If you don’t do this, you’ll get SQL0204 (Exception: com.ibm.db2.jcc.c.SqlException: DB2 SQL error: SQLCODE: -204). Commerce does not specify the schema when doing anything. Stagignprop allows you to specify the dbschema to get around this, but stagingcopy does not, so these aliases are a good idea.

Use dedicated IDs for data load, misc. batch jobs, and search, with permissions granted only explicitly as needed by the ID for the processes it is running on that particular server. This sometimes requires working through permissions one by one as failures are encountered, which can be

One last dedicated group is one that specifically has select access only on all tables – with membership severely restricted on production. This can be useful for people who should not be updating data. But on production there are two issues – #1 is access to confidential or personal data and #2 is the fact that a bad read-only query can tank database performance – something you don’t want to allow to happen on production.

What to revoke

One last thing to consider is one of DB2’s Default groups – PUBLIC. This group means “anyone who can log into the database server” (assuming you are using server authentication – please, never use client authentication). Rarely do you really want to allow anyone who can simply log into a server to access your database, even if it is a dedicated db server. By default it has CONNECT on the database, and select on all the system tables (db2 system tables – the data dictionary type stuff). Revoke at least CONNECt from it, and consider revoking other permissions as well. I like to leave package permissions in place because PUBLIC is very useful when doing BINDs. But when you revoke the CONNECT privilege, you change the meaning of the group to “anyone who can connect to the database” – a much more restricted and useful group.

Is it easiest just to use the db2 instance owning id for everything? Maybe, but it is not PCI compliant, and you may end up with issues with people doing something they shouldn’t. Proper division of IDs will lead to a system that runs more smoothly and reduce security concerns.

What is HADR?

HADR is DB2’s implementation of log shipping. Which means it’s a shared-nothing kind of product. But it is log shipping at the Log Buffer level instead of the Log File level, so it can be extremely up to date. It even has a Synchronous mode that would guarantee that committed transactions on one server would also be on another server. (in years of experience on dozens of clients, I’ve only ever seen NEARSYNC used) It can only handle two servers (there’s no adding a third in), and is active/warm spare – only with 9.7 Fixpack 1 and later can you do reads on the standby and you cannot do writes on the standby.

How much does it cost?

As always verify with IBM because licensing changes by region and other factors I’m not aware of. But generally HADR is included with DB2 licensing – the rub is usually in licensing DB2 on the standby server. Usually the standby server can be licensed at only 100 PVU, which is frequently much cheaper than full DB2 licensing. If you want to be able to do reads on the standby, though, you’ll have to go in for full licensing. Usually clients run HADR only in production, though I have seen a couple lately doing it in QA as well to have a testing ground.

What failures does it protect against?

HADR protects against hardware failures – CPU, disk, memory and the controllers and other hardware components. Tools like HACMP and Veritas use a shared-disk implementation, so cannot protect against disk failure. I have seen both SAN failures and RAID array (the whole array) failures, so it may seem like one in a million, but even the most redundant disks can fail. It can also be used to facilitate rolling hardware maintenance and rolling FixPacks. You are not guaranteed to be able to keep the database up during a full DB2 version upgrade. It must be combined with other (included) products to automatically sense failures and fail over.

What failures does it not protect against?

HADR does not protect against human error, data issues, and HADR failures. If someone deletes everything from a table and commits the delete, HADR is not going to be able to recover from that. It is not a replacement for a good backup and recovery strategy. You must also monitor HADR – I treat HADR down in production as a sev 1 issue where a DBA needs to be called out of bed to fix it. I have actually lost a production raid array around 5 am when HADR had gone down around 1 am. Worst case scenarios do happen.

How to set it up

HADR is really not too difficult to set up on it’s own. Configuring automatic failover is a bit more difficult, though DB2 has made it significantly easier in 9.5 and above with the introduction of bundled TSA and the haicu tool. I’m not going to list every detail here because there are half a dozen white papers out there on how to set it up. The general idea is:

1. Set the HADR parameters on each server

HADR local host name                  (HADR_LOCAL_HOST) = your.primary.hostname 

HADR local service name                (HADR_LOCAL_SVC) = 18819

HADR remote host name                (HADR_REMOTE_HOST) = your.secondary.hostname

HADR remote service name              (HADR_REMOTE_SVC) = 18820

HADR instance name of remote server  (HADR_REMOTE_INST) = inst1

HADR timeout value                       (HADR_TIMEOUT) = 120

HADR log write synchronization mode     (HADR_SYNCMODE) = NEARSYNC

HADR peer window duration (seconds)  (HADR_PEER_WINDOW) = 120

2. Set the Alternate Servers on the Primary and the standby (for Automatic Client Reroute)

3. Set db configuration parameters INDEXREC to RESTART and LOGINDEXBUILD to ON

4. Take a backup (preferably Offline) of the database on the primary server

5. Restore the database on the standby server, leaving it in rollforward pending state

6. Start HADR on the standby

7. Start HADR on the primary

8. Wait 5 minutes and check HADR status

9. Run db2haicu to set up TSA for automated failover

10. Test multiple failure scenarios at the app and database level

For chunks of this, your database will be unavailable. There are also a number of inputs you need to have ready for running db2haicu, and you will need ongoing sudo authority to execute at least one TSA related command.

Remember that the primary and standby servers should be as identical as possible – filesystems, hardware, and software.

Some clients also neglect step #10 – testing of failovers. This is an important step to make sure you really can failover. It is possible to think you have everything set up right, do a failover and then not have it work properly from the application’s perspective.

Gotchas

This section represents hours spent troubleshooting different problems or recovering from them. I hope it can help someone find an issue faster.

HADR is extremely picky about its variables. They must be exactly right with no typos, or HADR will not work. I have, on several occasions had numbers reversed or the instance name off, and spent a fair amount of time looking for the error before finding it. Because of this, it can help if you have another dba look over the basics if things aren’t working on setup. HADR is also picky on hosts file and/or db2nodes.cfg set up, and in some cases you may end up using an IP address in the db cfg parameters instead of a hostname.

HADR also sometimes fails after it tells you it has successfully started, so you must check the status after you start it.

Occasionally HADR doesn’t like to work from an Online backup, so an Offline one will be required. I have one note about it not going well with a compressed backup, but that was years ago, and I frequently used compressed backups without trouble.

HADR does not copy things that aren’t logged – so it is not a good choice if you have non-logged LOBs or if you do non-recoverable loads. If you are using HADR and you do a non-recoverable load, you have to take a backup on the primary and restore it into the standby – if you don’t, any table with a non-recoverable load will not be copied over, nor will future changes, and if you go to failover, then you will not be able to access that table. For this reason, I wouldn’t use it in a scenario where you don’t have good control over data being loaded into the database. If you do run into that, then you have to backup your primary database, restore it into your standby database, and start HADR.

HADR does go down sometimes without warning – so you must monitor it using whatever monitoring tools you have, and ensure that you respond very quickly when it goes down. I use db2pd to monitor(parsing output with scripts), partially because db2pd works when other monitoring tools hang. We look at ConnectStatus, State, and LogGapRunAvg.

On reboot, HADR comes up with database activation. Which means it usually comes up just fine on your primary database, but not on your standby database (no connections to prompt activation). So you’ll generally need to manually start hadr on your standby after a reboot. The primary database will not allow connections on activation until after it can communicate with the standby. This is to prevent a DBA’s worst nightmare – ‘Split Brain’. DB2’s protections against split-brain are pretty nifty. But this means that if you reboot both your primary and your standby at the same time and your primary comes up first, then your primary will not allow any connections until your standby is also up. This can be very confusing the first time or two that you see it. You can manually force the primary to start if you’re sure that the standby is not also up and taking transactions. Or if you’re rebooting both, just do the standby first and do the primary after the standby is back up and activated. If you need your standby down for a while, then stop HADR before you stop the servers. I would recommend NOT stopping HADR automatically on reboot, because the default behavior protects you from split-brain.

What is split-brain? It is simply both your primary and standby databases thinking they are the primary and taking transactions – getting you into a nearly impossible to resolve data conflict.

You must keep the same ids/groups on the primary and standby database servers. I’ve seen a situation on initial set up where the id that Commerce was using to connect to the database was only on the primary server, and not on the standby server, and thus on failover, the database looked fine, but Commerce could not connect.

You also want to be aware of any batch-jobs, data loads, or even scheduled maintenance like runstats or backups – when you fail over, you’ll need to run these on the other database server. Or you can also run them from a client that will get the ACR value and always point to the active database server. Frequently we don’t care which database server the database is running on, and may have it on what was initially the “standby” for months at a time.

Overall, I really like HADR and it’s ease of administration. The level of integration for TSA in 9.5/9.7 is great.

On Linux and Unix anyway. This is actually a common request during build when OS-level patches are applied or other OS-level work needs to be done, especially any related to disks.

Stop Commerce (and/or any other applications)

I’m not going to go into detail on this one – it is not my area of expertise. If I end up doing it myself, I ask the WebSphere Commerce Admin how.

Stop the db2 instance

First log in to the database server as the db2instance owner(su – is fine if you’re coming from root). This is the id you specified as the DBA on commerce build. Frequently the default is db2inst1.

Try stopping db2 cleanly using simply

$ db2stop
01/05/2011 21:14:41     0   0   SQL1064N  DB2STOP processing was successful.
SQL1064N  DB2STOP processing was successful.

This will fail if there are any connections left. If it does fail, do:

$ db2 list applications

This provides a list of connections to your database. If there are many, go back and make sure you stopped all applications properly. For any remaining connections, make sure there’s no reason you can see that forcing the connections off will fail. Then try this:

> db2stop force
10/01/2008 22:11:53     0   0  SQL1064N  DB2STOP processing was successful.
SQL1064N  DB2STOP processing was successful.

If that doesn’t succeed, there’s one more drastic command you can find that will work, but it’s rare that db2stop force doesn’t work. It may take a bit longer on an active database, as it has to make any remaining connection roll-back their units of work.

Stop the DB2 Administration Server (DAS)

I rarely actually use the DAS, but I always create it on build so it’s out there if I or anyone else needs it. To stop it, you must login as the DAS owner or su – to the DAS owner, and issue the following:

>db2admin stop

I’ve never seen that one fail

For all instances including the DAS, stop the db2 fault monitor

As the das owner (or su’d to it), issue:

db2fm -i <dasowner> -D

ignore any failures, as the failure usually just indicates that the fault monitor was not running in the first place.

As the instance owner (or su’d to it), issue:

db2fm -D

ignore any failures, as the failure usually just indicates that the fault monitor was not running in the first place.

Comment out db2 entry in /etc/inittab and kill any remaining processes

So there’s always one, and sometimes many processes hanging around at this point. One of these is owned by root, and is actually the result of a respawn in inittab. The line looks something like this:

fmc:2:respawn:/usr/opt/db2_08_01/bin/db2fmcd #DB2 Fault Monitor Coordinator

Comment out that line, and (if needed) refresh inittab in memory (I think this varies by OS).

The line above is obviously from DB2 version 8, but the version 9 ones are similar. This is a line added automatically by db2 on install, so even if you never thought it was there, it probably is.

Only after you have done that will the processes stay killed. At this point it is safe to issue kill -9 commands on any remaining db2 processes.

And db2 should be completely down. Might be a bit basic for some, but useful for some who are either new to DB2 or have only been on the logical side before.

So I’m doing a lot of posts in the DB2 Basics area this week. That’s because they are quick for me to write, and they’re questions I get all the time. So bear with me if you read for the more detailed Commerce topics – I’m not abandoning them.

DB2 Client

DB2 databases can be cataloged on DB2 servers or DB2 clients. Before trying cataloging, make sure you at least have the DB2 client installed. DB2 Clients are free (no licensing required). All App servers should have DB2 clients already installed on them. Web servers should not. You can download DB2 Clients here: https://www-304.ibm.com/support/docview.wss?uid=swg27007053

Select the same version and FixPack as the servers you’re connecting to. Then select the proper OS. Finally, select “IBM Data Server Client” (9. versions) or “Administration Client” (8. versions). Installation and Instance creation are outside the scope of this post.

Make sure you have all the inputs

You need:

• REMOTE: Short host name with proper entry in hosts file OR Fully qualified host name OR IP address (that is the order of preference if you have all)
• SERVER: Port number the db2 server is listening on. To get this, log into db2 server as the db2instance owner (frequently db2inst1), and do:

$ db2 get dbm cfg |grep SVCENAME

TCP/IP Service name                          (SVCENAME) = db2c_db2inst1

$ cat /etc/services |grep db2c_db2inst1

db2c_db2inst1   50001/tcp

• NODENAME: This is a name that you make up.  You might want to develop a standard for your company and stick to it.

Catalog the node

The general form of the syntax for doing this is:

db2 catalog tcpip node <NODENAME> remote <REMOTE> server <PORT>

For example:

Logged in (su – is fine) as the db2 instance owner or a privileged user:

$ db2 catalog tcpip node prod_ecom remote servername.domain.com server 50001

Catalog the database

The general form of the syntax for doing this is:

db2 catalog database <database_name> at node <node_name>

For example:

Logged in (su – is fine) as the db2 instance owner or a privileged user:

db2 catalog database wcs_prod at node prod_ecom

Refresh directory

After cataloging, do a db2 terminate to ensure everything shows up

$ db2 terminate

DB20000I  The TERMINATE command completed successfully.

Always test a connection!!

It is very important to test the connection because a typo could cause it not to work. Or there could be network issues preventing connection.
The general form of the syntax for doing this is:

db2 connect to <database_name> user <user_name> (you will then be prompted for a password)

For example:

Logged in (su – is fine) as the db2 instance owner or a privileged user:

db2 connect to wcs_prod user ecrooks

Enter current password for ecrooks:

Database Connection Information

Database server        = DB2/LINUX 8.2.9 
SQL authorization ID   = ECROOKS 
Local database alias   = WCS_PROD