HP Discover

Another year rolls around and HP Discover 2012 approaching. While its uncertain if I'll be attending, for reasons I won't wallow in, I will reflect on some of the highlights and concerns from last years event.

Certainly stealing the show was Paul McCartney and a performance that left me a new fan, yes I'm sort of doing this backwards chronologically.

Stepping back again, the venue, The Sands Conference center was all together very good, although chasing sessions around at the last minute and trying to get into rooms that I had signed up for was a bit difficult. It seemed information about session changes was tough to come by, even at a technology based event. I was disappointed more than once over having sessions moved, canceled or outright different from what was on the schedule. All of these were HP mainstream labs or presentations, none were HP NonStop sessions, Disappointing even more because of the limited nature of the offerings.

The HP Certified Professional folks, now HP ExpertOne, were excellent as always. Providing a great reception suite, news and information about the program going forward. Rich Gossman and his team were on target and provided a great experience for HP ExpertOne professionals. The only unfortunate blemish involved on-site testing for credentials, despite signing up for testing prior to the event, the testing team seemed overwhelmed, the suite too small the registration process was cumbersome and the whole testing process seemed very difficult compared to previous years. To be fair changed have been made to this process going forward, so improvements should be realized.

General sessions were glitzy and splashy as expected, but well attended and organized. The usual offerings from the now departed former CEO were of the kind level one would expect for a show this size. Little to nothing was mentioned about NonStop. Typical though, from a company concerned with volume.

The exhibits and tradeshow, while impressive from an HP viewpoint, included things like portable datacenter pods, the usual enormous displays from Microsoft loads of color and vast swaths of products targeting the majority of the PC and printer lines, as expected. The NonStop area seemed to be relegated to the ugly stepchild kiosks with little visibility and it took some time to locate in the sea of servers. True, it's a smaller volume of HP's business and granted many of these vendors are not interested in toting their tradeshow exhibits to Las Vegas at their cost, only to answer questions and hand out give-aways to a Linux or Windows professional about NonStop when they're really looking to speak with NonStop people. We've come a long way from the usual ITUG events, Regional User Groups and the future still seems uncertain. Especially with CEO's shuffling chairs, news of divisions consolidating within HP and we're left wondering what is HP doing with NonStop?

I'm not sure there is still value in large events like HP Discover, considering the cost, but I do appreciate the offer of $300 bucks off the registration fee for being a Connect member but I already get that for being an HP ExpertOne too and no doubt I'll get a "special" registration code from our HP Account team to register. This leaves me wondering what the real value is or if there is any value left.

The most important part of these events, to me, is seeing friends and colleagues and networking, sharing and comparing. This is true for NonStop product vendors and VARs. I always enjoy talking to folks about new products and ideas. A smaller event may fill that need? Perhaps, the NonStop Symposium in San Jose was apparently well received and attended, but no mention of any further events like that one.

Yes, we all secretly hope HP will spin-off NonStop, perhaps even call it Tandem again. But we all know that's not going to happen with a company like HP. We can hope that HP will someday realize the value of NonStop and it can come out from the shadows and smudges of printer ink.

-K

- Posted using BlogPress from my iPad

Availability is about BEING available

Another excellent resource focusing on the practices of Availability and what this means is The Availability Digest by Dr. Bill Highleyman.

His site focuses on the processes and practices, not a platform. 

Although many Tandem/NonStop users know Bill as the guy who wrote the book, literally, on Performance Analysis and Tuning for OLTP systems, his digest gives food for thought (no pun here) about the process of establishing and maintaining availability as well as some pretty interesting "never again" horror stories.

I invite you to take a look at http://availabilitydigest.com/ and sign up for his digest if you see something that sparks your interest there.

I've provided a link from this page, on the right side control bar, to this and other sites of interest. 

-K

Good read for NonStop Users and Mission Critical Fans

A quick post to invite you to take a look at well established blog by a good friend Richard Buckle, longtime Tandem Evangelist and Founder of Pyalla Technologies.

I had the opportunity to serve on the Board of Directors for the International Tandem Users Group, (or ITUG as many know it's former life, now called Connect) with Richard and his wife Margo. I learned a great deal from Richard and Margo and appreciate the opportunities we've shared.

Richard has some excellent writing about the past, present and future of NonStop and Mission Critical Computing.

His blog is at http://itug-connection.blogspot.com/ , and provides valuable insight on mission-critical computing.

-K

Switch and Stay on the new Database - Loading files using GoldenGate

The advantage of having scripts to execute the switch to new files online cannot be overstated. Manually typing the commands and queries to stop, rename and restart processes accessing the files really opens the process up to errors.

These files are generally accessed via database server processes, either TS/MP Pathway managed or the occasional individual generating reports with data from the files. This gave us the ability to 'interrupt' file activity with a relatively small impact.

The high level steps of the script involved, stopping database servers, stopping any ad hoc processes, renaming the current primary file to old primary file, renaming the new primary to the online primary file. Decision to leave the new alternate key file with its created name was so that fewer steps would be involved and since the alternate keys map from the file system it would not be a problem going forward. If  needed the ALTER of the alternate key file could be performed later at a planned downtime.

Keep in mind the intent here was to minimize impact to database access, add new alternate key functionality for application use and avoid a planned lengthy system downtime to load and switch to new primary and alternate key files.

The switch process for each set of database files took approximately 5-10 seconds using the script. At this point the new primary file with associated alternate keys was swapped in and the database server recovered. Similar but not exactly like a Base24 log rollover.

The current GoldenGate configuration also facilitated the swap on the Target site as it executed file renames with the GETFILEOPS parameter enabled on the Target. Essentially keeping us in sync across source system and target.

After all files were switched and the database functions were recovered, it was necessary to stop and delete the temporary Extractor and Replicator processes that had been configured to load the new files, they were no longer needed.

Then, a Veridata compare configured to evaluate the new files indicated the New Source and New Target were in Sync, except for one missing 107 records. To recover from this, the Extractor was repositioned on the Extract Trail to a point just a few minutes prior to the swap and restarted with HANDLECOLLISIONS ON. The records were applied and the file was checked again and found to be in sync across Source System and Target System.

Considering time required, the Initial file loads were performed 2 weeks prior to the switch. This enabled appropriate notification, change management and design of switch scripting. The loads were all performed online, during normal working hours and validated without impact to the user community. The file switches were performed without perceived impact from the user community, during normal business hours and using automated scripts in a few seconds.

"Your actual mileage may vary...."

Typically in past operations like these, it required a system downtime, use of FUP to CREATE, LOAD and LOADALTFILE during the outage. Considering the size and number of alternate keys to be added to theses database files the entire process would have had to be either an extremely lengthy downtime or several downtimes to perform.

By using GoldenGate to Load and synchronize the files, the majority of the effort could be focused on distilling the switch scripts to the smallest possible window and time dedicated to validating file integrity before and after the switch.

Again, your conditions and environment may vary, but reducing impact, eliminating a downtime and providing 100% record integrity before and after made GoldenGate an excellent choice for performing an online alteration to these database files.

If you'd like to know more, have questions, comments or I haven't covered something here, drop me a note.

Now I'm off to find another topic of NonStop interest.

-K

Verification of Large Enscribe file loads - Part 2 of Loading Files

Once the initial data pumps had completed the loading of the new files and alternate key files, a special extract/replicat that had been configured on the source system, was started to apply all changes captured during the initial load, this took several minutes to apply the data to the new files.

Next up how validation of synchronization was accomplished.

There is definitely an advantage to using tools designed for validation of file synchronization. In this case Oracle/GoldenGate Veridata is being used to validate ongoing replication and file synchronizations.

For those installations without Veridata tools, you will probably need to craft your own toolset or think about obtaining Veridata, it makes the entire process so much smoother. (I don't work for GoldenGate or Oracle and this is not an endorsement of any product.)

Veridata compare pairs were created for source to new source as well as source to new target files. This process took about an hour of work building the compare sets and execution of the comparisons varied from just a few minutes to about 34 minutes to compare roughly 41 million records.

All of the validations, following the initial data pump and subsequent change capture application, were successful indicating the new files with new alternate key files were in sync.

At this point it was a matter of scheduling a switch window and build cutover scripts to perform file renames and process shutdown/startup to cutover to the new files.

Next, I'll discuss the methods used to switch, specific to this installation and the impact expected and realized...

- Posted using BlogPress from my iPad

Loading Large Enscribe files, using GoldenGate - Timing estimates

I failed to mention the load times in the previous post, it took some time to collect the data so bear with me. 

The files being loaded ranged up to 10 Gb in size. By using GoldenGate Direct Data Pumps to load the files and their alternate key files, the largest file took approximately 10 hours, but keep in mind this was being performed during normal processing without impacting the current configuration. 

Smaller files ranged in the 100's of megabytes and took approximately 2-4 hours depending on size. All files were being changed from 2 alternate keys to 8 alternate keys. 

Time was not a critical factor since we knew well ahead that this change was needed and didn't fall into the urgent category. 

All of the loads were started after the peak processing time and allowed to run thru the night/early evening. This was a perfect time, although observation showed that GoldenGate tends to "play well" with other processing in our environment. 

Part two coming up describing file load completion and file validation....

Initial loading using GoldenGate - Part 1

The files being converted ranged from several hundred megabytes to over 10 gigabytes for the primary file. Adding more alternate keys automatically called for a larger alternate key file, in some cases larger than the primary file, due to the nature, size and number of keys contained within.
Sizing studies were done to estimate, design and create the correct file type (Type 2 in this case) size and extent specifications.

Once these tasks were complete the files were created on the Primary or Source system and available for data loads. GoldenGate replication immediately created identical files on the Target or DR site.

Initial load was accomplished by configuring a GoldenGate Extract/Replicat Data Pump to read directly from the Source file(s) and write to the new primary and alternate key files. This method used Enscribe to generate the new alternate keys as the new primary file was being loaded.

GoldenGate provides the ability to load files unattended or "no wait" and provides restart capabilities if the load stops or fails to complete. Two big advantages while loading multiple large files concurrently, since the data load can be setup to run at times of low system activity i.e. evening or overnight.
Additionally, the ability of GoldenGate to use TMF Audit Buffers by turning on the Audit flag for the target files, greatly speeds up the process of loading using the direct data pump method.

Previous loads using FUP were subject to failure in the event a dynamic terminal session window closed unexpectedly or a file extent allocation failure, and since FUP doesn't possess restart capability on a file load, GoldenGate was the better choice.

Data pump loads were performed from the Source files to the new Source files on the Primary system first, to avoid any contention or locking the Target files were initialized after all of the source files had completed initial loads.

While the loads were executing, no user impact was observed. Performance of the primary systems was normal and not impacted by the file loading.

During the file loads, database operations were always being captured and replicated to the Target site, this these changes captured during the load were then applied to the new primary and alternate key files after the initial load was finished. At this point both the source/new source and the target/new target files were all in sync, the differences at this point involved the alternate key files for each database file, in that source and target alternate key files only contained 2 alternate keys values and new source and new target alternate key files had 8 values. Success.

Next I'll examine verification and validation procedures before performing the file switch.

File builds in preparation for conversion

The files that were to have alternate keys added were all archived data, required by various applications and regulations to be online and available.
Many of these files were either Enscribe type 2 or were about to become candidates for a type 2 file because of the size, some contain data dating back to the late 70's early 80's and are several gigabytes of storage.
We created a template for the files, using FUP, and set the required alternate keys adding 6 new keys instead of the previous 2.
When file was created on the Primary site, we already had GETFILEOPS enabled for the current GoldenGate replication environment so the new files, primary with new alternate keys and the larger alternate file were created automatically on the Recovery or DR site.
A quick check of all files on both systems confirmed that we had new files ready for loading the data.
Next up I will describe how we loaded these files, discuss how long it took to move data and the benefits of using GoldenGate to perform the load.
- Posted using BlogPress from my iPad

Using Oracle™ GoldenGate to add alternate key files/values

This started as an alternative to a long and highly impactful application downtime.

The problem initially was the need to add alternate keys for application use, on the order of 4 times the key values.

Additionally, the issue of the sheer size of the primary file, Type 2 Enscribe and several gigabytes of archived data that needed to remain available. To reduce the outage window to the smallest possible amount of time, it was decided a copy of the primary file, with the additional alternate key definitions be created, then loaded with a direct data pump using GoldenGate to load both the primary and  alternate key files.

Advantages included being able to use restart capabilities of GoldenGate and using TMF Disk Buffers to expedite the load process.

Files were already replicated to a Target Site, so in addition to duplicating the primary and alternate keys at the Primary Site the process also called for duplicating the activity at the Target.

Source data generated by the applications was already being captured, extracted and replicated so it was a simple matter of building new file structures, performing the initial load of of primary and alternate key files, then enabling change capture until the switch could be scheduled at the least impactful time.

Next I'll explain the details of creating and loading the new file structures with the initial GoldenGate data pump....