Planet Network Management

Not too many years ago I ran a large network for the United States Air Force. For the first several years I was there we had an old copper-wire backbone and it was amazing to see the havoc that weather could wreak upon that network. Sometimes you practically...(read more)

Yesterday I, along with a large number of other people, watched the New Orleans Saints win their first Super Bowl.

Last year I watched the game in a hotel room in Milan (when my Steelers won their sixth championship – number 43 we miss you) but this year I was able to hold a little party.

Now, like hot dogs go with baseball, pizza is fast becoming the food of choice for football, and no Super Bowl party would be complete without some Papa John’s.

Of course, my Mom also showed up (that’s her behind the pies – “Hi mom”) and she brought enough food to feed an army. I think everyone left stuffed.

Anyway, Denise Dubie at Network World wrote an article about how Papa John’s uses OpenNMS, and how they delivered 6 million slices of pizza yesterday.

They worked so we didn’t have to (well, except for Mom. Thanks Mom).

Somewhere in a land a long time ago, there was this steel factory in the heartland of Pennsylvania.  In this steel factory there was a bulletin board on a factory floor room, and on the board were two large posters.  The first poster was a reminder of the upcoming layoffs on Friday with procedures for filing paper work and such.  The other poster was an announcement of the new IBM mainframe computer that was arriving that week.  This new marvel of modern technology just happened to be arriving on that same Friday.  The IBM poster encouraged employees to gather and witness this historic event, and that they did.  So when Friday rolled around, the poor IBM’rs with their blue suites and dark ties were meet with harsh resentment.  The factory floor workers harassed them unmercifully, and in some cases they were actually spat on.  I take it you see the irony of the poor steel workers assumed correlation between the new computer and the layoffs.  It’s easy to see now that the new “computers” had nothing to do with the layoff… Or is it?

Read the complete post…

Somewhere in a land a long time ago, there was this steel factory in the heartland of Pennsylvania. In this steel factory there was a bulletin board on a factory floor room, and on the board were two large posters. The first poster was a reminder of the upcoming layoffs on Friday with procedures for filing paper work and such. The other poster was an announcement of the new IBM mainframe computer that was arriving that week. This new marvel of modern technology just happened to be arriving on that same Friday. The IBM poster encouraged employees to gather and witness this historic event, and that they did. So when Friday rolled around, the poor IBM’rs with their blue suites and dark ties were meet with harsh resentment. The factory floor workers harassed them unmercifully, and in some cases they were actually spat on. I take it you see the irony of the poor steel workers assumed correlation between the new computer and the layoffs. It’s easy to see now that the new “computers” had nothing to do with the layoff… Or is it?

My first real experience with this debate actually occurred around 1987 when I created my first startup. A friend of mine and I had created an automation tool called Sybridge. This product was designed to help operators automate the mundane tasks they needed to perform throughout the day. Back in the day, computer operators working with large IBM mainframes had to enter all sorts of rote command sequences to perform their daily tasks. In 1987, the only programming interface for operators and system administrators was a “green” screen terminal. My friend and I came up with a clever way to screen scrape these terminals allowing operators and systems administrators to create “Rexx” scripts to automate these tasks. For example, back then the only way to enter a problem ticket into a problem management system was to manually enter the data into one of those unforgiving “green” screen monsters. With our Sybridge product, we could capture an online transaction (e.g., CICS transaction) and automatically enter the information into the problem database. Great idea, right? Actually, we spent a lot of time and money doing market analysis on the fear perception that operators might not think this was a great idea. At that time there was a big debate going on about whether automation products would replace humans. However, by the turn of that decade automated operations products were a ubiquitous main stay in almost every fortune 5000 enterprise IT shops. In fact, most of the “good” operators became automation analysts and scripting freaks. The operational productivity of almost every enterprise running IBM mainframes increased tremendously. At least until distributed computing snuck it’s way into the data center.

So I had to chuckle the other night when this same subject of infrastructure automation and people losing jobs came up at the Seattle Cloud Camp. During one of the breakout sessions about Opscamp, a lengthy discussion came about regarding infrastructure automation products like Chef and Puppet and the idea that such products might replace system administrators. Ironically, I was able to use an antidotal example from a conversation I had just a few weeks prior with a friend of mine. The discussion I had was about managing automation products for a large Fortune 1000 company. This friend of mine was a systems administrator who managed a number of proprietary IT management products. The conversation started by him asking me if I could recommend any open source products as replacements for the proprietary ones. He was currently using some of the larger vendor (Big Four) type products and was thinking about a change. I gave him my usual suspects, a list of my favorite operations tools. Then he said “Yeah, but management doesn’t feel comfortable with open sources.” At this point, I lost it. I told him, “Enough with the open source and management. It’s not about open sources versus proprietary, it’s about top line value." I told him. Then I said, “You have to ask what tools allow you to effect the business top line value for your company?” I had guessed that he spent at least 60% of his day working on the tools trying to fix them and make them work effectively, and less than 40% of his time providing real business value from the tools for his company. He said “John, you are wrong… I spend 80% of my time working on the tool and less than 20% of my time providing value.” That’s what you need to tell management, not OSS vs non-OSS.

The bottom line is that in my 30 years of working in this IT muck, I have never seen an example where automation eliminated valued employees. Automation, however, can be a used as an opportunity to clean up some dead weight. Automation, more often than not, makes good employees better and flushes out the bad ones. Therefore, as these topics of automated infrastructure, infrastructure as code, agile operations, and devops slip into the IT lexicon, we all need to be aware that good system administrators will typically agree that automation does not replace humans, whereas bad system administrators will beg to differ.

Who Do You Browse With?

I wrote a blog nearly a year ago expressing my mistrust in Google Chrome and asserting my position that when Chrome became available for Mac (my preferred computer) I would not be jumping on that bandwagon.

Well, what a difference nearly a year makes. I did stay somewhat true to my word and I was not the first, the second, or even the thousandth to jump on the Mac compatible Chrome bandwagon last spring. But jump, I eventually did.

The bugs I encountered a year ago; the incompatibility with Google’s own commodities such as YouTube and Gmail, had been worked out. The browser offered a variety of sleek new skins with which I could customize it and the load time for my favorite sites saw a noticeable, albeit not huge, improvement from Firefox.

And according to the numbers released by W3School’s Browser Statistics Month by Month last week, I wasn’t the only internet user to take notice of Chrome’s advancements: Chrome was used for 10.8% of visits to W3School site in January 2010. A solid 3^rd place after Mozilla Firefox (46.3%) and Microsoft Internet Explorer (36.2%). 

Out of curiosity I used Google Analytics to get an idea of how the browser war was playing out with the visitors of WhatsUp Gold’s external blog www.dailynetworkmonitor.com.

 

Sure enough, there was a marked upswing in visitors using Chrome to come to our blog in January as well. Our stats for January were about the same as W3School’s showing. Chrome rang in at 10% whereas Firefox held 33% of the use and Internet Explorer 51%.

But what was more interesting to me – and more impressive proof of this market trend – was the jump in Chrome use between November and December. In November Chrome barely showed up in our analytics. It was used for .03% of the visits, just coming in under Safari’s .035% share. Firefox counted for 22% while Internet Explorer dominated at an impressive 69%.

But December brought a noticeable shift; Chrome suddenly jumped to 7% of the use while Safari stayed around the same. Firefox stayed around the same overall percentage where Internet Explorer saw a drop off to 54%.

It’s my opinion that Chrome’s market increase is largely due to frustration with Internet Explorer. Security issues aren’t anything new with the browser, but renewed concern over it’s security led the French and German governments to advise people to switch browsers last month.

While IE still dominates the browser space, its market share has seen a steady decline; from 68.5% last March to 62.12% in January. Firefox hasn’t seen too much change in their market share over the last year; Mozilla’s browser suffered a one percent drop from their 23.30% in March mid year but caught back up to 24.43% last month. Chrome, however, held 1.62% in March 2009 and can now claim 5.22%. Google’s browser owes most of its jump to OS X users who did not have access to it until last spring.

I’d like to see Google Chrome’s market share continue to grow and challenge Microsoft to create a more user and security friendly Internet Explorer. While in corporate terms the Internet giant and computing czar are on a level playing field, Google’s Internet browser has a ways to go to catch up to Microsoft’s institution.

Google’s scrappy little browser could be the motivation Microsoft needs to create a level browser one should be able to expect from a company with resources like Microsoft.

It’s my opinion that the best thing to happen to any big company is a little competition. It keeps them from getting away with laziness and keeps prices and expectations in the market fair.

But then again . . . my opinion on the matter may be a little biased.

When I started my first company in 2002, I had a lot of previous employers to provide examples, both positive and negative, of how to run a business. At the time IBM and Hewlett-Packard were leaders in network management, so I could have modeled my business on them.

Instead I modeled it on Ben and Jerry’s ice cream.

Many might think it was a strange choice, but it seems to have worked out well, at least for us.

First, they make a good product. This is of paramount importance in any business.

Second, they limited the amount of money the highest paid people could earn in salary. In their case, the highest paid person could not make more than seven times the lowest paid person.

I am constantly disgusted by executive salaries these days. Being a previous employee of NORTEL, now in bankruptcy, I find it highly ironic that the executives responsible for driving the company into the ground received huge retention bonus to keep them from leaving. In a just world they would have had no where to go, and particularly they would not be financially rewarded for poor performance.

To me a salary should exist to cover the basic necessities of living, but the real compensation should be based on the performance of the company. Let me stress that I want there to be no limits on overall compensation – if the company is doing well I want everyone’s “upside” to be unlimited. But getting a huge salary just for showing up feels wrong, especially if the company is doing poorly.

Steve Jobs, one of the most successful CEOs ever, takes home a salary of just $1.

Back to Ben and Jerry’s. The one other thing they did that I admired was to donate a certain percentage of pre-tax profits to charity.

I like donating to charity, but I find that I am most eager to give to those organizations that are a) small and b) concerned directly with something I care about. Thus each year I give to the EFF, the FSF and the SFLC, plus a number of local charities.

When the earthquake in Haiti happened, we were shocked and saddened like most of the world. I wanted to help, but I wasn’t sure how. Luckily, the opportunity came in a most unexpected way.

Matt and Jeff (along with Alex) were hanging out in the OpenNMS IRC channel (#opennms on freenode.net) when a man named Andris Bjornson joined and started asking questions about OpenNMS. It turns out that he works for an organization called Inveneo that supplies bandwidth in rural and under-served areas in the developing world. Haiti was the perfect example of a place that needed their services, since a lot of the relief effort is run by non-government organizations (NGOs), and they rely on communications in order to maximize the good they can do.

Haiti’s communications infrastructure, such as it was, was destroyed by the earthquake, and Inveneo is using wireless technology to provide a timely replacement. Of course they need some way to manage this infrastructure (as you can imagine, it is in high demand) and they chose OpenNMS.

Andris installing an antenna in Port au Prince (click for more pictures)

Andris has been using OpenNMS for awhile, but he had some questions and there were some issues in managing the radios they were using. The guys in the channel were more than happy to help out, but we wanted to be involved in a more formal way.

We decided to donate a commercial support contract to Inveneo to help them out in Haiti.

It’s pretty cool to be involved, at least in some small way, with getting Haiti back on its feet. It was also cool to have OpenNMS chosen from all possible apps out there to play a role.

You can read more about Inveneo and OpenNMS in this press release, and please consider donating to their efforts.

Open source has a large social component, and I have a theory that being involved in open source software makes one generally more interested in social issues. I want to hear from others about their experiences with social causes tied to open source. Jon “Maddog” Hall is also a fan of Inveneo, and I’d love to have more examples.

UPDATE: Here’s a network diagram of the Inveneo network, and the “How to Deploy” document mentions us by name.

Cisco Press will be giving away 50 copies of its new CCNP Cert Kits and other study guides to help you prepare for the revised CCNP certifications. The mega giveaway is being sponsored by Cisco Press on Network World’s Cisco Subnet – a community website.


The chances are really high that if you enter, you will win something. All you need to do is find some words that form a specific sentence in various chapters that are provided and enter the response. Not too much work for some free study materials. The contest ends March 31. Register to win one of 10 copies of the following titles:

CCNP Route Cert Kit (Read excerpt.)

CCNP Switch Cert Kit (Read excerpt.)

CCNP Tshoot Cert Kit (Read excerpt.)

CCNP Routing and Switching Official Certification Libraries

CCNP routing and Switching Quick Reference printed bundle


Good luck and hope you win!

Oracle announced this morning that it is acquiring AmberPoint, the leading vendor of SOA management solution. AmberPoint is widely recognized as the Cadillac in the SOA management space, especially with its ability to enforce policies that help improve application performance and security, and to diagnose transactions not only within a composite application, but also across different applications. There had been speculations for a long time whether AmberPoint wanted to stay independent, or be acquired by a larger vendor. The answer is now known, and it is good that we got it. :-) AmberPoint, along with Sun Ops Center, will add to Oracle's capabilities in delivering application-to-disk management to customers.

Click here for the official press release about this acquisition.

Puppet Training is popular apparently! Due to demand, we’ve scheduled 3 public Puppet training courses in NY, London, and Germany. You can register and get more information about the training at this link. If you have any questions please contact Scott Campbell.

Location & Dates

• London, UK – March 29-April 2
• New York, NY – April 12-16
• Nuremberg, Germany (hosted by Netways) – April 20-22

Becoming a Puppet Master – 3 Days

Puppet Training consists of 3 days of hands-on training performed by a Reductive Labs Puppet professional. Attendees will be taught the principles and best practices of Puppet in a series of lectures and labs.This training is ideal for those who want a Puppet jumpstart. Newer members at an organization already using Puppet, or experienced sysadmins wanting to bring Puppet into their team will get everything they need to deploy solutions.

Topics covered include:

• Configuring Puppet and Puppetmaster
• Resource Types and the Resource Abstraction Layer
• Virtual Resources, Exported Resources and Stored Configs
• Meta-parameters, Dependencies and Events
• Classes, Modules and Definitions
• Tags and Environments
• Puppet Language Patterns and Best Practices

Puppet Developer Curriculum – 2 Days (NY & London Only)

This is an advanced course for those Puppet users who are interested in developing skills and learning best practices for creating their own custom Resource Types and Modules.

• Introduction to Ruby for Puppet
• Advanced Function and Fact development
• Resource Type and Provider development
• Testing practices and RSpec for Puppet

Looking forward to seeing you there!

In this blog, we talk a lot about the challenges faced by organizations trying to cope with multiplying services, diverse technologies and siloed monitoring and management tools.  You will also hear us mention that these challenges are far more daunting when services like VOIP and video need to be delivered globally and without interruption.

In our latest case study, Oracle’s Senior Director of Enterprise Automation & Tooling, Tony Miranda, talks about Oracle’s decision to consolidate monitoring and management of their global networks with Monolith.

In a global implementation that took mere weeks to complete, Monolith has allowed this global business software giant to simplify fault, availability and performance monitoring across the company, while cutting licensing, headcount, hardware, and annual maintenance costs.  Oracle’s team now uses Monolith’s dashboard engine to quickly create custom real-time IT dashboards, improving executive visibility across their entire network.

Oracle turned to Monolith to monitor and manage global services.  Shouldn’t you? Read the full case study.

Technorati Tags:
Monolith Software, IT Management, fault, availability, performance monitoring, dashboards, Oracle

I was interested to see a blog post discussing the benefits of the new 4G wireless standards currently in development. It struck me just how long it really takes for a technology to be in use by the majority of people. Here we are at the dawn of the 4G world and yet 3G isn’t widely deployed. The 3G licences were auctioned in the UK around ten years ago.

I’ve had an Apple iPhone 3G for a few months now and I am able to use a 3G signal for a small fraction of the time. In fact, outside of major cities, you’ve very little chance of getting a decent 3G signal. Most of the time I’m stuck on GPRS speeds or worse. If 3G hasn’t spread outside of the main metropolitan areas ten years after the original spectrum auctions, then it seems likely that there is no business case for ever doing so. If it isn’t commercially viable to implement 3G then what hope is there for 4G?

I wonder if the auction process itself could be to blame for the patchy deployment? Whilst the government in the UK did very nicely out of the auction, the bidders did pay very handsomely for their spectrum. Perhaps a better solution would have been to cap the auction price but place a service guarantee onto the bidders to ensure a more even deployment.

A broader implementation of 3G technology I’m sure would be a boon to the hi tech sector in the UK and would have had the effect of increasing economic activity. Whether the increased economic activity would have made up for the shortfall in the spectrum auction revenue is hard to say. But, the auctions were one off events and the increased economic activity would keep paying year after year.

Will the areas that don’t already have 3G never benefit from high speed wireless internet access? It isn’t looking promising…

This small, unofficial project integrates an open-source AIM (AOL Instant Messaging) Chatbot with Splunk 4, allowing ad hoc searching, running of prepared searches, and real-time search alerting via instant messaging.

What’s real-time searching? It’s new in Splunk 4.1, out shortly, and will allow users to search for “real-time” events, within seconds of them reaching Splunk. Most usefully, you can set up real-time searches and be IM’d with the matching events the second they show up. You could ask to be IM’d, for example, whenever someone logs into your system, whenever there’s an error, whenever someone logs in as root, etc.

Above is a screen capture of real-time alerts printing out for each time someone downloads Splunk!

Note: You can use this project with Splunk 4.0, and everything other than real-time searches will work. That means you can do ad hoc searches and run saved searches over historical data.

Download Project

Example Searches

?	prints out a help message explaining commands.
rtsearch login root	set up a real-time alert to IM you whenever a user logs in as root.
rtlist	get a list of all your real-time alert jobs.
rtstop *	cancel all your real-time alerts.
search login | top 5 username	run an historical search reporting to top 5 users who logged in the most.
admin error	IM’s not starting with known commands will search existing saved searches (here we search for saved searches about admin errors).

Here is a compilation of articles on the subject learning Lua.

Part 1
Part 2
Part 3
Part 4
Part 5

Part 5 is the final part and a complete Lua example script, monitoring Apache worker threads.

Think back to your first day of work at a new job (could be your current one or a past one.) Remember how exciting things were, you were in HR orientation learning about your benefits and vacation policy – learning about your 401k options… all that good stuff that you get filled with when you join a company. Then the HR rep walks you to your new cubicle (because who really gets an office anymore) and you’re ready to get to work. Then, the emptiness sets in. Over the next 45 min you try to log on to your machine, you try to get email set -up, you try to get access to (enter system/application du jour here.) Then you get on the phone with the help desk after you get the number from your new cube mate (who’s already annoyed that that you’re there sharing their space) because you have no access to systems to look the help desk phone number up. An hour and a half later, you are finally up and running – at least for today. Sound familiar? 

Welcome to the world of user provisioning. What seems like such a simple task -- giving people access to the systems, applications, and general business data they need -- is really more of a three headed monster than most of us earthlings will ever realize. Dave Kearns wrote a great article on user provisioning earlier this week that I found particularly insightful. In the piece, Kearns reveals three key events in an employee’s life at a company where provisioning comes into play; when they join the company, when they change jobs/ responsibilities, and when they leave the company, and why they are important.

While giving employees access to the right resources in a timely manner is critical to business productivity, just as critical is removing access to resources when they are no longer required to perform a task or after that employee has left the company. After all, do you really want someone who has moved from a marketing management position into a sales position to have access to the payroll data for the entire marketing organization? I think not. So, what’s the answer? You got it!  User provisioning. How does your organization remove access to resources, while providing access to new resources in a timely manner for your employees? 

While productivity and burden on the help desk are great cases for getting user provisioning right, the first time, Kearns proposes that security is an even more important reason to get your provisioning right. And I must say, I agree here. Unnecessary access to information after a user has changed roles or left the company poses one of the greatest threats to the security of your business. All it takes is one person with malicious intent who abuses their access to critical data for a data breach to occur. Yes folks, it only takes one – one person – to run the train off the tracks and the next thing you know, your company is front page news due to a data breach.

So how do you prevent these things from happening? It all starts with your approach to user provisioning. If you take the steps towards automating user provisioning, you have the ability to reduce deviation from process (and the mistakes that can easily result) and drastically reduce the time it takes to provide or revoke access to critical data. Automation really is the difference between user provisioning and securely provisioning users. The latter enables you to really protect your business, while still meeting its needs.

Highlights from Planet Network Management + Planet Sys Admin for Week 5.

• Running Wireshark as You – how to run Wireshark whilst not logging in as a super user
• Free Tool and Free Tips for Network Configuration Management – Josh Stephens introduces Network Config Generator, a new tool from Solarwinds, network engineers can create a change template to re-use for future configurations, saving time and frustration as well as ensuring consistency in configurations across devices
• Improving Network Discover by using SNMP OID Includes/Excludes – an investigation into finding the device needle amongst your network haystack all using SNMP
• Using a Weather Map as your Background for your Maps – using a network weather map as the back ground for your maps in Solarwinds Orion
• Which virtualization platforms will matter in 24-36 months time? – Groundwork Open Source ponders the future of virtualisation platforms
• Opsview 3.5.2 released!

(Note: this article applies to INM version 4.0.5233 and later)

INM makes it easy to generate SLA reports on any level, wether it be a whole network
or a single object. A default 'Availability report' is included with the INM installation, so let's look at how we can tweak this report
to fit your needs.

The Availability report is a template, and as with any report template in INM it's possible to
apply it in any context. First, let's run this report on a single network.

1) select the network from the relevant view
2) select the 'View report' command and then the Availability report to generate it.



As we can see, INM reports individual Up and Downtime on each device in the network,
as well as reporting the averages for the entire network. By default, INM calculates
the average Up and Downtime for each individual entry but this can be changed to the sum if required.

Now let's modify the report template, to let INM break down the statistics even further and
report statistics on each individual monitor.

1) click on the Availability report in the Report templates view
2) click on the edit icon for the Downtime report item
3) select the "Report downtime for monitors" option
4) we're going to uncheck the 'Include monitors with no downtime in the report' option to only list the monitors that were
down during the period.

Let's re-run the report on the Fileserver object only this time.



In this case, it was a Disk time monitor that contributed to the downtime in the Fileserver object.

On some occasions you want to generate a SLA report that only takes data within a specific time interval into account.
INM can do this as well.

1) Open the properties for the Downtime report item again, and specify a time period in the
Time limit textboxes. Now the report will only include statistics that lies within this time interval for each day.

There is one more interesting option, and that is to only base the SLA report on specific monitor types.

1) Again, open the properties for the DOwntime report item.
2) In the advanced section, select monitor types from the 'Monitor limit' box and add those to be included.
Only statistical data based on the selected monitor types will be included in the final report.

Finishing up, you may want to setup a scheduled event to run the SLA report periodically.
Full details on how to setup a scheduled event can be found in the online manual, located here:
http://www.intellipool.se/4_0/doc/generate_a_report.htm

BERLIN/ZURICH (Reuters) - A Swiss lawmaker likened German attempts to buy data on cross-border tax evaders to bank robbery on Tuesday and the Swiss banking lobby said Berlin was acting as a receiver of stolen goods. Reference: Swiss lawmaker accuses... Marcus J. Ranum

Members of our support team created two nice video tutorials to make it even easier for you to start monitoring your network with PRTG Network Monitor. Lean back a few minutes and learn how to install our software and let PRTG do all the work by using the built-in auto-discovery function which searches for and adds your network devices automatically. Another tutorial shows how you can set up different notifications in PRTG to keep you informed whenever there's something special going on in your network – be notified via email, ICQ, SMS text message and more. Check out the videos at our support website.

In Oracle Unified Methods, the elaboration phase follows the inception phase of the project. This is the time when detailed analysis is done and key design decisions get flushed out. Traditionally, the focus of this phase is on coming up with detailed application functional design, especially the user interface, the data model, the means of integrating with other applications and data sources, or even the technical architecture of the deployment, etc... However, the same vigor is often not applied to the tools that are needed to manage the applications. This is very different from other complex engineering endeavors such as automotive and aerospace design, in which far more thoughts are put into the dashboards and the avionics. Using the right set of tools and implementing the tools properly are important to successful application projects.

Several deliberate decisions need to be made in tool selection for managing applications. The first one is whether to build home grown tools or buy packaged products. Some people prefer to build their own tools, but it is a difficult effort to sustain in the long run. Developing tools is like developing any software. To do it properly, they need to be properly designed, implemented, tested and maintained over time, which get expensive. Instead of building tools from scratch, most organizations opt to reuse something that they already have, which in many cases are generic management tools that were originally designed to manage servers or networks. The problem here is that a fair amount of effort is still needed to adapt these tools to manage applications, and they always provide only generic functionalities that do not address the real needs of managing applications. A better thing to do is to use tools that are designed specifically for the job of managing specific applications, while maintaining a balance of avoiding tool proliferation. I wrote about the topic of tool selection in greater depth in article last year. Click here if you want the details.

Besides getting the application management tools, it it important to design the tools to be an integral part of the runtime environment and allocate capacity to run them. Many people treat tools as an overhead. If you go by the definition that tools do not perform any actual processing of business transactions, then it is indeed an overhead. However, tools form a critical part of an application infrastructure. Without tools and the instrumentation to collect management data, the application becomes a black box that cannot be managed. No one would design an aircraft without proper avionics, and the same thing should apply to tools also.

Another set of decisions are related to the deployment architecture of the tools. The architecture needs to be designed deliberately with a similar level of care taken to design the deployment architecture of the applications, especially if the tool will be used to manage a complex application environment. In fact, there are similarities in designing tools deployment architecture and application deployment architecture. For example, one has to decide between centralized single instance tool deployment vs. multi-instance deployment of tools such as Oracle Enterprise Manager, just like one has to decide how many production application instances to deploy. This sort of decision is highly environment specific.

Oracle Enterprise Manager Grid Control, by building on Oracle Fusion Middleware and Oracle Database, allows the tool to scale both horizontally and vertically. Therefore, from a technical perspective, a single instance of Oracle Enterprise Manager can scale to manage thousands of applications, database, and servers targets spanning development, testing and production environments. However, some organizations may still want to deploy multiple instances so that different units within the organizations can maintain control over their own instances of Enterprise Manager in order to maximize control and flexibility. Others may want total separation between Enterprise Manager instances used to manage pre-production vs. production environments in order to maximize security. The final decision needs to be made based on not only technical factors, but also organization and other considerations.
Whether you are going to have a single or multi-instance Oracle Enterprise Manager Grid Control deployment, you still need to make sure that you set up at least one separate test instance of the tool. Before you roll a version of Oracle Enterprise Manager into production use, for example, you should have it tested in order to minimize any surprise.

Another potential decision is to decide whether high availability (HA) deployment is needed for the tools. Just like a pilot cannot fly with non-working avionics, it is virtually impossible for administrators to manage their applications effectively if the tools are not available. Some management tools support high availability deployment. For example, Oracle Enterprise Manager Grid Control servers, known as Oracle Management Service (OMS), can be set up to run in a clustered configuration or even a multi-site clustered configuration. The underlying Oracle Database repository can be made highly available by leveraging Oracle Real Application Cluster (RAC) and data guard technologies. More information about HA Oracle Enterprise Manager Grid Control implementation is spelled out in Oracle Maximum Availability Architecture guidelines.



Related Articles

- Want to Have Smooth Running Applications? Start with Good Planning.
- Building Application Management into Your Capacity Plan
- People, Process, Technology – The Right Tool

Mark Burgess was interviewed (with a sore throat) on FLOSS weekly.

A new feature in the Cfengine Community Core is attracting some interest from system administrators. It is the simplest of ideas, but then such ideas are often the best.

Moving into a new year, the privately owned and funded Cfengine company has changed its board to include some power members of the Free and Open Software community. "The time has come to change the style of our board work, as we move into a different phase of growth," says CEO Thomas Ryd.

The OpenNMS Group has finally moved into double digit employee numbers with the hiring of Brad Miesner as our Vice President of Sales.

I know what you’re thinking – a sales guy? Earlier you post that you hired some folks to do marketing, and now you hire a sales guy?

First, let me point out that I’ve known Brad for over ten years and he started off in a technical role. So he’s not just some guy with no network management knowledge who’s going to pester people to spend money.

Second, interest in OpenNMS has grown to the point that it can be difficult for us to handle, in a timely manner, requests for information about our services. I always focus on our existing customers first, sometimes to the detriment of potential clients, but Brad will insure that our future clients receive the attention they deserve.

But most importantly Brad will have the role of “customer satisfaction manager”. We tend to build close relationships with our clients, and if we should happen to drop the ball, these clients might be a little hesitant to complain directly to the people at OpenNMS with whom they are working. Brad will proactively be in touch with all of our partners to insure that we’re providing the best service we can, and if there are ways we can improve, it is hoped he will hear about them.

Brad comes to us from Network Appliance, voted by Fortune Magazine as the number one “Best Place to Work” in 2009. He was doing really well there, and I think his decision to join our band of open source revolutionaries speaks well for both the company and our future sales prospects.

We are extremely happy to have Brad join our team.

Oh, at one time he worked for a little software company called Zenoss, but I think he’ll quickly adjust to working in open source.

(grin)

Last month we kicked off a contest to design the new tagline for our next t-shirt and now is the time for you to tell us who should win. We received a bunch of great entries and have narrowed it down to the following 8 finalists. Please vote for your favorite and decide the winner.

The person who submitted the winning entry will receive a BugBundle from Buglabs. It’s in your hands to determine who wins.

Running Wireshark on Linux involves an interesting challenge¹: Capturing packets requires root access, but Wireshark is big program and we strongly recommend against running it with elevated privileges. On Linux it’s common to see Wireshark running as root, but this is nearly unheard for similarly-sized applications like Firefox and GIMP. How can we avoid running Wireshark as root?

A good way

Notice how I said “capturing packets requires root” above? Here’s a secret — Wireshark doesn’t capture packets. A separate program called dumpcap does. Compared to Wireshark, dumpcap is tiny. It’s much less complex and much safer to run as root. We can make it so that dumpcap runs as root and that only users in a particular group can run it:

$ sudo -s
# groupadd -g packetcapture
# usermod -a -G packetcapture gerald
# chgrp packetcapture /usr/bin/dumpcap
# chmod 4750 /usr/bin/dumpcap

A better way

It’s also possible to let dumpcap do its job without involving root access at all. For a very long time Linux has allowed the use of fine-grained permissions called capabilities. In many recent distributions you can use the setcap utility to add capabilities to individual files.

Dumpcap needs CAP_NET_RAW and CAP_NET_ADMIN, so what do we need to feed setcap? On my Ubuntu Karmic system the setcap man page points you to cap_from_text. Cap_from_text points you to _cap_names, an array in the kernel. It would be nice if the setcap man page included a list of capability names along with a few examples. As it turns out, the names need to be in lower-case.

$ sudo -s
# sudo apt-get install libcap2-bin
# groupadd -g packetcapture
# usermod -a -G packetcapture gerald
# chmod 4750 /usr/bin/dumpcap
# setcap cap_net_raw,cap_net_admin=eip /usr/bin/dumpcap

Fully-functional filesystem capabilities is something the Linux world has needed for a very long time. I’m glad they’re finally seeing wide deployment.

1. This is a problem on other systems too, but it’s usually easier to solve. On Windows you can run the NPF service at startup. On OS X you can use ChmodBPF.

An issue that frequently comes up for IT managers is the need to find only certain types of devices within a heterogeneous network that contains many types and manufacturers of networked devices. I recently worked with a customer that wanted to locate about a hundred Windows Servers from a network that contained several thousand devices.

One way to approach this task is to discover all the devices, then pick out the ones you are looking for, the old needle in the haystack routine. This approach is time consuming and error prone. A better method is to leverage the information available from devices that support the SNMP protocol, which includes most operating systems. SNMP includes an object library of OIDs (Object Identifiers) that are set up by each manufacturer. A Google search for “Windows OIDs” found this site which listed the OIDs to identify Microsoft Server Operating Systems.

As you can see (table below), the OIDs are built in a hierarchy so, if I could search my network for servers which contain the OIDs below for workstations, servers and domain controllers, I should find all my Windows Server boxes.








You can make the difficult task of finding and sorting networked devices much more manageable. I use dopplerVUE, a network management tool that simplifies the whole process and helps find the needle in the haystack faster and without issues.

dopplerVUE provides an OID include/exclude discovery feature that makes it easy to accomplish this task.

Here are some steps for using dopplerVUE to improve the network discovery process. To get started the server must have the SNMP agent service running and you need the credentials (called a community string) to enter in the SNMP service “security” tab. Most servers use “public” as a default and are case sensitive. SNMP service is usually turned off by default, so you’ll need to restart the service when you are done making changes.


Once you have the servers set up, you should create a discovery job within dopplerVUE to find the Windows Servers. dopplerVUE provides a discovery wizard that guides you through the step by step process as follows:


Step 1: Select a discovery method appropriate to the task. Use an IP address range that provides the most control over your discovery results.


Step 2: Set an IP range that includes the Windows Servers you are looking for in your search. Be careful, the larger the range you select, the longer it will take to complete the discovery.


Step 3: Select SNMP protocol.


Step 4: Enter the community strings for the servers. Your admin can provide these and you can always use public which is set as default on most servers.


Pictured below is a tab marked “Show sysObjectID include/exclude options”. You can click on the tab, expand the Window and then select “include”. You can then enter the OIDs we found earlier.
















Step 5: In the workstation column you’ll want to select SNMP poller and then Host MIB if you want to collect information about processor utilization, memory usage and disk space.



Step 6: Optional: Enter a name and description for this discovery job.


Now you can click finish and go to the Inventory>Discovery Jobs tab to watch the progress of the task. The job will start automatically assuming your dopplerVUE discovery service is running and you had the “run now” checkbox selected in step 6. If not, click on the job and start it.


You can watch the progress in the job details section and keep an eye on your inventory tab to see if new devices are being found. When new devices are found, they should appear in the workstation classification. You can change classifications or create new ones easily by right clicking on the objects in the workstation classification list.


This technique works for any search where you can separate the devices by manufacturer. Since each manufacturer determines how they want to build their SNMP library, you’ll need to understand how they created their hierarchy. Fortunately there is a lot of good information available on manufacturer websites to help you. Here is more information about SNMP support within Windows.


If you’re looking to improve network discovery and automate IT tasks to save time, try dopplerVUE for free for 30 days.

Linuxquestions.org is running a poll on the best open source projects and there is a network monitoring application category.

If you like OpenNMS, we’d appreciate your vote

Unless you have had your head in the sand, SQL Injections have made a fierce comeback to the top of the threat vector charts this year. According to the WHID (Web Hacking Incidents Database), SQL injection is still king of the attack vectors, accounting for 19 percent of attacks, followed by authentication abuse (11 percent), content spoofing (10 percent), DDoS/brute force (10 percent), configuration/admin error (8 percent), cross-site scripting (8 percent), cross-site request forgery (5 percent), DNS highjacking (5 percent), and worms (3 percent).

Reflect on the recent increase in compliance legislation requiring businesses to provide dynamic data access to customers for banking, healthcare, or the influx simple purchases on the web, and the concern may be scarier for all of us. Recently, Dark Reading reported on the number of companies who have been compromised through SQL Injection attacks.

What is SQL Injection, and How Does it Work?

If you don’t know what this is, and just learned what SQL is, I recommend going to OWASP.org and reading up a little. It is a great resource, and the mass amount of security professionals dedicated to the Open Web Application Security Project deserve a big shout out.

Lets start with how SQL Injection actually works. SQL Injection occurs when an attacker is able to insert a series of SQL statements into a ‘query’ by manipulating a data input, usually a form for users to update their account information. Some common relational database management systems that use SQL are: Oracle, MSSQL Server, DB2, Sybase, Informix, MS Access, Ingres, and so on, with the most popular being MSSQL of those.

Whether you are a potential attacker, auditor, researcher or an application developer, you may go through the same steps to exploit or find exploitable code:

1. Input Validation
2. Information Gathering
3. 1=1 attacks
4. Data extraction
5. OS interaction
6. OS Cmd Prompt
7. Expand influence

More information available at OWASP (Victor Chapela, OWASP, “Advanced Topics on SQL Injection Protection”)

Splunk and SQL Injections

Splunk approaches this attack a little differently because of our ability to make all IT data security-relevant. Within the Splunk index, organizations will collect logs, custom application logs, traps, configurations, stack traces, scripted outputs, auth data and metrics for analysis. Splunk can help in applying security/audit logic in various detective controls to aggregate IT data to one place, make simple sense of the data, apply relationship logic into what might appear to be a standard operational issue. This logic can give you a “report”, alert, dashboard, e-mail, run a script to gather more information, or simply create a news feed of the ongoing event to send to a ticketing system, incident software or other system. Not just designed for tier 1 troubleshooting, Splunk can help incident handlers and analysts backtrack events by digging into logs across geographies, datacenters, applications and technologies. Incident handlers, auditors, security professionals can then persist the same logic in a “search” to identify the next occurrence, proactively defending sensitive assets.

How Does Splunk Do This?

Well, for this case, lets use the Splunk Security application, Enterprise Security Suite (ESS). ESS enables simple searching to illuminate information in the muddy, challenging environment of security and operational data accessed by more people than just security androids and SysAdmins. I use it to help organize security data into categorical security areas: Access Protection, Endpoint Protection, Network Protection, Incident Response, and Governance.

You Already Have the Answers: In Your IT Data

The same Splunk rules still apply, you have to put your data in, to get good information out, so we need a few key pieces of data to find a SQL Injection – some more damning than others – to identify what starts out as an operational issue, but turns into a security investigation.

IDS Logs and Events

Though there are many methods of subterfuge to avoid IDS/IPS detection of the 1=1 statement, getting a look at the application data in a purported attack via a Snort/Cisco/Juniper alert, is very helpful as part of a correlated event. SQL injection may include of logic, depending on the input validation, a ‘;’ may help; seeing JOIN or UNION statements may also be indicators of misuse.

Packet Capture

Always good, certainly looking at application data in the packet with SQL statements is going to be helpful. Thing is, often times, database replication, linked databases, etc. are all capable of using HTTP as the transport protocol, so be advised- this could be a lot of data, and it may be legitimate. Alerting on these events in Splunk would let you execute a script to trigger TCPdump or something based on an event, if the Splunk instance is enabled with tcpdump.

Vulnerability Assessment Tools

Nessus events, or other audit tools, can help qualify the actual threat of the injection language based on the type of systems you are protecting. MSSQL statements are more forgiving than say, Informix for example, and if you are a UNIX shop, MSSQL attacks do not pose a risk, though this may mean some interrogative work.

Anti-virus

This is handy should malicious code be dropped, downloaded and/or propagated via SQL language over HTTP, FTP, SSH or other file transfer protocols. When an event turns off AV, or a failure occurs after a noticed injection, there should be concern as to the sanctity of the system it failed on.

Host Data

Perhaps the application server and database servers have file integrity monitoring, maybe a scripted output of binaries such as top, psstat, or in windows, netstat and ipconfig? Looking at a new listening service you didn’t install, may be after the fact, but at least identifiable with Splunk. If you happen to have something like OSSEC installed, or another kernel monitoring software, perfect. An example SQL Injection provided by OSSEC, looks like this:

200.96.104.241 - - [12/Sep/2009:09:44:28 -0300] "GET /modules.php?name=Downloads&d_op=modifydownloadrequest&%20lid=-1%20UNION%20SELECT%200,username,user_id,user_password,name,%20user_email,user_level,0,0%20FROM%20nuke_users HTTP/1.1" 200 9918 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)"

note the UNION statement followed by %20SELECT%200. This is the beginning of pulling all the successful users from PHPnuke site, and getting table data on all of their usernames, IDs, passwords, and e-mails. If you aren’t careful about your web application accounts all being the same, you could be the subject of more than a twitter DDoS.

Application Logs

Web server logs, depending on verbosity, this may be a good starting point to determine the source/destination centered around IP traffic, as well as success/failure and some explanation of why. Nothing special about these logs, we use them for operations and security. Several types of HTTP status errors can be helpful in determining what is going on – for instance, a 403 or a 401 error on a customer facing application.

User Audit Log

Finding failed and successful authentication attempts on a local system is extremely helpful- especially around the time something changed in our environment. Make sure to note, actual injection code may be executed on the application tier or webhost, rather than the database server itself. Detective controls should apply to the entire transaction architecture.

MSSQL Error Logs

Error logs provide some good information for Splunking account access, and database errors especially after isolating the application error occurring in the transaction model. To add the error log on a forwarder as an input:

[source::...MSSQL\\LOG\\ERRORLOG]
CHARSET = UTF16-LE
NO_BINARY_CHECK = true

(Make sure to do this on the indexer as well)

The most valuable source of information is actual data across the wire. You are using tcpdump, or wireshark, monitoring the SQL server (if you don’t, you should at least have the capability should there be a threat). If extended procedures like “xp_cmdshell” are being executed they may actually be logged when invoked. Xp_cmdshell enables a virtual cmd shell within an SQL statement. Maybe you have Windows event logging, and registry settings baselined, and a profile of your SQL server’s persisted connections base lined so you can apply a diff to them frequently, uncovering new network connections, and services alike. Similarly, user logon/logoff events to the operating system and application, both successful and denied, are handy and can indicate if a system account seems to persist failure a-la brute force attack. We may also see a large amount of auth attempts to SQL server, or the database through the windows Application log as well.

Determining Contextual Relevance

The same information has different context depending on who is looking at it, what it is used for, and how it relates to other data, what we call contextual relevance. In Security, almost every technology is interrelated in either debunking a threat, or validating it. We use Splunk to help solve Operations availability problems, Information Security problems, Compliance requirements, both on the fly and proactively, bringing these problems to light with alerting and notification.

Any of these events may show up, though logging may be turned off/system clock reset given a system may have been compromised with a Splunk forwarder. The stopping of event data in logs is also telltale, so be sure to no alerts=immediate concern in Boolean search, and persist a search that alerts when no events appear.

Input validation is really the first way to stop the problem in development, but stored procedures can be modified to add a regular expression for a credit card number, an SSN, or other fields. Preventing the additive SQL command in the input space of a form, stops the problem ahead of time, so does fuzzing before code is live, but secure coding is another chapter.

Some of the information available in Splunk will also allow the operations personnel or administrators to go back and audit configuration files, to see if DBerrors are being thrown to the user’s screen enabling enumeration/info gathering. Even if the code cannot be pulled from production, enumerating poor SQL input validation can be more difficult if error reporting is turned off.

For instance, is your config file configured to throw exceptions on that server? Have a look in Splunk. Take HTTP status errors for example a 403 Forbidden error status code, this may be something that gives valuable information to a potential attacker. 403 errors may be something you don’t want to show folks, unless in a development environment, so this may suggest turning off error responses if you see them in Splunk. At the same time, this generates an event beneficial to the application owner, but maybe doesn’t need to be broadcast. Users should see 404 errors when error notification is disabled, rather than a 403 error. When in doubt, look in Splunk! Maybe a quick search for all 403 errors may let you know there is an example of a potential SQL Injection occurring.

Once you find a 403 occurrence, or a bunch of failed account logins to a database, or an application server unavailable (503), use Splunk to span a 15-minute window by dragging the search range to the 15 minutes around the error. Then, look for all events during that time, on the host that may have hosted the error event. Perhaps a system has been compromised, and credentialed access may allow downloading/uploading?

Look for failed and successful authentications to the same system. Look for escalated account privileges. Splunk allows you to run a “System Profiler” as well to look at things like the change in listening services and ports on the Network, and things like infections or the state of anti-virus from the Endpoint Protection dashboard. When you find the behavior in all of the layers of events that correlate the event, create a search to populate a summary index as a “notable event” appearing in the Security Posture dashboard, and create an alert for it. Next time, Splunk users and Incident Handlers alike will know without re-creating the on the fly search we just did.

Splunk Helps Make the Seemingly Unrelated – Related

What does all this mean? Simply this- simple users can apply complex concepts from experts in the field, to search, detect, alert and report on security threats like SQL Injections, using Splunk. ESS allows a wide variety of users to look at the same information and derive different conclusions based on contextual relevance. Most real security risks, don’t flash “SQL INJECTION ATTACK” in your dashboard, you need to understand your environment and what pieces work together. Given the increased frequency of Web application threats, specifically SQL Injection, identifying the threat itself depends on multiple layers of security as well as a way to simply search through all of those layers simultaneously. Found an IP of a suspected system? Splunk it. Found an error for authentication to a back-end system? Splunk it. Can’t find the relationship between events in a given period of time? Use Splunk to make the seemingly unrelated, correlated.

For more information on how to monitor your MSSQL-driven application using Splunk, drop us a line: info@splunk.com

Microsoft has provided their own suggestions as well. http://msdn.microsoft.com/en-us/library/ms998271.aspx

Moscow, Russia (CNN) -- Russia tested its fifth-generation Sukhoi fighter jet in the Russian Far East on Friday. The plane, provisionally called T-50, is the country's first fighter jet based on the stealth technology and is viewed by military experts... Marcus J. Ranum

We’re proud to announce that ntop.org has been accredited as Endace technology partner as recognition for ntop contribution in the open-source world and also as guarantee for Endace customers that products such as ntop and nProbe run smoothly (and faster) on Endace DAG cards.

Here’s a document that can get you analyzing real data and making real charts, in about an hour or two…

Dive into Splunk

Feedback really, really appreciated.

Colasoft Announced the Release of Capsa Network Analyzer 7.1 FOR IMMEDIATE RELEASE: 2/2/2010 Contact Information: Jane Hu Email: jane.hu@colasoft.com Tel: +86 28-8512-0922 Website: http://www.colasoft.com Chengdu, China – Feb 4, 2010 – Colasoft, an innovative provider of all-in-one and easy-to-use network analyzer software, today announced the newest version of its flagship product- Capsa Network Analyzer. Version 7.1 is based on the second-generation Colasoft [...] 0

We launched a new OpenNMS Group website this week and I am having a small problem. On the home page we have a jQuery script called Crossslide that rotates some pictures in the banner:

It works fine on Firefox, Chrome, Safari and IE8. It doesn’t work on IE7 and I have absolutely no idea why. I’ve reformatted the code, used both relative and explicit paths, and … nothing. No errors either.

I have no experience debugging Javascript issues within IE, so if you can help I would appreciate it.

One area of process improvement, for knowledge workers, used here to mean people using networked computers and other digital systems, performing financial services activities, customer support, IT services and the like, has been a particularly intractable problem. In the past, a common approach to improving these processes has been to send consultants out with clipboards and stopwatches, stand behind the user and record activities, timing and process data, and then, back in the office, to draw a process map, perhaps capturing and printing out a few screen shots for process documentation. There are a number of problems with this approach. First, users at a work station behave differently when they are actively being watched. While watched, these workers tend to be more productive and typically don't engage in personal activities like surfing the web or chatting with co-workers like they might otherwise do. Then, when the observation period is over, they generally lapse back to normal behavior. This phenomenon, which presents an inaccurate view of the normal process, is called the Hawthorne effect, after a landmark productivity study conducted at the Cicero, Illinois Western Electric Hawthorne plant in the 1920's. Second, and more importantly, the amount of data observers can gather is a tiny fraction of the total computer and network-based activity that goes on in a large enterprise, leading to an inaccurate process model, lacking the details necessary for process improvement with the well known approaches mentioned above.

In recent years, several vendors in the Business Process Management space have used software to collect data from applications and networks as a more robust input to process mapping. Inputs have typically been extracts of database or other enterprise application logs. This approach is more accurate and more scalable than human based observation, but is dependent on the quality of the logs, which are generally not very granular and have the disadvantage of not having user screen captures for detailed analysis of user behavior. Additionally, each application has its own log format and content, making for a difficult data collection and analysis task.

Achieving breakthrough levels of process improvement requires both very granular and scalable data collection and storage as well as world class process improvement methodologies. While collecting process data, one must also avoid materially changing the performance characteristics of the process under study and the enterprise infrastructure. Typically it is not feasible to ask an IT department to retrofit applications with performance metrics and reporting. Ideally, then, one would use passive network taps and/or switch or router based constructs like Cisco VACL Capture or mirror ports. These collection points are realtively inexpensive, non-invasive and, for a single process under study, even in the largest enterprises, can generally can be collocated near the server farm in a small number data centers. Given government regulations such as HIPAA security, and the need to protect financial transactions, the system must encrypt the collected data in place and in real time. In large enterprises, such as healthcare insurance companies, which process upwards of 100 million claims per year, there are several gigabytes per second of data to manage, requiring powerful servers and fast storage. There is also one broad case where traffic across the network, unlike web oriented applications or the still-common dumb terminal emulation, does not accurately reflect the user's behavior. That is the fat client, where an application on a PC does significant work. For fat client applications, a distributed agent with a small footprint and a low reporting traffic rate can generally capture screen shots and field changes, forwarding the information to a central server for storage and later analysis. This approach has proven economical even in VPN-based home office workers, thanks to the widespread availability of broadband data services in most homes.

As we know from Lean and Six Sigma, in order to improve the process, one must measure its key performance indicators (KPIs), including items like user think time, system time, key data inputs and granular transformation procedures. The data that is collected as described above is the key input to a business process discovery. Once we discover and analyze the process sufficiently, we generally have sufficient information to do root cause analysis of problems, hypothesize the changes needed to improve the process, do experiments to better understand the main factors and their interactions and to measure changes to the process as we make them. Since we now have powerful computers at our disposal, it is possible to automatically map the collected data into a business process along with timestamps, userids and other user interactions, as shown in Figure 2. Having large volumes (empirical) data available, rather than the usual anecdotal data and estimates regarding the process, gives us the ability to analyze it scientifically, to graph metrics like end user response time, identify paths users took through the process, analyze the status of key indicators at various times and places in the process, whether straight-through, as we we'd like, or rather, down exception paths. We can also analyze how often and why the long tails of exceptions take place, then relate user and process behavior back to business rules for improvements.

Process metrics don't just materialize and processes have no pre-existing context, unless they were originally imbued with BPMN or similar metadata. In 2010, the vast majority of processes in use were built before BPM standards gained traction, so it is necessary for business subject matter experts to label the discovered activities (applications) with names that reflect what they are called in everyday usage. The discovery engine can then generate meaningful event-based intelligence from transitions occurring in the monitored business process.

One approach to generating business events is commonly called screen scraping, though in reality it is a fairly sophisticated form of data analysis in its own right. One must be able to analyze the data moving between client and server, whether that data is from dumb terminals, web servers or whatever traverses the network. Then one must render the data in the same way that the target machines do, in order to recreate what the user saw and how he responded. See Figure 3 for an example of this kind of data analysis, recreating the user's experience for the business analyst and providing information needed for the second use of the original data, an analysis of user and system behavior. An additional requirement for breakthrough improvement is the need to analyze and report on process data without the restrictions of traditional Business Intelligence systems with pre-defined schema, event summarization and the subsequent restrictions on analysis and reporting. In the real world, these concepts and methods have been embodied in commercial, off the shelf software and put to use in healthcare insurance companies, improving processes like claims operations, as well as in banks to improve call center operations. By discovering and analyzing process intelligence, workforce intelligence and customer intelligence, enterprises save millions of dollars annually through identification and elimination of defects and waste, especially work in progress. Often, these enterprises write robots to automate some or all of the work previously done by humans, speeding up the process and freeing people up to do more sophisticated tasks and improving customer satisfaction.

About Comprehend and OpenConnect Systems

For more information about product improvement for knowledge workers, the reader is invited to visit the web site of the author's employer, OpenConnect Systems, to read about Comprehend, a product suite which implements the concepts described above and from which the screen shots are excerpted. OpenConnect Systems, based in Dallas, TX, delivers software and service based solutions focused on improving knowledge workers' business processes.