Monitoring Scripts vs. Tools

If you have been monitoring databases for awhile, you probably have a set of scripts that you have to run against the database to provide you valuable information. The scripts might tell you if a tablespace is getting full, what indexes might need to be rebuilt, if there are any errors in the alert logs and other health checks against the database. If the monitoring provides good information in a timely manor, the DBA is able to be more proactive, like adding datafiles to tablespaces before they run out, or even reacting quickly to an issue that might arise in the alert logs and contacting the application team before they have a chance to pick up the phone.
So, are monitoring scripts being replaced by tools? Tools such as HP Openview or Oracle Enterprise Manager will provide alerts and notifications about several issues as well. Just configure a couple of thresholds and away you go. But what if the configuration takes more work then the quick kornshell script? For example, monitor tablespaces and let me know when they get under 20% free, but if it is a large tablespace such as 4TB use 80GB as a threshold instead of percent. I'm sure that this can be done with tools, but still haven't figured out quite how to do it yet. Where my script has and can still provide this list very easily.
So, how do we let go of these monitoring scripts that have been around since Oracle 7? Something that we have depended on for all of these years to do our checks of the database, and use a tool to do this for us. Well, I'm sure that maintaining the scripts does take time, and learning new things is fun as well. I think that they both have a place in our environments. Setting up a tool out of the box, might even provide a quick report much faster which might have been something you wish you had.
When looking at the tools be grateful for having them, because some of these scripts were developed because the budget didn't always allow for tools in the environments. But consider what is important to monitor, consider the ease of the tool to configure and then change if needed. Let them run in parallel for a little bit to confirm the same alerts and information is being sent. Then if there are those one or two little things that the scripts have been able to do better, keep the scripts around (maybe even let a tool company know of an enhancement idea). Also, keep an eye on the tool upgrades, for new things that they monitor that you might not have thought of. Enjoy getting health checks and proactive monitoring from whatever is available to you in the environment, because isn't it really about being able to address a problem very quickly or prevent one from happening in the database anyway!

Something is wrong with the database

So, the emails start flying, something is wrong, the database has a problem. That is a very typical situation, and instead of defending the database right away, take some time to do a quick check of a couple of things.
Check number one might just be too obvious, but check the alert for errors. Validate that there is nothing goofy going on. And while you are checking out the bdump directory, a quick glance at udump for any trace files that could also be out there might show some information.
Check number two, any invalid objects or unusable indexes? Make sure that all procedures, views, triggers have a status of valid, but before recompiling, make sure you grab that last_modified date, because it might be needed later. Also, unusable indexes that might need to be rebuilt should be noted for what tables they are on and see if they are part of the issue.
Check number three, validate that statistics are up to date on indexes and tables.
And then check to make sure that there are no objects that were recently changed. Check that modified date on all of the objects. Even a modification to a data type can cause a join that was previously working to fail.
Maybe you use the checks in a different order, but with just this four, any obvious errors on the server have been found, anything that has changed has been validated and noted as changes made to the database and statistics have been checked, which can either show that this regular type of maintenance is not running or things are looking good and up to date on the datebase.
So, something wrong with the database, possibly, but now after these quick checks you can pull out more details about what they are seeing and what can be wrong. There is also supporting information if things have been changed or modified and help drilldown to more of the issue at hand.

DBA Lock Down

So, what is the sys password really needed for anyway? Not having the SYS password really going to keep a DBA out from logging into as SYSDBA or getting the job done? Well, probably not, especially if this access isn't locked down at the host level. Also, if a DBA is logged in to the host as oracle, there is probably a way to login as sysdba, either as sys or granting the access to the DBA login. Another question, DBAs do you really want to login as SYS? If it is a habit to go the host as oracle, then do a login as sysdba, isn't this just setting you up for trouble? Hopefully there is some sort of auditing in place to capture when the database is accessed as sysdba, but logging into a system with a least privilege user is always a good idea. It not only prevents accidentally doing something on the system without consciously knowing you are going to make a change and need special access, but also gives you the separation of duties from normal day monitoring to performing changes.
There are not too many times that I have needed to log in as sysdba. One example has been at creation and configuration of a new instance. Of course since it is a new instance, there is no data or users to mess up with any changes, a fairly safe way to login. Also, it was needed to restore a database and clone another. Even thinking about that it There are scripts that can be setup to stop and start as well as specific permissions granted, and then logging in as SYS seems not to be needed.
So, what is the big deal about logging in as SYS? Well, besides having all of the permissions to do anything in the database, I guess I have normally viewed it as a best practice and might even protect me from myself. But maybe I have been the only one to shutdown a wrong database. I have also found it easy to complete my job without the permissions and the few times that it is needed, there is a way to grab the password and complete the task.
Hide that password, lock it away, forget you even know about SYS, and use only the permissions needed.

Time to apply what was learned...

Even though Collaborate 09 - IOUG Forum has come to a close this year, and in going back home I am thinking of all what can be applied back in the "real world". The amount of learning and information that is packed into such a short amount of time is incrediable. Everything from OEM tricks and tips on installing and configuring to RAC and 11g new features. Support for the current Oracle 10g database has been extended, but with all of the new features of 11g upgrades should be in the planning. Orlando was really the place to be this past week if you use any of the Oracle stack, learning about the individual pieces as well as how they all work together is really a big advantage of having this conference. Getting to know members of the IOUG and learning what they one to hear about and if the sessions that they attended were useful was also great conversations in the evenings. I did really enjoy hearing about all of the different presentations and what was good and not so good. It is amazing that you can pick up a tip to improve your backup strategy, learn how a company is using streams and then the best way to secure you database, all before noon each day. I was also able to step out of my normal database realm and learn about what Oracle is doing the content management and record management area. Then there were also sessions on SAP and Peoplesoft. So, starting planning if you are sorry you missed all of the great learning, Las Vegas, April 2010.

Next CPU...

So, if you are like me and having to deal with a very large environment, you probably feel like you just finished patching with January Critical Patch Update. It is April already and the April CPU was released last week. However, since we all have our plan and process in place, it is a piece of cake, right? OK, so we might not all have a complete process in place, and some of this seems that we are just constantly patching databases, but maintaining a secured environment is important.
In reviewing the release notes, there are some important patches to apply, there are new exploits on the database side. The affected components are listed in the documentation as well, allowing for focus in these areas for testing and validation and not having to worry about the other areas. This is also beneficial if when installing Oracle only components are installed that are used, the patches can still be applied, but testing would probably be made very simple at that point if there are is only one or two components that are affected.
Having a policy from the security team in place has really helped with deployment of patching. It isn't just the DBAs saying we need to patch, but overall security policy requiring us to. This has additional support for testing and getting the needed downtime windows. Overall security patching also helps for coordination of the different level of patching from OS to application layers. Exceptions are then required from any application team not able to allow the patching, which will then push back on vendors of these applications, and I believe getting them to work on developing standards around patching and security fixes. I think that this would even help with overall security posture of these systems.
So, policies, processes and patching all good things for those of us supporting these important business applications and environments.

Backup Strategies

I really should say recovery strategies instead of backup strategies. Every time I setup a new database or learn about what an application really does, in the back of my mind I am wondering if something were to happen to this database is the current recovery strategy going to work? Sure I can use RMAN and even exports to take backups of the system. I can also verify that backups run every night and the tapes are good, but is the application going to be in a state that I can recover it and is it really going to be as simple as recover database.
In moving to even a more high available system with RAC, I wonder if that because you can failover to another node backup strategies might not be considered as important. But there are so many other things that can go wrong. What if a security patch isn't applied correctly or a hotfix for the application is rolled out and results in a table are incorrect because of it? Or even better, because you and I know that there are places for ad-hoc queries in applications, and someone runs and update or changes a table structure, what is going to be the best way to recover now?
I think that the best thought out backup strategies are ones that include these thoughts and considerations. Thinking of the end result of actually recovering a database can give insight to what needs to be backed up and how frequently. Also the understanding of what pieces might be the most important and customized. In a large environment it is very difficult to implement several different strategies, but at least considering if I have RMAN, flashback and exports implemented, which one am I going to use first to recover. Can I just flashback a query or a table and how big does that flashback area really need to be to provide what I need to be able to get it back quickly. Import might take too long to run, but can I use that information in a test database to reconstruct what is needed to not have the production system down. With the high availbility can I failover quickly, or do I have a place to run a restore from RMAN in a real disaster?
So, think recovery and think what things are in place to restore a database, and if you want to even have more discussions about this, join me at Collaborate09 - IOUG Forum, which will be a great place to discuss recovery techniques as well as learn other things near and dear to Oracle technology professionals.

Repeatable Process Worth the Effort

So, I might be stating the obivous here, but taking the time to create, develop and review a process to get a task done is always going to provide benefits, make things more efficient and produce better results. Take for instance, upgrading databases or applying patches that is something that will consistentantly be part of the life of a DBA. What if the deadline to get the upgrade done very quickly and there was a need to show results as soon as possible. So, is it showing results by developing a process, and putting together a test plan?
Isn't that some of the problems we have when faced with deadlines? We might have to upgrade a database much quicker then planned so the steps or a test plan may not be documented as needed. Then if wanting to handover the upgrade to another team member or team for patching in production, there is time wasted "guessing" what was done in the test environment because there wasn't time to at least document the steps or create the process.
Even if there are only a couple of databases this time around, there will be future upgrades and patches to be applied. A repeatable process, a plan that is documented can go a long way for current and future tasks.
With the IOUG Security Patching survey results, I have been ask recently about what it takes to get the patches out there, what are some best practices. My thought is a repeatable process. We can collect best practices on upgrades, adapt them for our environments, create test plans around the applications and other pieces of our environment, throw in a little bit of documentation and then before we know it, a repeatable process. The trick here is to setup this process the first time around while not putting the deadlines at jeopardy. Honestly it might take working more hours in a day, but not having to go through the whole effort each time will be well worth it.