So, the emails start flying, something is wrong, the database has a problem. That is a very typical situation, and instead of defending the database right away, take some time to do a quick check of a couple of things.
Check number one might just be too obvious, but check the alert for errors. Validate that there is nothing goofy going on. And while you are checking out the bdump directory, a quick glance at udump for any trace files that could also be out there might show some information.
Check number two, any invalid objects or unusable indexes? Make sure that all procedures, views, triggers have a status of valid, but before recompiling, make sure you grab that last_modified date, because it might be needed later. Also, unusable indexes that might need to be rebuilt should be noted for what tables they are on and see if they are part of the issue.
Check number three, validate that statistics are up to date on indexes and tables.
And then check to make sure that there are no objects that were recently changed. Check that modified date on all of the objects. Even a modification to a data type can cause a join that was previously working to fail.
Maybe you use the checks in a different order, but with just this four, any obvious errors on the server have been found, anything that has changed has been validated and noted as changes made to the database and statistics have been checked, which can either show that this regular type of maintenance is not running or things are looking good and up to date on the datebase.
So, something wrong with the database, possibly, but now after these quick checks you can pull out more details about what they are seeing and what can be wrong. There is also supporting information if things have been changed or modified and help drilldown to more of the issue at hand.
Wednesday, June 17, 2009
Friday, May 29, 2009
DBA Lock Down
So, what is the sys password really needed for anyway? Not having the SYS password really going to keep a DBA out from logging into as SYSDBA or getting the job done? Well, probably not, especially if this access isn't locked down at the host level. Also, if a DBA is logged in to the host as oracle, there is probably a way to login as sysdba, either as sys or granting the access to the DBA login. Another question, DBAs do you really want to login as SYS? If it is a habit to go the host as oracle, then do a login as sysdba, isn't this just setting you up for trouble? Hopefully there is some sort of auditing in place to capture when the database is accessed as sysdba, but logging into a system with a least privilege user is always a good idea. It not only prevents accidentally doing something on the system without consciously knowing you are going to make a change and need special access, but also gives you the separation of duties from normal day monitoring to performing changes.
There are not too many times that I have needed to log in as sysdba. One example has been at creation and configuration of a new instance. Of course since it is a new instance, there is no data or users to mess up with any changes, a fairly safe way to login. Also, it was needed to restore a database and clone another. Even thinking about that it There are scripts that can be setup to stop and start as well as specific permissions granted, and then logging in as SYS seems not to be needed.
So, what is the big deal about logging in as SYS? Well, besides having all of the permissions to do anything in the database, I guess I have normally viewed it as a best practice and might even protect me from myself. But maybe I have been the only one to shutdown a wrong database. I have also found it easy to complete my job without the permissions and the few times that it is needed, there is a way to grab the password and complete the task.
Hide that password, lock it away, forget you even know about SYS, and use only the permissions needed.
There are not too many times that I have needed to log in as sysdba. One example has been at creation and configuration of a new instance. Of course since it is a new instance, there is no data or users to mess up with any changes, a fairly safe way to login. Also, it was needed to restore a database and clone another. Even thinking about that it There are scripts that can be setup to stop and start as well as specific permissions granted, and then logging in as SYS seems not to be needed.
So, what is the big deal about logging in as SYS? Well, besides having all of the permissions to do anything in the database, I guess I have normally viewed it as a best practice and might even protect me from myself. But maybe I have been the only one to shutdown a wrong database. I have also found it easy to complete my job without the permissions and the few times that it is needed, there is a way to grab the password and complete the task.
Hide that password, lock it away, forget you even know about SYS, and use only the permissions needed.
Thursday, May 7, 2009
Time to apply what was learned...
Even though Collaborate 09 - IOUG Forum has come to a close this year, and in going back home I am thinking of all what can be applied back in the "real world". The amount of learning and information that is packed into such a short amount of time is incrediable. Everything from OEM tricks and tips on installing and configuring to RAC and 11g new features. Support for the current Oracle 10g database has been extended, but with all of the new features of 11g upgrades should be in the planning. Orlando was really the place to be this past week if you use any of the Oracle stack, learning about the individual pieces as well as how they all work together is really a big advantage of having this conference. Getting to know members of the IOUG and learning what they one to hear about and if the sessions that they attended were useful was also great conversations in the evenings. I did really enjoy hearing about all of the different presentations and what was good and not so good. It is amazing that you can pick up a tip to improve your backup strategy, learn how a company is using streams and then the best way to secure you database, all before noon each day. I was also able to step out of my normal database realm and learn about what Oracle is doing the content management and record management area. Then there were also sessions on SAP and Peoplesoft. So, starting planning if you are sorry you missed all of the great learning, Las Vegas, April 2010.
Wednesday, April 15, 2009
Next CPU...
So, if you are like me and having to deal with a very large environment, you probably feel like you just finished patching with January Critical Patch Update. It is April already and the April CPU was released last week. However, since we all have our plan and process in place, it is a piece of cake, right? OK, so we might not all have a complete process in place, and some of this seems that we are just constantly patching databases, but maintaining a secured environment is important.
In reviewing the release notes, there are some important patches to apply, there are new exploits on the database side. The affected components are listed in the documentation as well, allowing for focus in these areas for testing and validation and not having to worry about the other areas. This is also beneficial if when installing Oracle only components are installed that are used, the patches can still be applied, but testing would probably be made very simple at that point if there are is only one or two components that are affected.
Having a policy from the security team in place has really helped with deployment of patching. It isn't just the DBAs saying we need to patch, but overall security policy requiring us to. This has additional support for testing and getting the needed downtime windows. Overall security patching also helps for coordination of the different level of patching from OS to application layers. Exceptions are then required from any application team not able to allow the patching, which will then push back on vendors of these applications, and I believe getting them to work on developing standards around patching and security fixes. I think that this would even help with overall security posture of these systems.
So, policies, processes and patching all good things for those of us supporting these important business applications and environments.
In reviewing the release notes, there are some important patches to apply, there are new exploits on the database side. The affected components are listed in the documentation as well, allowing for focus in these areas for testing and validation and not having to worry about the other areas. This is also beneficial if when installing Oracle only components are installed that are used, the patches can still be applied, but testing would probably be made very simple at that point if there are is only one or two components that are affected.
Having a policy from the security team in place has really helped with deployment of patching. It isn't just the DBAs saying we need to patch, but overall security policy requiring us to. This has additional support for testing and getting the needed downtime windows. Overall security patching also helps for coordination of the different level of patching from OS to application layers. Exceptions are then required from any application team not able to allow the patching, which will then push back on vendors of these applications, and I believe getting them to work on developing standards around patching and security fixes. I think that this would even help with overall security posture of these systems.
So, policies, processes and patching all good things for those of us supporting these important business applications and environments.
Monday, April 13, 2009
Backup Strategies
I really should say recovery strategies instead of backup strategies. Every time I setup a new database or learn about what an application really does, in the back of my mind I am wondering if something were to happen to this database is the current recovery strategy going to work? Sure I can use RMAN and even exports to take backups of the system. I can also verify that backups run every night and the tapes are good, but is the application going to be in a state that I can recover it and is it really going to be as simple as recover database.
In moving to even a more high available system with RAC, I wonder if that because you can failover to another node backup strategies might not be considered as important. But there are so many other things that can go wrong. What if a security patch isn't applied correctly or a hotfix for the application is rolled out and results in a table are incorrect because of it? Or even better, because you and I know that there are places for ad-hoc queries in applications, and someone runs and update or changes a table structure, what is going to be the best way to recover now?
I think that the best thought out backup strategies are ones that include these thoughts and considerations. Thinking of the end result of actually recovering a database can give insight to what needs to be backed up and how frequently. Also the understanding of what pieces might be the most important and customized. In a large environment it is very difficult to implement several different strategies, but at least considering if I have RMAN, flashback and exports implemented, which one am I going to use first to recover. Can I just flashback a query or a table and how big does that flashback area really need to be to provide what I need to be able to get it back quickly. Import might take too long to run, but can I use that information in a test database to reconstruct what is needed to not have the production system down. With the high availbility can I failover quickly, or do I have a place to run a restore from RMAN in a real disaster?
So, think recovery and think what things are in place to restore a database, and if you want to even have more discussions about this, join me at Collaborate09 - IOUG Forum, which will be a great place to discuss recovery techniques as well as learn other things near and dear to Oracle technology professionals.
In moving to even a more high available system with RAC, I wonder if that because you can failover to another node backup strategies might not be considered as important. But there are so many other things that can go wrong. What if a security patch isn't applied correctly or a hotfix for the application is rolled out and results in a table are incorrect because of it? Or even better, because you and I know that there are places for ad-hoc queries in applications, and someone runs and update or changes a table structure, what is going to be the best way to recover now?
I think that the best thought out backup strategies are ones that include these thoughts and considerations. Thinking of the end result of actually recovering a database can give insight to what needs to be backed up and how frequently. Also the understanding of what pieces might be the most important and customized. In a large environment it is very difficult to implement several different strategies, but at least considering if I have RMAN, flashback and exports implemented, which one am I going to use first to recover. Can I just flashback a query or a table and how big does that flashback area really need to be to provide what I need to be able to get it back quickly. Import might take too long to run, but can I use that information in a test database to reconstruct what is needed to not have the production system down. With the high availbility can I failover quickly, or do I have a place to run a restore from RMAN in a real disaster?
So, think recovery and think what things are in place to restore a database, and if you want to even have more discussions about this, join me at Collaborate09 - IOUG Forum, which will be a great place to discuss recovery techniques as well as learn other things near and dear to Oracle technology professionals.
Thursday, March 12, 2009
Repeatable Process Worth the Effort
So, I might be stating the obivous here, but taking the time to create, develop and review a process to get a task done is always going to provide benefits, make things more efficient and produce better results. Take for instance, upgrading databases or applying patches that is something that will consistentantly be part of the life of a DBA. What if the deadline to get the upgrade done very quickly and there was a need to show results as soon as possible. So, is it showing results by developing a process, and putting together a test plan?
Isn't that some of the problems we have when faced with deadlines? We might have to upgrade a database much quicker then planned so the steps or a test plan may not be documented as needed. Then if wanting to handover the upgrade to another team member or team for patching in production, there is time wasted "guessing" what was done in the test environment because there wasn't time to at least document the steps or create the process.
Even if there are only a couple of databases this time around, there will be future upgrades and patches to be applied. A repeatable process, a plan that is documented can go a long way for current and future tasks.
With the IOUG Security Patching survey results, I have been ask recently about what it takes to get the patches out there, what are some best practices. My thought is a repeatable process. We can collect best practices on upgrades, adapt them for our environments, create test plans around the applications and other pieces of our environment, throw in a little bit of documentation and then before we know it, a repeatable process. The trick here is to setup this process the first time around while not putting the deadlines at jeopardy. Honestly it might take working more hours in a day, but not having to go through the whole effort each time will be well worth it.
Isn't that some of the problems we have when faced with deadlines? We might have to upgrade a database much quicker then planned so the steps or a test plan may not be documented as needed. Then if wanting to handover the upgrade to another team member or team for patching in production, there is time wasted "guessing" what was done in the test environment because there wasn't time to at least document the steps or create the process.
Even if there are only a couple of databases this time around, there will be future upgrades and patches to be applied. A repeatable process, a plan that is documented can go a long way for current and future tasks.
With the IOUG Security Patching survey results, I have been ask recently about what it takes to get the patches out there, what are some best practices. My thought is a repeatable process. We can collect best practices on upgrades, adapt them for our environments, create test plans around the applications and other pieces of our environment, throw in a little bit of documentation and then before we know it, a repeatable process. The trick here is to setup this process the first time around while not putting the deadlines at jeopardy. Honestly it might take working more hours in a day, but not having to go through the whole effort each time will be well worth it.
Wednesday, March 4, 2009
Black Belt Attitude
I started martial arts recently, and our instructor was describing to us the how important attitude is during class and outside of class. The questions were posed do you have a "Black Belt Attitude"? Do you have a "Can Do" attitude? Black belts have a positive attitude and they can get it done no matter what it takes. So, I can look at class with the thought that I am just a white belt, there is no way I can do these things yet that he is asking, or I can be there trying every move, being enthusiastic that I am going to get it and setting my goal for the black belt.
The attitude doesn't stop with class. This is something that can easily be carried over to other parts of life, especially work.
A positive attitude at work goes along way for how things get accomplished. Taking ownership for the task at hand and to do it to the best of your abilities, setting goals to develop new skills and keep other skills and knowledge current, willing to take on new responsibilities or even ones that others don't want, these are all part of that "Black Belt" attitude.
There are tasks I don't want to do and people I may not want to deal with that pull me away from my goal of developing this attitude. There are projects being cut, people being given less incentive to do their current tasks, but this should push us even more to do what we can with what we have. Those of us who stay positive and work now maybe a little harder and smarter will be reaching that goal even sooner.
Just like I can't go from being a white belt to a black belt tomorrow this attitude also can't happen over night. There is training that is needed with in both technical and mental skills. Developing the attitude of "I can do this" and learning to maintain that good attitude is a key part to the mental area. Along with this training, focusing on a goal is helpful. My goal is to earn a black belt, learn something new and conquer a challenge. I am also not alone, so when my bad attitude surfaces there are people who can assist. It is good to have accountability for meeting goals and staying on track. Having people I can learn from and encourage is important and good attitudes are contagious. For martial arts, I have a class to go to with my girls, but for work I have IOUG, user group network. I think that this is a main reason that I have been active in the user group community and enjoy sharing and learning from others. So, I encourage you to get involved in a community to help sharpen your skills and have the accountability to do an attitude check.
Just image what would happen if we all came to work with a "Black Belt Attitude". The encouragement, positive outlook and the willingness to get things done could make projects happen that you never thought possible.
The attitude doesn't stop with class. This is something that can easily be carried over to other parts of life, especially work.
A positive attitude at work goes along way for how things get accomplished. Taking ownership for the task at hand and to do it to the best of your abilities, setting goals to develop new skills and keep other skills and knowledge current, willing to take on new responsibilities or even ones that others don't want, these are all part of that "Black Belt" attitude.
There are tasks I don't want to do and people I may not want to deal with that pull me away from my goal of developing this attitude. There are projects being cut, people being given less incentive to do their current tasks, but this should push us even more to do what we can with what we have. Those of us who stay positive and work now maybe a little harder and smarter will be reaching that goal even sooner.
Just like I can't go from being a white belt to a black belt tomorrow this attitude also can't happen over night. There is training that is needed with in both technical and mental skills. Developing the attitude of "I can do this" and learning to maintain that good attitude is a key part to the mental area. Along with this training, focusing on a goal is helpful. My goal is to earn a black belt, learn something new and conquer a challenge. I am also not alone, so when my bad attitude surfaces there are people who can assist. It is good to have accountability for meeting goals and staying on track. Having people I can learn from and encourage is important and good attitudes are contagious. For martial arts, I have a class to go to with my girls, but for work I have IOUG, user group network. I think that this is a main reason that I have been active in the user group community and enjoy sharing and learning from others. So, I encourage you to get involved in a community to help sharpen your skills and have the accountability to do an attitude check.
Just image what would happen if we all came to work with a "Black Belt Attitude". The encouragement, positive outlook and the willingness to get things done could make projects happen that you never thought possible.
Subscribe to:
Posts (Atom)