« Connecting Oracle 10g on Solaris Sparc 64 to SQL Server on Windows using FREE ODBC Driver - Part Two | Main | Rhawn Fixes My Electric Bike! »
September 30, 2009
Oracle Database Gets Confused, Then Fixed
On September 9th Sundown crashed and was rapidly repaired by our highly competent staff. However, about a week later I received a call from John Rushing - he could not get his (previously just fine) file i/o application to work. The error he received was "illegal file operation" - not too helpful. The next thing I know Mike Soukup is on the phone. His file i/o application also quit working - "illegal file operation". Well, obviously something had gone terribly bad with the file system when the server crashed. It was possible to log in to the server as the Oracle user and view all directories and files - even write to the files. Dave said that something was corrupted on the Oracle database, but obviously it was something corrupted on the file system! After all, I had even bounced the db without so much as a whimper or an alert log. Well...... no. After several days worth of moving files, creating new directory objects and a host of other "tweaks", I noticed that the alert_aero.log had some recurring errors in it. Researching that error (it was arcane) resulted in advice from Oracle to rebuild the entire SYS schema on AERO due to probable corruption of the schema and/or the data dictionary. Oh.
So, one night Michael Ellis, Ron and I set about rebuilding the entire SYS schema and, sure enough, the file i/o problems disappeared. Let that be a lesson to you, uh, to me - listen to Dave! By the way, I had opened an SR with Oracle for this issue, but this time they were unhelpful. Sometimes an SR is so very helpful, and sometimes it is a waste of computer ink.
I did actually learn quite a bit about creating dictionary objects on the database. The old way (which is what I was still using) was to create a long list of dictionary paths in an initialization parameter called utl_file_dir. Each change or addition required a change to this parameter and bouncing the database to activate the change. Not such a great plan bringing the database down just for that. Well, Oracle must have thought so too, as they implemented directory objects that can be created and activated while the database is running - much better!
Anyway, that's a recent lesson learned about Oracle and crashing servers.
Posted by rossm at September 30, 2009 11:08 AM
Comments
Post a comment
Thanks for signing in, . Now you can comment. (sign out)
(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)