Cheers!

This week we decided to move the go-live date of our SIS implementation. Here is a good summary explanation from another source: Load testing was conducted this month to test the system’s capability to handle the student information demands the colleges will generate, and the results were not sufficient to meet our needs. Certain other aspects of the new system were testing insufficiently as well. Therefore, the system will continue to be enhanced in the ensuing months to ensure a successful delivery next spring. We simply cannot compromise our ability to meet the needs of our student population, particularly during a high-registration month.

And so we decided to move the date.

And so you may ask, “where’s the pony?” The pony is that Maricopa made a wise decision not to gamble with the student experience. There is no doubt we could have brought the system up as scheduled, but with some real risk of significant disruption to students and faculty. Our egos might have said “yes, we can do this. Let’s go for it!” but the compassionate voices were at the forefront and said “let’s not make it harder for our students to succeed. Let move the date and make sure it is right.”

Some times the best things you ever do are things you decide not to do. Here’s to good choices.

It’s September and we have the first two weeks of the formal semester under our belts. Our main teaching and learning environment did not fare well. First it was slow, then it was down, then certain parts of it were still slow, and during this writing it seems repaired. The root problems are mostly poorly designed software that only reveals itself when being pounded by a very large institution like Maricopa. On one hand, it’s disappointing the bad code exists. On the other hand, the vendor has done an admirable job of helping us locate the problems and writing fixes.

I could fill up this blog with comments, frustrations, suggestions and metaphorical analogies that we’ve received the past two weeks. Being a songwriter, the metaphors show some real insights and creativity. Our situation has been drawn as a parallel to B52 bombers too heavy for take off, 18 wheeler semi-trucks that could be replaced with 12 pickup trucks, and the inevitable suggestion “put a fresh hamster in the wheel.”

My own predominant analogy has been with a firefighting situation. Once the software began to malfunction under load, we had an urgent situation to deal with. When there is a fire in the canyon, the calls to figure out how the fire started, and whether we should have allowed campers in the forest, and whether we needed a controlled burn last winter — well, they are important conversations to have, but first we need to put out the fire.

Our vendor is still working on some particularly complicated fixes, but it is looking like we have the fire controlled. The next big test comes Tuesday when “high winds” are expected. The high winds will come in the form of tens of thousands of students returning to their studies after a 3 day weekend. It’s a little strange to be home on a Saturday of a three day weekend and thinking “I can’t wait for Tuesday to get here.” Stay tuned…

I think it is a matter of looking for a middle ground. All things considered, we aren’t going to go 100 mph, so we have to be reasonable. On the other hand, unless we embrace some significant change we don’t reach our goals.

Dear Maricopa,

In case you have not already been informed through other sources, yesterday the District Office computer room suffered a power outage at approximately 12:30pm due to a malfunction between the Uninteruptible Power Supply and the Power Distribution Unit. Ironically, these devices are used to ensure a proper response to power fluctuations, but in this case they were the source of the problem, not the solution. Due to some complications in resetting the UPS and PDU, the power was out to the entire data center for approximately 3 hours.

Once power was restored, ITS went through a careful process of bringing services back online, and ensuring proper operation of the applications. This process takes about 2 hours of clock time. Almost all services were completely restored by 6pm. We had some lingering issues today caused by the after effects of computers and databases suddenly losing power. To my knowledge, these are now fixed.

As is always the case, we will take away lessons from this experience. We will make changes to the ways the UPS and PDU can be powered on and off. We will sharpen our communication, because we learned yesterday that when both phone and email are completely out, we need to have cell phone numbers, and alternate email accounts on standby. There will be other changes required as well.

We also can use this as a message that Maricopa needs a second data center. Gratefully, we had already identified and acquired a property for this use, and will begin to make plans and allocate budget to make it a reality in the months ahead.

It probably goes without saying, but we regret the outage of yesterday and we in ITS pledge to use it as a learning experience to do a better job in the future.

***********************************************************************

Now it is a couple of days later, and I have these further comments:

1. I am really proud of the core service providers in ITS. The DBA’s, the helpdesk, the server team(s), the network team, email team and the applications folks. Presented with an unexpected wrinkle, they handled it with professionalism. At one level, it was an unscheduled test of “what happens if someone pulls the plug?” We found out some things, and we had people back in business PDQ.

2. We learned that having the phones down is a bad combination with any other outage. I’ve been exploring external services that would send SMS messages to a pre-determined group. So far the one that looks promising for me to use personally is Communicator from Clickatell.com. Stay tuned for a progress report.

3. It confirms a belief I have had for a long time. The problem you prepare for is not the problem that is going to happen. So now we will prepare for having the phones out at the same time e-mail is out. However, the outage gremlins have something else in store for us, I bet.

4. As I said in my note, I am glad we have begun the process to bring up a second data center. One thing that we don’t talk about that much, because it sounds like excuse making, is that it costs money to never fail. In the months ahead we will no doubt be talking more about investing money to reduce our outages. However, I’ll predict that when we look at the total cost of doing all the things that make sure we never have an outage, we will surely decide to live with some risk so that we can keep our investments at a frugal level. Stated simply: Google and Amazon can afford some strategies that we can’t.

5. The biggest lesson learned is that work is more enjoyable with technology tools. Once the data center went down, time dragged a bit for most employees. We had things to do, but the stuff we needed to do the most required technology. In a way, that’s reassuring. We are on the right track.

Another glimmer occured when the Dean of IT at Rio had the following cool suggestion. I am paraphrasing: “since I have a license to sell software to authenticated Rio employees for home use, and since we have District-Wide authentication working for the Sharepoint servers to support the Blue Ribbon Panel… what if I can extend our license so that all Maricopa employees can authenticate and buy authorized software for at-home computers?” I am not filling in all the details here, but it is an example of how a couple of pieces fit together and then you can do neat things.

Another example is acquiring a building block for Blackboard to allow text messaging to students from within Blackboard. It turns out the company that does that will also enable Colleges to send text messages, essentially for the price we’ve already paid, outside of the Blackboard environment as well. Technology isn’t always complicated, sometimes it makes things simpler.
That’s why we are going to all the pain of implementing a modern Enterprise Application System for Students. It sets the foundation to be able to analyze the data in new ways, to offer more services, for more hours in the day, and then throughout 2008 and beyond people will say “Since we are already doing X, and we know how to do Y in the new system, what if we started doing Z?” And because of what the teams did in the prior years, the answer will be “Yeah, Cool. Why not?!?”

The SIS project also had a milestone to begin integration testing in June. More heroics, and that happened too. 60 people came to integration testing kickoff. That shows that across Maricopa people are mobilizing to shake the problems out during the summer so that things go smoother in October. Inspiring. We discovered that our planning for the number of remote printers was off, almost by a factor of ten. At first glance, that’s a bummer — but on second thought it is wonderful… we know NOW that we have to do some scrambling, instead of finding out on a Monday morning in mid-October. Good job.

Besides that, District Office staff located a good site for a 2nd data center, and moved with record speed to acquire it. A longstanding need is on the threshold of being achieved. Also our Identify Management development accomplished their target to roll out product by end of June. We have a place on the ITS website that summarizes all our points of pride. You can get ongoing details there. But I wanted to mention here that July has been cancelled, and June has 61 days this year.

Congrats to ITS and to the SIS/NSS project teams. Keep the winning streak going!