During one phase of my career I was the entire IT department for a startup satellite communications company. I was responsible for software requirements management, design, development, testing, deployment, 24 x 7 operations, system administration, network administration, and customer service. Our system was redundant and built in house from the ground up. Due to financial constraints we weren't able to purchase third party software or hardware (or people!) to monitor system health.
Before this job I was an innocent and happy software developer. Afterward, I had a lifetime of experience and I needed therapy. Here are a few of the lessons learned pertaining to high availability and 24 x 7 operations:
That startup company was called Mobile Datacom and is now a division of Comtech Telecommunications (CMTL). CMTL was just added to the Standard & Poor's Smallcap 600 index the day of writing of this position paper. I left Mobile Datacom and am now a senior developer building J2EE applications. I hope participate in the 'Fostering Software Reliability in an Increasingly Hostile World' workshop to hear of others' experiences with building high availability applications in the J2EE application server arena.