True Tales From IT Land

Before I joined a financial services company I was a systems programmer, but for the last 15 years I've been doing a variety of jobs in what's known as "IT" (Information Technology). During those 15 years some goofy things occurred which I will share with you reader.

The Never Ending Elevation
No, this doesn't have anything to do with an erection lasting longer than four hours. An elevation is the introduction of new software into the production environment. Think of it as software having to get a job following college. Well this particular elevation involved the company's Intranet application with some major new features: single-signon with Windows and personalization of the home page.

Note: These two are interdependent. The single-signon allowed us to identify the user and this permitted the user's preferences to be stored and recalled.

This release had been delayed for several months because the project was dependent on another team to deliver the single-signon solution. When this feature was finally ready we scheduled the elevation to start on Friday evening and finish on Saturday. From the start things did not go well. There were configuration changes that could only be tested in the production environment because it was deemed too costly to replicate the hardware. By Saturday evening it was apparent that the elevation would continue into Sunday.

This was right at the time of my life when I was suffering from severe lower back pain. The stress of a major elevation (BTW I was the Technical Lead responsible for the elevation) was compounding my physical discomfort as I followed the proceedings by an extended conference call.

By Sunday morning we had the configuration right so that the single-signon part was working. Testing of the home page preferences was giving mixed results, however. Sometimes it worked and sometimes it did not. What was particularly strange was that the tests always worked in our test environment.

We finally isolated the problem to the database -- the production database was not behaving like the test database. It was time to get the on-call DBA on the line. This was my job and I spent four hours on the phone with the guy. He couldn't resolve the problem but I couldn't let him go because my boss wasn't going to let us go home until the problem was solved.

Finally the DBA called one of his colleagues and learned that a patch was needed to the production database driver. The patch had been applied everywhere else, but not to production. Finally the patch was applied, and everything worked. The elevation had taken around 40 hours to complete. 40 hours of pure hell.

The Urge to Kill
For some reason as Technical Lead I was expected to know every operational aspect of the Intranet. In other words when there was any operational problem, I had to diagnose it and fix it. The common term for this procedure was to find the "root cause" even if this meant tracking down the cosmic particle that flipped a bit on the memory chip of the server.

Remember I said that our Intranet had single-signon? This means that if you signed on to your Windows computer you did not have to signon to the Intranet application as it already recognized your login. This was no small feat because our Intranet ran on Unix servers with all kinds of crazy network devices in the picture.

For 99.999% of the time the single-signon worked perfectly but there were occasions when out of the clear blue, the home page would present a login form. We had seen this during testing and knew that the cause was some process consuming an unexpected amount of CPU such that the server could not complete the Kerberos protocol within a given number of milliseconds.

Usually this problem wasn't noticed and everything went back to normal. But one day operations noticed it and filed a trouble ticket. I was assigned to resolve the trouble ticket. Now our Intranet involves at least a half dozen servers not to mention specialized network devices. The problem was sporadic and did not follow a pattern. Here's your haystack; have fun finding the needle.

To make matters worse I had to report status via a daily teleconference called the DSR (Daily Status Report). The DSR is designed to improve the overall operations at the firm by the relentless pursuit of "root cause." Actually it's a means to pillory Tech Leads who have no idea why the software occasionally misbehaves.

Anyway, day after day I had to report no progress. Then suddenly someone from operations was on the call and said that he had traced the problem to one of the servers that was used to store employee pictures. He further stated that the problem occurred when a certain script was run by the database group. He arranged for a rep from that area to attend the DSR the next day.

The next day a manager from the database group was on the teleconference and was confronted with the miscreant script. He said "we don't have to run that script." The case was closed without the slightest apology to me or the rest of my team who had spent hours trying to diagnose it.

The Computer is in Control Here

My last tale is one of the most mysterious that I've ever had in almost 40 of computer work. Without going into tremendous detail I'll just say that I made a configuration change in the way one server authenticated to another server using stored credential (username/password) data.

To improve operations I needed to change the credentials the server used. First I tested this on my desktop by running the first server there. The credentials change worked fine.
Next I proceeded to the production environment. To ensure it worked as I made the change I monitored the second server's log file. If the new credentials were rejected it would show immediately in the logs, and I would be forced to back out the change. But the change worked and I could see it working in the log files.

About 30 minutes passed and the next thing I know my colleague tells me that he's seeing authentication errors on the first server. So I go directly to the credentials file that I had changed and see that it had reverted my change and is now using the previous username but no password. This is truly bizarre and marks the first time I have seen the system spontaneously reject a configuration change made by a human.

My Polar Plunge

"This is stupid," I thought. I was standing in my cabin wearing a bathing suit and a thick terry cloth robe with Ocean Nova embroidered on it. Ocean Nova was the cruise ship that had taken me to Port Lockroy (64^o50'S 63^o30'W) in the Antarctic Peninsula. What I thought was stupid was taking the polar plunge into the icy waters just for the right to brag about it later in life.

I walked down to the deck where we usually exit the ship to board the Zodiac boats that had been taking us to the shore for excursions. I found more than a few of my fellow passengers were willing to take the plunge. Plus the staff were prepared to give each plunger a shot of brandy afterwards. "Okay, I'm doing this," I said to myself.

I was the fifth in line. The process consisted of putting on a life vest and tying a pull line to it. The first plunger suited up and then jumped. A cheer went up from the onlookers. Now standing sans bathrobe in approximated 35^o F temperature I start to feel the cold. Plunger No. 2 goes next. A splash followed by a cheer. Same for plunger No. 3. Then while plunger No. 4 was suiting up there was an unexpected "kersplash." The ship's head chef, a real daredevil, had just jumped off the top deck of the ship without life jacket or safety line. "What a jerk," I thought later. "Just think if the guy got injured -- no food for the passengers."



They fished out the chef and it was back to the process. There goes No. 4 and someone points out that the plungers are not looking at the cameraman taking pictures from the Zodiac at water level. "I will not going to disappoint the cameraman," I resolved.




Now it's my turn. The fricking life vest is freezing and my teeth are chattering big-time. I can't wait to get this over with. I look directly at the cameraman and make a silly face. At least that's what I tell people when they see the pictures. I leap.



"Jesus H. Christ it's cold," I tell myself as I sink below the surface. I feel them pulling me right out of the frigid water. I get some water in my mouth and it's salty alright. I'm so eager to get out that I slam my left heel into the gang plank with a force that leaves it hurting for weeks afterwards.


Okay, I survived. I throw down the brandy and it warms me ever so slightly. I'm shivering full throttle now despite the fact that I have the Ocean Nova robe back on. I hurry back to my room feeling warmer the drier I get. I dry off completely, put on clothes, and lie in my bunk knowing that I did it. The battle's over, the war is won.



Facts:
1) 10 minutes in frigid waters is considered life-threatening.
2) Lynne Cox, world champion open-water swimmer, swam 1.2 mi. off the Antarctic Penisula for 25 minutes.

Trip to the End of the World

It's when I travel that I feel most in the hands of God. Today I'm embarking on the longest physical journey of my life. In the next 26 hours I will travel 6,500 miles from Philadelphia, PA to Ushuaia, Argentina on the island of Tierra del Fuego.

I'm on my way to the Antarctic Peninsula. Why Antarctica? It's a place I've always want to visit since I saw the series "The Last Place on Earth" which recounted the race between Amundsen and Scott to be the first humans to the South Pole. Their story was full of hardship and deprivation, victory and the cruelest form of defeat. I appreciated their thirst for adventure and I wanted to see what they saw even it's in the form a comfort-filled expedition on a modern cruise ship.

The trip to Ushuaia consisted of three flights: Philadelphia to Atlanta (2.25 hrs.), Atlanta to Buenos Aires (10 hrs.), Buenos Aires to Ushuaia (3.5 hrs.). When I arrived in Buenos Aires it was morning and I had to wait three hours between flights. I was surprised how warm it was as I walked between terminal buildings. I sunned myself outside the domestic terminal because the terminal lobby was small and noisy.

I had a window seat on the flight to the city that calls itself the southernmost city in the world. As we approached the island I could see distinct waterways that separated the small but rugged mountains that filled the many small islands. These mountains were sporadically covered with snow as if they were decorated by a confectioner. The plane passed directly over the airport which looked like a field strip for small aircraft. But then the plane circled and dropped altitude and I realized that the small strip (Malvinas Argentinas International Airport) was our destination. In a minute we had landed at Ushuaia, my port of call for Antarctica.

At Home and at Work with a Kill-A-Watt

This is a Kill-A-Watt. This is a Kill-A-Watt in action.

It's a specialized device that measures instantaneous energy consumption. I used it at home to measure power consumption from a Verizon set-top box under three conditions: 1) the box "off" 2) the box tuning a channel 3) the box tuning a channel with a 40" LCD TV on. I was surprised that with the box nominally off, it was drawing around 28W of energy. While tuning a channel it fluctuated around 28.5W. Finally with the TV on, the power consumption jumped to around 128W total.

So I called Verizon technical support to find out if I could turn the set-top box off overnight. They recommended not doing that. Why? During the night they occasionally distribute software updates to the unit. If the unit is off during an update, it may not be able to get it later resulting in a support call. Now that's 28W per hour on a 24 x 7 basis. That amounts to 245 KW per year times two since I have two set-top boxes in my house. That amounts to over 6% of my home's energy consumption last year!

I repeated a similar experiment at work attempting to measure my PC's energy consumption under these conditions: 1) the PC "asleep" in the morning with the monitor powered off 2) the PC "asleep" but displaying the login screen 3) the PC and monitor in use. I came up with these numbers: 108W, 175W, 190W.

I was surprised by the results of these experiments. The Verizon policy and the Vanguard policy concerning not powering-off the device were the same. Vanguard's policy revolves around the need to distribute software to PCs. The TIP Green Team, of which I am a member, has been told that Tech Ops is looking into powering off PCs remotely when there are no software distributions scheduled. If only Verizon would do the same for me!

The Four Greatest Mathematicians

"History of Mathematics" was my final course in the master's program. I took it because I thought it would be fun, informative, and perhaps easy. The textbook we used was not the dreadnought A History of Mathematics by Victor Katz, but the sloop Journey Through Genius by William Dunham. Both books emphasize the development of mathematics from the perspective of Western civilization, but let's quickly thank the Arabian scholars who preserved the works of the ancient Greeks from annihilation. It was those Greeks who got the ball rolling with demonstrative mathematics around the sixth century B.C. It was the Arabs in the twelveth century who translated and expounded on the Greeks, and brought their works back to Europe.

During the course I learned the top four mathematicians of all time:

• Archimedes (287 - 212 B.C.)
• Newton (1642 - 1727)
• Euler (1707 -1783)
• Gauss (1777 - 1855)

I have taken the liberty of writing a short obituary for each of these towering figures.

Archimedes was a true self-starter who liked to apply math to everyday problems. At an early age he invented a water pump called the Archimedes screw which is still in use today. He exploited the techniques of "method of exhaustion" and double "reductio ad absurdum" to prove many theorems regarding familiar two and three dimensional figures. In order to save his city Syracuse from ruination by the Romans he devised innovative war machines to repel the overwhelming army.

Newton was perhaps the greatest self-learner the world has ever known. In his early twenties he spent two years in rustic isolation during which he: discovered the generalized binomial theorem; invented differential and integral calculus; recognized the universal gravitation as the key mechanism of the solar system; and developed insights into the nature of light by refracting it through a piece of glass called a prism. Having little interest in publishing his world-shattering findings, he turned his attention to alchemy for almost 40 years, but was persuaded to publish something by Edmund Haley. The result was a physics text called Philosophiae Naturalis Principia Mathematica which is considered the greatest contribution to science ever made by one man.

Euler was the most prolific publisher of pure and applied mathematics in history. His mathematical contributions ranged over number theory, calculus of variations, graph theory, complex analysis, and differential equations. He applied math to acoustics, engineering, mechanics, astronomy, and optics. The publication of his complete works was started in 1911, and the end is not yet in sight. Originally planned for 72 volumes, the discovery of new works pushed the project to an estimated 100 volumes. In order to aid his work in number theory he memorized the first 100 prime numbers, their squares, their cubes, all the way to their sixth powers. As a child he memorized the entire Aeneid and could recite it flawlessly late in life.

Gauss was a child prodigy who at the age of 17 invented a technique for inscribing a regular 17 sided polygon within a circle. This was a ruler and compass construction that stunned the math world since no one since classical times thought such a construction was possible. For his Phd. thesis he proved the Fundamental Theorem of Algebra, but being a perfectionist (more about this later) he improved upon the proof three times in subsequent years. His superlative text on number theory established modular arithmetic as the fundamental tool for its study. He tired of pure mathematics and turned his skills to scientific endeavors. In order to predict the position of the asteroid Ceres, he invented the technique of least squares and error theory in general. He mapped the earth's magnetic field, and along with Weber invented an early form of the telegraph. His perfectionist nature made him reluctant to publish until every detail of proof was beyond criticism. One biographer claimed that his unpublished work would have advanced mathematics by 50 years.

The most shocking thing I learned from this course is that all of the math I learned in high school, college, and graduate school only brought me up to around the year 1900. Seems that the twentieth century and beyond is reserved for Phd. students.

WHYY recording session (11/11/2009)

I submitted a "This I Believe" essay to the Philadelphia public radio station WHYY, and was lucky enough to have my submission accepted. It was about math and though I thought it was a longshot, I knew it was unusual enough to attract attention. And yes, it was selected (broadcast on 11/13/2009). So this blog is about my adventure to the WHYY radio station to record it.

I left from Villanova station since I had a math class that very evening. In fact the building in the background is Mendel Hall where I have all my math classes.


No use in getting nervous so to relax on the train I'm playing Sudoku on my iTouch.

I exit the train at Market St. East and emerge from the station at 10th and Arch Sts. In this shot you can see the entrance to Chinatown and that storefront to the left below the traffic light is the Tracadero. The Troc is now a rock music venue and my son Tony has played at the balcony stage.

Where is the WHYY studio? Well it's across the street from the Consitution Center.

Here's the entrance.

And here's the reception desk. The receptionist didn't want her picture taken and is trying to hide in the picture. Why do people do that?

The station person who selected my essay was running behind schedule. She took me right to this studio where I snapped this picture. There's a special clock to the right of the screen that tells the radio people exactly how much time they have. It looks like the doomsday clock.

I was hoping to get some photos of my host, Elisabeth Perez-Luna, and myself but there was no time. You would see me in the photo below, but I had to take the picture myself.

After 30 minutes of reciting this essay over and over again I was beat. I tried my damnedest to put on a "radio voice." Ms. Perez-Luna had me make some alterations to the essay and she recorded it all using ProTools. I can't say what version is going to end up on the air after editing.

Goodbye WHYY, it was quite an experience.


Note: My essay was broadcast on Friday, November 20 at 5:45 pm. The station failed to give me any advanced notice so I missed it. You can find the essay and a podcast here.

Is There Anything as Simple/Complex as a Group?

My summer 2009 session course was Group Theory. What is a group?

A group is an ordered pair (G, ·) such that G is a set, · is an associative binary operator on G, and $ e Î G such that:

1. If a Î G the a · e =a,
   


2. If a Î G, then $ a^-1
   Î G such that a · a^-1 = e.
   

What could be simpler? A group is a set of objects and some kind of operation that has straightforward properties. Groups are all around us and come in all shapes (2D, 3D) and sizes (finite, infinite). Examples of finite groups are:
1) the symmetries of a square and rotations through various axes and
2) the set {0,1,2,3,4,5,6,7} with modulo 8 addition.

Examples of infinite groups are:
3) the set of integers and addition
4) the set of rational numbers – {0} and multiplication.

Okay we have sets of objects with an interesting behavior. Can we find any more of them? Well how about taking a subset of them and seeing if they are also interesting. Viola, consider the subgroup.

A subgroup of a group G is a subset which is a group under the operation in G. More precisely, a subgroup of a group (G, ·) is a group (H, *) such that H is a subset of G and * is the restriction of · to H x H. This means that multiplication is the same in H as it is in G.

Looking for subgroups in the examples above:
1) the horizontal and vertical rotations of a square is a subgroup of all the symmetries
2) {0,2,4,6} with modulo 8 addition is a subgroup of Z₈ above
3) the set of multiples of 3 is a subgroup of integers and addition
4) the set of rational numbers with even denominators – {0} is a subgroup.

Really simple stuff, right? Well let's define two terms, a theorem, another term, and another theorem and reconsider.

Definition: The normalizer of N(S) of S in G is defined by N(S) = {x Î G xS = Sx}.

Definition: If S and S' are subsets of group G, then S is conjugate to S' iff
$ x Î G such that S' = x^-1 S x.

Theorem: Conjugacy is an equivalence relations.

Definition: The conjugate class on a subset S of a group G is the set Cl(S) of subsets S' of G which are conjugate to S.

Theorem: If S is a subset of a group G then [G:N(S)] = o(Cl(S)).

Oh no, now we're in the deep end of the pool and we don't know how to swim. This is typical of Group Theory (and higher math in general). Define something that's pretty simple and prove some stuff about it. Then define a new abstraction on top of that and prove some more theorems. Then define
some more stuff on top of THAT... You get the picture.

In this manner group theory leads to these kinds of groups: cyclic, abelian, free, solvable, divisible, decomposable, p-groups, supersolvable, M groups, etc. How can something so simple now be so complex?

P.S. the text was "Group Theory" by W. R. Scott which I would avoid like the plaque. It has very, very few examples and I think examples are needed to wrap your head around all of the abstractions being thrown at you.