Head Above Water

Head above Water…

I’ve been working a lot lately. We’re right at the “go live” date for the big Trading Project at work, and there’s been the inevitable last minute push to get things all ship-shape.

One big issue is that we’ve been developing on a special Development environment. This is kind of a sandbox set of computers, where we can code up stuff, try different things, and not have to worry about sending live trade orders to Wall Street. However, this Dev environment only has a small subset of data.

Once we got things working in Dev, we moved the code over into a more official “Parallel Testing” environment. As the name implies, this environment is parallel to the actual live Production environment. So while it’s still a playground, it’s a full set of data (tens of thousands of accounts, instead of hundreds).

We’ve been doing performance testing in the Parallel environment, and uncovered some pretty serious issues. For example, if a certain process (placing an order) takes two minutes for two-hundred accounts, how long will it take for 1000 accounts? Since 1000 is five times bigger than 200, you might say it will take five times as long: ten minutes. That answer is correct ONLY if the processing time scales linearly.

We’ve found some cases were the processing time was scaling exponentially. Things that were perfectly acceptable (a few minutes) in Dev were taking HOURS or DAYS in Parallel (if we extrapolated… we’d killed processes after a couple hours). That was obviously unacceptable.

A small group of us worked pretty hard this weekend and made some really big improvements. Because some of the original stuff was designed so poorly, now that we’ve learned a lot, we were able to really see what was going on (lots of this stuff was designed by contractors who are no longer with us), and come up with some good fixes. What’s really satisfying about the whole thing is with some relatively minor changes, we took processes that were taking hours and turned them into minutes. Or something that took 15 minutes, and now runs in 30 seconds.

To make it a little more concrete: imagine that when you clicked “Send” on your email, you had to wait 2 hours to use your computer again. Now imagine that someone comes and tweaks something, and now sending that same email takes 5 seconds. We did stuff like that.

When the head of the Systems department comes in on Monday and calls you a “rock star”… that’s never a bad thing.