We have a rule at iotum that we don’t deploy upgrades on Friday. It’s a simple recognition that folks would like to go into the weekend without the spectre of a major outage haunting them. Much as the folks at Facebook describe their experience with deploying server upgrades, our philosophy is to deploy, listen and fix rather than polish apples. It works well for us, and lets us crank out updates to Calliflower on a two or three week cycle, which means that we can be much much more responsive to customers feature requests than a traditional software house can. It also means that releasing on Friday is a bad idea, because we often find small gotchas after any release.
We should have followed our own rules last Friday. Instead, Friday afternoon Noam (our Director of Products and Engineering) and I made a decision to deploy a “critical” fix that upgraded our PIN range from 4 digits to 5 digits. We had a customer who needed to send invitations to a public call to more than 10,000 people. Noam proposed we make the upgrade, and before I could object, assured me that he would babysit the servers over the weekend personally. I don’t know about you, but I like that kind of dedication, so I agreed and said “Yes, let’s break the rule”.
Murphy struck. The ramifications of that change flowed through the Calliflower system like a bad smell through an air conditioning duct. By about 8 pm, complaints from upset customers started to flow in and we had a small scale emergency on our hands. It wasn’t enough to change code. Voice prompts had to be re-recorded, switches restarted and more. At 1:43 in the morning everything was in good shape again, after an evening of yeoman’s effort by Noam and senior dev Rob.
Thank goodness it was Friday evening of a holiday weekend. Not too many people were affected.
For those Calliflower users that were affected by these problems, please accept our humble apologies.
And for the rest of you reading this… well, that’s why we don’t release code on Fridays.
2008-09-01 3:11 pm | 1 Comment »
Tags: Tech and Business, calliflower, Iotum, management
Bowing to pressure from irate customers earlier this year, Rogers cut data rates dramatically just ahead of the iPhone launch. For a limited time, Rogers is offering a 6G data plan for $30/month. At the time Rogers announced that these new data rates would apply across the board to all smartphone customers, whether iPhone, BlackBerry, Windows Mobile, or Nokia devices. And subsequently, it extended that offer to the end of September in order to allow new BlackBerry Bold owners to also take advantage of it.
Little noticed, however, was the provision that Rogers new data pricing didn’t apply to users of the BlackBerry Enterprise Server (BES) — a large constituency of BlackBerry owners. If you get your BlackBerry email from a BES, you pay from a different rate table.
| |
Other Smartphone Users |
Blackberry Enterprise Server User |
| 300M |
$30 |
$45 |
| 1G |
$60 |
$100 |
| 6B |
$100 |
Not available |
When iPhone 3G came out I signed up for the $30/mo 6G special rate. Even taking advantage of Rogers new $45/mo fee for 300M of data on my BlackBerry, data on BlackBerry is still costing me 20x the cost of data on iPhone.
Moreover, there are even better plans available to other smartphone users, including a Flex Rate plan which charges between $50 and $100 per month for data usage ranging from 500M to 5G. No such plan is available to the BlackBerry Enterprise User.
BlackBerry’s forte has always been email. Current generation products continue that heritage, but the competition is not far behind. Whether it’s a Windows mobile product from HTC, Apple’s iPhone, or Nokia’s super new E71 handset, they’re all now capable email devices. Rogers’ move puts a ton of pressure on the already embattled RIM, as it introduces one more reason not to choose BlackBerry with biased pricing that unfairly tips the playing field to RIM competitors.
No doubt RIM’s Jim Balsillie must already be on the telephone asking Ted Rogers why Rogers Communications is giving one of their best handset partners of the last decade such a public shafting.

| 4 Comments »
Tags: Canada, Tech and Business, gigaomdaily