New System Cut-Over—Epic Failure

Many of you are aware that we have been down for over two days while attempting to cut-over to a new membership (subscriber) system.

A Bad Start

The whole process started badly when the developers forgot a critical issue (that I had documented in four different places) and wasted four hours.

However, things seemed to be getting better and, as at quitting time last night (Wednesday), I was still optimistic we would be good to go this afternoon (Thursday), as the developers assured me.

Reality Check

Still, I also knew that developers are notorious—kind of like boatyards—for overestimating progress. So this morning I got up early and took a personal look at what they had achieved and quickly realized that they were nowhere near as far along as they claimed; in fact, they were less than half done. And, worse still, what they had done was full of data inconsistencies and bugs.

Not a situation that we were willing to subject you, our valued members, to.

So, long story short, we pulled the plug and have reactivated the old system.

I then had a conference with the CEO and lead developer to lay out what we expected on the next attempt:

  • Testing code before deploying it.
  • Pre-testing data transfers.
  • Reading and working off the specification.

This should have been an easy set of goals to meet since I had documented everything they needed to know—back in another life I managed big computer accounting system cut-overs—but they felt that Phyllis and my expectations were too high, so they quit—they did promise to return all our money, but we have not seen it yet.

The result is that we are back to square one and out a huge amount of time—the biggest loss here, since we could have used that time to create new content.

Just so you know, we did a huge amount of due diligence on these people. But then, I have done the same on boatyards and still got terrible results. It’s no guarantee. In fact, this whole experience feels like the worst possible boat project you can imagine…and we all know how bad that is!

The Future

As to the future of this project, right now Phyllis and I just need a rest—I have at least 200 hours in this. And there are articles to write and it’s nearly time to go sailing, so nothing much will change before the fall.

Sorry

Our apologies to all of you who were inconvenienced by this. This was the longest time that AAC has been unavailable, ever…by a factor of 10.

This Will Not Hurt You

Let’s close on some good news. The old system, despite being a bit antiquated, is, and always has been, amazingly robust. And that, like with boats, is the most important thing. The even better news is that we had not in any way changed the old system, so it has not been compromised by this.

That said, it will take some 24 hours for all the account changes that should have processed on our end while we were down to catch up. (Normally it would not take that long, but the part of PayPal we need to access to make this happen is down—it never rains but it…).

Like what you just read? Get lots more:


Please Share

Meet the Author

John

John was born and brought up in Bermuda and started sailing as a child, racing locally and offshore before turning to cruising. He has sailed over 100,000 miles, most of it on his McCurdy & Rhodes 56, Morgan's Cloud, including eight ocean races to Bermuda, culminating in winning his class twice in the Newport Bermuda Race. He has skippered a series of voyages in the North Atlantic, the majority of which have been to the high latitudes. John has been helping others go voyaging by sharing his experience for twenty years, first in yachting magazines and, for the last 12 years, as co-editor/publisher of AAC.

30 comments… add one
  • Robert Muir Mar 21, 2019, 5:44 pm

    Wow, I’m sending boatloads of sympathies to you John. Amazing the level of “professional” expertise there is out there.

  • Oliver Schonrock Mar 21, 2019, 5:45 pm

    Hi John

    Sorry to hear it didn’t work out.

    I have worked in this industry for the last 20 years, and it always reflects badly on all of us when some people bite off more than they can chew and make us all look bad.

    Oliver

  • Ernest E Vogelsinger Mar 21, 2019, 6:07 pm

    John, it seems you made a good move on keeping the old system alive for a complete rollback – but then you’re a dyed-in-the-wood IT expert from your “former life”. And as far as I’m concerned, I know these situations, and I am constantly battling the very same issues you were facing here with companies you’d expect to be reliable (I won’t tell names, but it’s only 3 letters…).

    Let it rest, and see how you’ll tackle it this fall. And maybe it’s just the same as it is with boats – these shiny new showboats mostly do not deliver what they promise.

    In closing – if I can be of any help at the next approach just holler, you have my contacts.

    Best wishes and a safe and merry sailing season,
    Ernest

  • Richard Elder Mar 21, 2019, 7:07 pm

    Hi John;
    My condolences!

    In my current job I’m always amazed when I encounter somebody who actually does anything as promised. “Trust but verify!”

    If anyone has been around large ski area construction projects in challenging mountain climates they’ve undoubtedly heard several variations of the same story. It begins with “how many California Architects does it take to understand that water runs downhill?”

  • Flemming Torp Petersen Mar 21, 2019, 9:19 pm

    I feel sorry for you guys … It must be so hard … frustrating …
    But to me, the most important thing is that you will continue your work.
    And that I still have easy access to all the invaluable information you produce.
    All the best – Flemming Torp

  • David Logan Mar 22, 2019, 4:08 am

    Always good to roll back when not 100% certain. Good work, take a break, come back refreshed. Go for a sail! ?

  • Marek Nowicki Mar 22, 2019, 7:00 am

    My sympathy….on the larger note, I think we are getting into economical phase that a large correction is just around the corner. Emergence of businesses run by inept folks (lack of experience but full of aspirations) but somehow able to finance it (=easy money) is a sure sign. And I am not talking about US only. I don’t want to bother you with details but I am in Mexico with my boat and see a similar situation here.

    • John Mar 22, 2019, 7:39 am

      Hi Marek,

      I agree completely, it’s a symptom (and a flag) at the end of every up cycle.

  • John Mar 22, 2019, 7:41 am

    Hi All,

    Thanks for all the warm thoughts on this. Means a lot to both of us.

    On a positive note, we learned a lot and are already thinking about how to use that new knowledge to make the site better going forward.

  • MICHAEL A SMITH Mar 22, 2019, 8:50 am

    I’m no IT guy — I still use a camera with film — but the old system seems to work fine. Why fiddle with it? There’s probably an issue I don’t understand, but from this user’s viewpoint, it’s fine. And it’s a great site, too.

  • Colin Speedie Mar 22, 2019, 11:24 am

    Hi John

    commiserations – this sort of thing shouldn’t happen, but so often does. As you say, it’s the time wasted that’s the most painful part, followed by the money. Good that the old platform saved the day, and – better luck next time!
    Best wishes
    Colin

  • Paul Rutherford Mar 22, 2019, 11:39 am

    When Plan B doesn’t seem to be working rolling back to Plan A is always a good idea….
    Hope you can both recover and make sense of the issues, get some sailing in and revisit another day.

    We all appreciate all your hard work on the site which benefits all of us subscribers and guests alike.

    Paul

  • Doug Hennessy Mar 22, 2019, 11:43 am

    John,
    I empathize, having been the HR client managing payroll and benefit technology cutover projects with outside vendors. When the whole thing starts feeling like an chapter from Alice in Wonderland it is time to revisit the basic discipline:
    Testing code before deploying it.
    Pre-testing data transfers.
    Reading and working off the specification.
    Good luck…..this too will pass.
    – Doug

  • Marc Dacey Mar 22, 2019, 12:45 pm

    My sympathies to you and it probably irks you given your resume that this unfolded (or failed to unfold) as you anticipated, even allowing for the usual factors of confusion and delay.

    I feel vindicated, however, in my decision to bring along a sextant and a Walker log!

  • VERNON ADEN Mar 22, 2019, 12:55 pm

    When your going through HELL, just keep going is the best advise I can give you two!

    Next time run parallel systems for 30-45 days before you switch over to make sure all the bugs are worked out.

    Vern in Portland Oregon

    • John Mar 22, 2019, 3:00 pm

      Hi Vernon,

      Parallel systems only work when the inputs and outputs are the same. Works well for accounting systems. Not the case here. That said, it’s a good suggestion that we might me able to partially implement next time.

  • Paul Turje Mar 22, 2019, 12:58 pm

    No worries John. The old site works very well. There is an adage that ” the refined version of the old technology almost always outperforms the unrefined version of the new technology”. Probably first said by a geezer…like me, to justify their existence.

  • JOHN SHEPARD Mar 22, 2019, 1:22 pm

    John, I was surprised on Wednesday when I met the “We’re not home” message. Then sometime yesterday the clock started to function…
    No worries, just disappointment. Been waiting 2 years for the US Government to complete an accounting/purchasing system changeover for one of the agencies… So, I retired instead. Still not fully functioning. It is an industry problem. If you find someone who can grasp both the Macro and the Micro scope of such projects grab them, hire them and put them in house.
    It will all work out in the long run. And you guys have a long run to go…

  • John Kane Mar 23, 2019, 1:19 am

    I have been through the exact same scenario, and I know exactly what you’re going through. You have an amazing website and I enjoy your content immensely. The best thing to do is take a deep breath, step away for awhile and get back to your basics of providing your great and always informative content. (Don’t forget to go Sailing)
    The problem will be there when you’re recharged and ready to take it on again. Hang in there.

  • Ben Garvey Mar 23, 2019, 9:39 am

    John – was this done by a local company here in NS?

    …I ask because we are looking at possibly similar requirements with some clients, and I’d like to know who to avoid!

    condolences to you and phyllis – it’s hard enough to keep this stuff going without that sort of incompetence!

    I don’t think it’ll lose you any followers though. I believe we’re a tenacious bunch by nature…

    bg

    • John Mar 23, 2019, 10:47 am

      Hi Ben,

      No, no one local. But if you do find someone good around here, I would love to get a recommendation.

      And thanks for the comforting words.

      By the way, I don’t think the company in question are fundamentally a bad outfit, rather they screwed up on this one, but I think they will learn and grow from that.

  • Kit Laughlin Mar 24, 2019, 9:40 am

    Overpromising and underdelivering? The new normal, John and Phyllis. And as I always read/view content on a laptop, any kind of App is not a benefit to me. And to echo Ben G: we are tenacious. You are going to need to do a LOT more than that to lose me! Best, KL

    • John Mar 24, 2019, 5:17 pm

      Hi Kit,

      Thanks for the vote of confidence, much appreciated. That said, I think we will need to do something to better accommodate our members who read on phones and tablets since they are now up over 60%. But we also need to be really careful not, in so doing, to screw up the site for readers using larger screens. The good news is that there is a lot of good work going on to solve those issues, so we can leverage that work.

      • Kit Laughlin Apr 1, 2019, 5:40 pm

        Please let us know if the original company did, in fact, return your money. That would be one positive, at least.

        • John Apr 2, 2019, 8:36 am

          Hi Kit,

          Thanks very much for your concern. They were happy to return the money, but in the end we got together and figured out how to save the project, or at least I think so. Still working out the details, but as soon as we have it finally sorted out one way or another I will share the outcome in a post. The bottom line on this is that their base subscription package is definitely the best option for our members, going forward. Where it went wrong was the conversion and custom changes.

        • Kit Laughlin Apr 2, 2019, 6:51 pm

          As a two-person organisation that sells our video products into all jurisdictions ourselves, we sympathise deeply. Many companies still ignore the local sales taxes (of which there are a great many as all countries struggles to get money from internet sales), and display currencies and conversion costs are another massive headache. All this is MUCH more difficult to get right than the ordinary buyer or subscriber realises. Good luck, John and Phyllis.

          • John Apr 3, 2019, 8:01 am

            Hi Kit,

            Ain’t that the truth!

  • alistair miller Mar 24, 2019, 1:25 pm

    You’re ability to enthusiastically share failures as well successes is one of the features that we readers love . I expect most subscribers are fairly resilient (otherwise they wouldn’t keep going with sailing as a hobby).
    So keep going – there’s lots of goodwill respect and patience for you out here…
    PS Love the site !!

    • John Mar 25, 2019, 7:19 am

      Hi Alistair,

      Thanks for the kind words. Good point about the persistence that sailing requires!

Only logged in members may comment: