A Mistake, Responding Well, and a Public Call for a Public Apology

I am about to break one of the rules of this blog. Years ago, I decided to never write about things that happen with clients until at least one year had passed. I am going to make an exception and write about something that happened very recently in the first part of this post. I confess I have been writing the second and third parts of this post for a couple years. I have waited to publish those parts for much the same reason I wait to write about client interaction: There is no substitute for the perspective given by time.

Part 1: A Mistake

“Nothing Has Changed”

It was the last possible day to load the Operational Data Store (ODS) with data that would be used for 2013 year-end reporting. Automation was weeks away. The team had been working for weeks on a manual checklist for the load. Since some of the team had planned vacation time approaching, others were cross-trained to execute their portions of the checklist. It was a clean hand-off of work between professionals.

The end-of-year load was scheduled for a Saturday. Early in the week before the Production load, I backed up the Production ODS, restored it to the Development server, and practiced. My part of the checklist contained over 100 manual steps. I ran through them three times, with analysis following each load. Tweaks were made each time. By Friday afternoon all were confident the load would succeed without a hitch.

I began walking through my checklist, executing the load in Production, at 4:00 AM Saturday morning. The Production database was largely static, since the loads were occurring manually and infrequently. I had a backup that was a few days old. I distinctly remember thinking, “Nothing has changed; if I need it, I’ll just use that backup.” My engineering-spidey-sense tingled a little but I ignored it. “Nothing has changed,” I told myself again. Sitting here now, typing this, I know what you know as you sit there reading this: That was an assumption. A rather stupid assumption, at that.

I have excuses.

1. No one told me additional data had been loaded into the Production ODS. As a data professional, I am the data’s keeper. People should not have to tell me about changes. I should operate as if no one tells me anything. I should execute defensively. What am I defending? The integrity of the data. As excuses go, this is weak.
2. Our team made exceptions to our own rules. On larger consulting gigs, we work in pairs. On smaller gigs, we work alone but share our thoughts and plans with other team members. There’s no substitute for a second set of eyes. Why did we deviate? Vacations and holidays conspired with an immovable deadline; the end of the year could not be postponed. This excuse falls into the combination-of-tolerances / comedy-of-errors category. Somewhat avoidable with better planning, but mostly inevitable once the second end of the candle was lit.
3. I was exhausted. As a data professional and as a consultant, I owe it to my clients to work only when I am capable of performing at my very best. It must be “A Game” or “no game.” That’s one of the reasons the aforementioned vacations were not cancelled – even though an important deadline loomed. It’s the reason we strive to travel during the week and not on weekends. And that’s the way it is most of the time. This was that one exception brought about by the perfect storm of the amount and nature of the work remaining and a deadline that could not be moved. This excuse has some validity.

Excuses stink. The short version of a long story is: The Production ODS load didn’t work like the three loads I practiced in Development. Hindsight is 20/20 and looking back, I realize the reason my practice loads worked differently from the Production load was data had been added to the Production ODS since my earlier backup. This prompted the execution of “Plan B.” When I executed Plan B and restored from the earlier backup, I effectively deleted the data I didn’t know had been added and then re-executed my 100+ steps. Not surprisingly, this load went much better.

Something had changed.

“What Do I Do Now?”

I was made aware of the missing data during a meeting. The VP who owned the system that was expecting the data I wiped out was in attendance. It wasn’t an ambush – merely a coincidence. The team is just that small. I put the pieces together quickly in my mind and thought, “What do I do now?” I wanted to remedy the mistake, but I didn’t have a time machine that would allow me to go back and smack myself in the head at 4:00 AM Saturday. So I spoke up, “I know exactly how that happened. It was me. I restored from an older backup. Durnit! That was stupid of me. I apologize.” No one was happy I had deleted data from the Production system. But they acknowledged my apology and we moved forward with the meeting.

After the meeting, still motivated to remedy my mistake, I contacted the developer who had loaded the now-missing data. I apologized to her and offered to help reload the data. As is often the case in our line of work, the hard part was figuring out which data to load, how to find and validate and cleanse it. The actual data-loading part is minor by comparison. Plus, she had saved her work. Reloading would take a few minutes. She appreciated my apology and offer, though.

Happy Ending

By sheer serendipity, my error worked in our favor. The ODS is a source for downstream systems, and it proved a lot easier to load one of those systems without the additional data present. With less effort we loaded that system, reloaded the data I deleted, and moved forward. All’s well that ends well, right?

No.

All’s well that we manage with integrity and intelligence.

Part 2: Responding Well

Several apology-generating incidents (AGIs) occurred around the same time. You can read about a couple of them here:

• SQLServerCentral is issuing a public apology to Itzik Ben-Gan
• This is how to fail

Responding to Plagiarism

In the first instance, a content author plagiarized Itzik Ben-Gan’s content from his book, Microsoft SQL Server 2012 High-Performance T-SQL Using Window Functions. Once the plagiarism was identified, SQL Server Central issued a public apology and held a contest which gave away 10 copies of the book – generating 10 sales for the book.

Why did SQL Server Central apologize? I think that’s a great question. SQL Server Central apologized because they are the organization – the website backed by a corporation – that published the plagiarized material. Did SSC intentionally publish the stolen ideas? Goodness no. It was a mistake.

Everyone makes mistakes.

Just like my mistake with the backup, they didn’t realize it was a mistake until after the damage was done. And, just like my mistake, they took immediate action to rectify the matter. First, they acknowledged their error and apologized. Second, they did what they could to correct the negative impact by purchasing copies of Mr. Ben-Gan’s book for the contest. Both responses were appropriate. Kudos to SQL Server Central for stepping up.

Buffer Overflow

Tim Mitchell wrote an SSC guest editorial about Buffer’s response to hacking. In October, Buffer was hacked. Spammy messages, appearing as though sent by Buffer users, began appearing on social media profiles of Buffer users. As Tim notes: To Buffer’s credit, they opted for transparency before they really knew what was happening.

In and of itself, this is simply awesome. But that’s not all. Joel Gascoigne, Buffer’s CEO, wrote the post himself, and he opened with this line:

I wanted to post a quick update and apologize for the awful experience we’ve caused many of you on your weekend.

The ninth word in Joel’s initial update on the topic is “apologize.”

Everyone makes mistakes.

I make mistakes like the incident described in Part 1. SQL Server Central didn’t catch the plagiarism until after the article was published. Buffer detected a breach soon after it occurred. In each case an apology was issued early.

Apologies are most effective when they are delivered immediately. But given the choice between late and never, late works. Which brings me to …

Part 3: A Public Call for a Public Apology

For the reasons I outlined in this post, I believe the Professional Association for SQL Server (PASS) owes Steve Jones a public apology.

Not everyone agrees with my original thoughts and I do not expect everyone to agree with my public call for a public apology. Before scrolling to the comment text area, please read and consider these next thoughts.

Do Apologies Convey Weakness?

I know some believe apologies convey weakness and I disagree. In fact, the opposite is true. Everyone makes mistakes. By definition, mistakes are not intentional. Admitting a mistake and apologizing demonstrates an awareness of this reality. Conversely, not apologizing could be interpreted to mean, “We never make mistakes,” and that is not a healthy or helpful way to think.

Do Apologies Admit Malice?

Goodness no. A wise person recently wrote we should always assume good intentions. I concur. I didn’t intentionally wipe data from that database. SQL Server Central didn’t intentionally publish plagiarized material. And Buffer didn’t leave a back door open for hackers.

Everyone makes mistakes. Stuff happens. Even when we mean well. Apologizing is a way to admit, “We know you didn’t like the way this happened and we’re sorry you were hurt.”

Why Are You Asking for a Public Apology Publicly?

I’ve made this request to members of PASS leadership privately. Repeatedly. I’ve pleaded, in fact. Why? I believe this is a blemish on PASS’s relationship with the SQL Server Community. PASS leadership alternately positions itself as community leaders and as the community. I will not address the merits of either argument in this post, but I will point out that the 2010 elections episode is pivotal in the history of the relationship between PASS and the SQL Server Community.

Why Are You Doing This Now?

“It has been 3.5 years. Why now, Andy?” That’s an excellent question.

Zerost, I love the SQL Server Community. And I love PASS. I felt – and still feel – the 2010 PASS elections were a dark hour for both. Truthfully, most SQL Server Community members are not aware of details. Most of those who are aware of the details stopped worrying about them long ago – some because they felt it futile to “fight city hall;” some because they didn’t see an issue with the decision itself, only with the response / reaction to the decision; some for other reasons; and some just because. I respect everyone’s decision. I firmly believe people do the things they do and believe the things they believe for reasons that make sense to them.

I will never win a PASSion Award, but it won’t be for lack of passion for both the PASS organization and the SQL Server Community. I care deeply about both. I perceive a divide between PASS leadership and the Community and I would like for that wall to be removed. I believe this dark hour is the strongest part of that wall. I know PASS and the Community are stronger and better together.

First, I have changed. My initial reaction to the decision to not allow Steve Jones on the 2010 PASS Board of Directors ballot was not positive. The decision by PASS leadership hurt my friend and I was ticked. At the top of this post, I mention my policy of not blogging about incidents until at least one year has passed because time brings perspective. I needed time to move from a place of not-wanting-to-help to a place of wanting-to-help. It took a while. A long while. I am there now. And I want to help.

Second, PASS leadership has changed. The makeup of the Board of Directors, Executive Committee – even Corporate Founders – has changed. Kevin Kline points out many ramifications of CA leaving the PASS Board of Directors in his excellent post, What’s the Hidden Issue in the Recent PASS Bylaw Changes? But it goes deeper than that: Thomas LaRock is now President of PASS. I know Tom and, like most people who know him, hold him in high regard. I respect Tom. I like Tom. I expect him to lead PASS well, and differently. I am not so naïve as to think Tom can single-handedly solve every challenge PASS faces. But I am confident he can continue to nudge the organization in the best direction.

I believe PASS has been moving in a better direction since 2010, but I do not believe PASS leadership has crossed the middle point between where it was and where it needs to be.

That last statement is completely subjective. At this time and in this post, I choose not to discuss the reasons I feel this way. As I stated earlier, people believe things for a reason. I have many reasons for believing what I believe about this. For now, I will engage with anyone privately to discuss what I believe and why – there’s an Email link in the “This Blog” section of this page – up and to the right.

In my opinion, the real shame would be for us to look back 3.5 years hence with this issue still unaddressed.

Conclusion

I believe PASS, the organization, should apologize to Steve. Publicly. I think the way PASS treated Steve was a mistake. I think everyone realizes that. I think an apology is the appropriate response and I believe an apology can be made with integrity.