Thanks to Jeff Mlakar for hosting this month’s T-SQL Tuesday, a monthly collection of blog posts by data bloggers who work and live in the Microsoft SQL Server space.

This month’s topic is about projects that have gone wrong.

“Why Do Good Projects Go Bad, Andy?”

Deltas.

That’s all. Deltas. Allow me to explain.

In life, engineering, and projects, we experience deltas in a variety of forms.

IRL…

As a manager I learned there’s a difference between how we are perceived and how we think we are perceived. I label this the “self-awareness delta.”

In Engineering…

In engineering terms, delta is the letter of the Greek alphabet used to represent a difference or change or gap.

In Projects…

Projects that will eventually fail go astray early on in ways that are either invisible or ignored. Projects disappoint or fail. I’ve seen (and believe) statistics that 85% of business intelligence projects fail or disappoint. Disappointment or failure begins as some delta – some difference – between what is real and what is perceived as real.

Detecting Deltas

Think about how we discover there’s a delta in our self-awareness… it requires, well, self-awareness! It turns out that detecting deltas requires meta thinking. The risk of being too meta? We can succumb to circular logic. Thinking meta involves some circular – or, at least, circumspect – reasoning.

Our perspective is also limited by our perspective. A wise man once said it’s difficult to remove the speck from our brother’s eye while we have a plank in our own eye. (Note: Andy’s Plank Removal Service Offering currently has but a single customer…)

So how can we achieve delta detection?

I regularly reference Kathryn Schulz’s TED talk on Being Wrong. Kathryn begins her (excellent) TED talk by stating that we do not know what it feels like be wrong. But she concludes her introduction by contradicting herself (which is totally appropriate… think about it…) and states that we do know what it feels like to be wrong; it feels like we’re right.

Ownership

I’ve seen projects fail. Heck, I’ve seen companies fail. I’ve made mistakes and when I’ve caught them, things have sometimes improved. Sometimes I catch them too late to effect a complete or partial remedy. It’s rare, but it happens.

The first step on the path to (project) health is to admit we have a problem:

“We have a problem.”

Another step is communication. I prefer over-communication.

Over-Communication

I’ve never once had a client say to me, “Andy, you are communicating too much about this project.”

Not. Once.

I’ve had projects go sideways because I failed to communicate effectively or enough. Here’s one thing I’ve learned about communication in projects:

If I communicate enough, I will communicate effectively.

Failure, By Degrees

Projects that fail first go sideways. Were we to graph the trajectory of a project failing, we would not see a 90° turn at the point where the project goes sideways. Instead we would see a gradual departure – a curving away – from the path to success. The departure would reach 90° and then – if not corrected – continue to 180°. At this point the project is irrecoverable. A project may become irrecoverable at 90°, even. It depends on the project.

Fix It!

I can hear you thinking, “How does one fix it, Andy?” I’m glad you asked.

If you catch the departures at 1° – or even 9° – it takes much less effort and time to address and correct the departure.

This sounds simple and it is, but it’s also difficult. Remember that meta thinking is required, and meta thinking is hard. Simple, yes; difficult to implement and accomplish.

DfG

I address “Deviation from Good” (DfG) by over-communication. It’s the drum I beat throughout any project I lead. How do I implement DfG?

TPS Status Reports

I loathe bureaucracy. Especially bureaucracy for bureaucracy’s sake. I learned the reason I loathe status reports is I am not naturally good at tracking status. I struggle to remember details of what I did all week at the end of a week. Being an engineer, I engineered a solution:

A few years ago, my friend (and hero) Donald Farmer (Treehive Strategy | @DonaldTreeHive | LinkedIn) shared this Notepad hack with me. If you start a Notepad file with the text “.LOG” followed by a carriage-return / line feed (hit the Enter key), Notepad can provide a log template. Save the file with “.LOG” on the first line. Each time you open the file in Notepad a new line will be added containing the current date and time. Try it. It’s cool.

I use the following format to log project time:

I use this to track my time. I’ve even built an SSIS package that reads this file and stores the data in a database. I’m able to generate status reports from this data very efficiently.

My former boss and mentor, Bennett McEwan (LinkedIn), challenged me when I reported to him at Unisys 10+ years ago. When I complained that my status report was taking an hour or more each week Ben said, “Your status report should only take you five minutes.” My kneejerk reaction was not positive. But then I engaged to solve the problem, remembered Donald’s Notepad hack, and the rest is history.

Come To Jesus Meetings

I’ve been wanting to use “Come to Jesus Meetings” in a heading for a long time. It’s not a perfect fit here but I must confess I’ve become tired of waiting. So I shoe-horned it in right here.

I visit (if possible) or call the sponsor for each project on which Enterprise Data & Analytics is actively engaged and ask questions similar to:

• What questions do you have for me?
• How can I better serve you?

These conversations identify expectation deltas at 1° – or 9° – deviation… a place where almost any issue can be efficiently addressed within the confines of the current work effort.

These conversations keep a project on track.

Are these conversations pleasant?
Not always.
But they are extremely valuable.

Failure is normal.
Failing fast is almost always a good thing.

I’ve learned over-communicating is vital to mitigating project disappointment or failures.

In my experience, people either love or hate triggers in Transact-SQL. As a consultant, I get to see some interesting solutions involving triggers. In one case, I saw several thousand lines of T-SQL in a single trigger. My first thought was, “I bet there’s a different way to do that.” I was right. The people who’d written that trigger was unaware there was another way to solve the problem.

I see that a lot.

Software and database developers do their level best to deliver the assigned task using the tools available plus their experience with the tools. I don’t see malice in bad code; I see an opportunity to mature.

Granted, when I’m called in as a consultant I get paid by the hour. That’s not the case for many folks.

One ETL-Related Use For Triggers

Change detection is a sub-science of data engineering / data integration / ETL (Extract, Transform, and Load). One way to detect changes in a source is to track the last-updated datetime for the rows loaded by an ETL load process.

The next time the load process executes, your code grabs that value from the previous execution and executes a query that returns all rows inserted into or updated in the source since that time.

Simple, right? Not so fast.

Where to Manage Last-Updated

How the value of the last-updated datetime is managed is crucial.  I’ve had arguments discussions with many data professionals about the best way to manage these values. My answer? Triggers.

“Why Triggers, Andy?”

I’m glad you asked. That is an excellent question!

Imagine the following scenario: Something unforeseen occurs with the data in the database. In our example, assume it’s something small – maybe impacting a single row of data. The most cost-effective manner to manage the update?

Manual update.

How is the last-updated value managed? It could be managed in a stored procedure. It could be managed in dynamic T-SQL hard-coded inside a class library for the application. If so – and given a data professional is now manually writing an UPDATE T-SQL statement to correct the issue, there is an opportunity to forget to update the last-updated column.

But if the last-updated column is managed by a trigger, it no longer matters if the data professional performing the update remembers or forgets, the trigger will fire and set the last-updated column to the desired value.

Change detection works and all is right with the world.

Troubleshooting Triggers

Troubleshooting triggers is… tough. There are some neat ways to troubleshoot built right into tools like SQL Server Management Studio (which is free). For example, in SSMS you can Debug T-SQL:

Conclusion

I like triggers for the things triggers do best, but triggers can definitely be misused. Choose wisely.

:{>

My friend and #SqlFamily sister Mala hosts this month’s T-SQL Tuesday topic: Setting learning goals for 2018. It’s a timely topic for me as I’ve recently engaged in more formalized training. Mala’s questions:

1. What do you want to learn? (specific skills and talents)
2. How and when do you want to learn? (methods of learning and timeline on learning)
3. How do you plan to improve on what you learned? (Putting it to use at work/blogging/speaking)

What Do I Want to Learn?

Earlier this year (2017) I picked up a certification in Processing Big Data in Hadoop with Azure HDInsight. I have experience building SSIS packages that load Azure and Amazon’s cloud products, including Aurora. I want to expand that and gain experience using Google’s Cloud offering.

I’m focusing on cloud technology, specifically data integration in the cloud. Scott Currie (Varigence | @scottcurrie) demonstrated Biml and Azure Data Factory v2 at the PASS Summit 2017 in the Day 1 keynote.

How and When Do I Want to Learn?

I like online training these days – a lot. I can wear my sweat pants while learning! I deliver and consume live and recorded online training. Christy and I home-school our three children and use online curricula. I used to believe online learning was the future of education. I now believe online education is the present.

Operating a business is almost a full time job all by itself. Operating more than one business is… challenging, especially when you want to maintain any kind of schedule. Separating the things that appear urgent from the truly urgent is more art than science. It’s no longer merely a matter of discipline, I have to actively make time to remain engaged in learning. I do this by setting my alarm for 5:00 AM currently. I may have to set it for earlier to increase my learning time.

How Do I Plan to Improve on What I’ve Learned?

I will put what I learn to use. I’ll build demos for presentations, and then deliver those presentations when given the honor and opportunity to do so. I will blog about what I am learning and what I have learned, especially when I encounter an error and then overcome that error through sheer stubborn will persistence.

Conclusion

If you want to get into technology – or want to remain current in technology – you simply must engage in lifelong learning. It’s not optional; it never has been. Now, more than ever, it’s important to continue your education.

:{>