Lift and Shift is Not a Strategy…

Lift and shift is a bridge.

If your enterprise is considering migrating enterprise data and data integration workloads from on-premises to the cloud, lifting and shifting is not the end of the story. Lifting and shifting is, rather, the mechanism – the conversion project – that positions your enterprise to leverage the economies of scale afforded by cloud technology.

Andy’s Lift and Shift FAQ

“Should We Lift and Shift?”

I hear this question often and my response is, “When it makes sense to do so, yes, you should lift and shift.” This begs the next question, which is…

“How Do We Know It’s Time to Lift and Shift?”

My engineer-y response to this question is, “You will know it’s time to lift and shift to the cloud when you want to leverage functionality available in the cloud that is not (or not yet) available in on-premises versions of the enterprise platform(s) in use in your enterprise.”

“What Are Some Advantages of Migrating to the Cloud?”

The biggest advantage of lifting and shifting enterprise data to the cloud is the ability to efficiently scale operations. By efficiently, I mean quickly and easily – especially when compared to the time and expense (don’t forget opportunity cost when calculating expense) to scale up systems on-premises.

The ability to scale up and scale down on demand is a huge advantage for some business models which experience “spike-y” demand for operations at different times of the year, quarter, or month. But even if that’s not the case, all data scales. It’s very handy to be able to connect to the Azure Portal and move a slider (as opposed to purchasing and provisioning more hardware…).

There’s a brand new (in my opinion) “knob” exposed by cloud-enabled efficient scaling. As I wrote in my post titled Time and Money in the Cloud:

Let’s say you pay $100 to incrementally load your data warehouse and the load takes 24 hours to execute at the scale you’ve selected in the cloud. Prior to thinking in DTUs, engineers and business people would think, “That’s just the way it is. If I want more or faster, I need to pay for more or faster.” But DTU math doesn’t quite work that way. Depending on your workload and DTU pricing at the time (FULL DISCLOSURE: DTU PRICING CHANGES REGULARLY!), you may be able to spend that same $100 on more compute capabilities and reduce the amount of time required to load the same data into the same data warehouse to minutes instead of hours…

The fact that the cost/performance curve can be altered in seconds instead of months meta-changes everything.

“Are There Disadvantages of Migrating to the Cloud?”

It depends. (You knew that was coming, didn’t you?)

Enterprise Data & Analytics helps enterprises migrate data, data integration, lifecycle management, and DevOps to the cloud. In some cases (~30%), the enterprises spend a little more money in the near-term. There are two reasons for this:

  1. When upgrading, it’s always a good idea to operate new systems in tandem with existing systems. In a lift and shift scenario, this means additional expenses for cloud operations while maintaining the expense of on-premises operations. As cloud operations are validated, on-premises operations are shut off; thereby reducing operating expenses. In truth, though, this dynamic (and expense) exists whether one is lifting and shifting to the cloud or simply upgrading system on-premises.
  2. “Standing on the bridge” (more in a bit) can cost more than remaining either on-premises or lifting and shifting the entire enterprise workload to the cloud.
  3. Regulatory requirements – including privacy and regulations about which data is allowed to leave nation-states – will constrain many industries, especially government agencies and NGOs (non-governmental organizations) who interact heavily with government agencies.

Standing On The Bridge

One option we at Enterprise Data & Analytics consider when assisting enterprises in lift and shift engagements is something we call “standing on the bridge.” 

Standing on the bridge is present in each lift and shift project. It’s one strategy for implementing hybrid data management, which almost every enterprise in the cloud today has implemented. Hybrid means part of the enterprise data remains on-premises and part of the enterprise data is lifted and shifted to the cloud. 

Hybrid is implemented for a variety of reasons which include:

  • Mitigating regulatory concerns; and
  • As part of the normal progression of lifting and shifting enterprise data and data workloads to the cloud.

Standing on the bridge for too long is a situation to avoid. 

“How Do We Avoid Standing on the Bridge For Too Long?”

Planning. Planning is how an enterprise avoids standing on the bridge too long. Your enterprise wants advice from experienced professionals to shepherd the lift and shift operation. 

Enterprise Data & Analytics can help.

Helpful Tools

Enterprise Data & Analytics has been delivering, and writing and speaking about Data Integration Lifecycle Management for years. 

We’ve built helpful tools and utilities that are available at the DILM Suite. Most of the DILM Suite tools are free and some are even open source. 

Honored to Present Lift and Shift SSIS to ADF at #Azure DataFest Reston

I am honored to deliver Lift and Shift SSIS to ADF at the Azure DataFest in Reston Virginia 11 Oct 2018!

Abstract

Your enterprise wants to use the latest cool Azure Data Analytics tools but there’s one issue: All your data are belong to the servers on-premises. How do you get your enterprise data into the cloud?

In this session, SSIS author and trainer Andy Leonard discusses and demonstrates migrating SSIS to Azure Data Factory Integration Runtime.

Register today!

:{>

Time and Money in the Cloud

What Could Your Enterprise Do With Extra Time?

Benjamin Franklin is credited with saying, “Time is money.”

The 2011 movie, In Time, depicts an entire economy based on time. (A friend pointed out that time is not fungible, but I digress…)

So, what could your enterprise do with more time?

When you think about it, this question lies at the heart of many data- and software-related enterprise activities, such as:

  • Disaster Recovery
  • High Availability
  • DevOps
  • Software development methodologies
  • Project management methodologies
  • Lifecycle management

I believe this is an important question – perhaps the important question of the cloud-era.
I believe time is replacing money in a way we’ve never before experienced.
I believe the cloud is driving this new economy.

Technology Economy Conversations

Not long ago I wrote of my experience when I left an instance of Azure Data Factory SSIS Integration Runtime running overnight and it cost me about $30USD. The post was titled The Cloud Costs Money and reflects some of the thinking of this post.

Not long after that, I was honored to chat with Stuart Ainsworth (@codegumbo) at Atlanta Azure DataFest:

Chatting with Stu

In this Data Driven DataPoint, captured while attending the inaugural Atlanta Azure Data Fest, I was honored to speak with Rie Irish, Julie Smith, Tim Radney, Geoff Hiten, and Stuart Ainsworth.

The event itself was an astounding event on two levels:

  1. The velocity of technological innovation is increasing (“well duh, Captain Obvious”) so, if you haven’t attended such an event recently – and by “recently” I mean the past eighteen months – you should attend to see how folks are combining cloud, Internet-of-Things (IoT), analytics, machine learning, artificial intelligence, on-premises, and hybrid technologies to deliver – frankly – amazing solutions.
  2. Community. Networking with people will change your career. It will change your career in a way that will change your life. Ask anyone who is engaged in a Microsoft data community. My synopsis of Atlanta Azure DataFest is here and my theme is “it is not too late to jump in”:

The next Azure DataFest (@AzureDataFest) is in Reston Virginia 11-12 Oct 2018!


Learn more and register here!

On Time and Money…

Stu and I spoke about the dynamic of time and money and how both relate to DTUs – the unit of measure for Azure data-related computing. So what’s a DTU?

According to Andy Mallon (@AMtwo) – paraphrasing Microsoft’s documentation in this post titled What the heck is a DTU? :

A [Database Transaction Unit] is a blended measure of CPU, memory, and data I/O and transaction log I/O in a ratio determined by an OLTP benchmark workload designed to be typical of real-world OLTP workloads. Doubling the DTUs by increasing the performance level of a database equates to doubling the set of resource available to that database.

That’s a great definition. But what are the implications?

Stu and I discussed the following data integration scenario:  Your enterprise hardware is currently fixed – which fixes the capacity of your data-related workload. You can change your enterprise’s workload capacity at any time; you can increase capacity by buying more or better hardware.

Imagine your enterprise migrates your data and data-related workloads to the cloud. (I know a company that can help! :))  After migration, your enterprise can scale hardware up to meet demand, and then scale it back down again when demand drops. The economics of pay-for-only-what-you-need-when-you-need-it is compelling, to be sure, and it drives almost all decisions to migrate to the cloud.

But there’s more.

Time to market matters to many enterprises.
Time to market matters more than ever to some enterprises.
The impact of time to market is easy to underestimate.

Thinking in DTUs

Consider the math: A DTU is a DTU. How the DTU cycles are distributed across time and processors doesn’t really matter.

Let’s say you pay $100 to incrementally load your data warehouse and the load takes 24 hours to execute at the scale you’ve selected in the cloud. Prior to thinking in DTUs, engineers and business people would think, “That’s just the way it is. If I want more or faster, I need to pay for more or faster.” But DTU math doesn’t quite work that way. Depending on your workload and DTU pricing at the time (FULL DISCLOSURE: DTU PRICING CHANGES REGULARLY!), you may be able to spend that same $100 on more compute capabilities and reduce the amount of time required to load the same data into the same data warehouse to minutes instead of hours.

That’s DTU Math.

The shift to DTU thinking is subtle but vital.

We are used to thinking the only way to make things faster is to spend more money. That’s simply no longer accurate. The shape of the line between cost and performance is may still trend linear but you can dramatically – and very, very quickly – alter the slope of that line, especially with regards to time.

The fact that the cost/performance curve can be altered in seconds instead of months meta-changes everything.

The statements above are examples of DTU thinking and DTU math. So, please ask yourself: “What could my enterprise do with more time?”

Why is that so important?
Because Ben was right: Time is money.

SSIS Catalog Compare vNext Preview

I am happy to announce that the next version of SSIS Catalog Compare is officially in Preview!

What exactly does “Preview” mean? I’m glad you asked. It means I am sharing copies of SSIS Catalog Compare with users of the current version to get their feedback, find bugs, and help identify improvements.

Here’s a screenshot:

Looking at this screenshot (click to enlarge), you may notice some things:

  1. Font color indications – in addition to background color changes (which provide a shading for people who have difficulty seeing colors), the font color of “different” items is also different.
  2. The SSIS Catalog loaded into Catalog 2 is an Azure Data Factory Integration Runtime. SSIS Catalog Compare now speaks cloud.

I will be demonstrating using this preview of SSIS Catalog Compare in an upcoming (free) webinar titled Use SSIS Catalog Compare to Lift and Shift SSIS to ADF. The webinar is 19 Jul 2018 at noon EDT. I hope to see you there!

:{>

The Cloud Costs Money

It’s true: the cloud costs money.

Yesterday, when I logged into my Azure Portal to set up an Azure Data Factory Integration Runtime for my presentation, the balance for my Azure free subscription (which allows me $100USD per month) was sitting around $82USD. This morning, after leaving a demo instance of ADFIR running overnight, my balance has dropped:

I absolutely love the cloud! This is why I’ve been delivering webinars about the cloud and blogging and presenting about Azure Data Factory.

I would just like to clarify a few things:

The Cloud Is Not Free

This much is true. And while the cloud can save enterprises – especially smaller enterprises – lots of cash on “burst-y” workloads via scale-on-demand – the cloud can actually cost more money when unneeded services are accidentally left running.

The Cloud Will Not Fix Bad Code

In a previous consulting life, a customer contacted us and asked for an evaluation of their architecture. They were attempting to scale the business and encountering… obstacles. A team looked over their software and database designs and recommended a rewrite of their custom code. We supplied a proposal (including an expensive estimate) to deliver the re-architecture, redesign, and rewriting of their code.

They felt the price was too high.

We understood, so we countered with a proposal to coach their team to deliver the same result. This would cost less, the outcome would be the same, and their team would grok the solution (because their team would build the solution). The catch? It would take longer.

They felt this solution would take too long.

We understood, so we offered to help in any way they would allow. The company sought help from other firms. A competing consulting firm offered to migrate their code and databases to a cloud provider. The company liked this solution and embarked on a migration.

The migration succeeded.
It took longer than expected.
The company hemorrhaged cash and clientele throughout the process, and is now a fraction of its previous size in people, clients, cash flow, and profit.

The lesson? Migrating to the cloud wasn’t a bad move for them.
Ignoring the advice of experienced architects was a bad move for them.

Did this client need to move to the cloud? Perhaps. More than anything, this client needed to pay off a tremendous amount of technical debt. By the time we were called, they were well into technical inflation and slipping into a technical depression.

Yes, the cost for a fix was high.
No, the cost for a fix wasn’t as high a price as the costs they have paid for not fixing their code. And…

… the cloud did not fix their bad code.

Heartbreaking

This was heartbreaking to witness. We could have helped them. Why were we unable to help them? We told them the truth; our competition told them what they wanted to hear.

As Brent Ozar pointed out in his post titled What It’s Like to Work for a Bad Contracting Company:

See, bad contracting companies have a huge sales force that goes out and asks customers, ‘Hey, what problems are you having? Sure, we can solve those. Just sign here.’ They make impossible promises about their – well, YOUR – capabilities and timelines.

Brent is right. This is why I ask, “Are you getting the best technical help available? or are you being sold technology by the best sales people?” There is sometimes a difference between consultants who are good at sales (or who hire good sales people) and consultants who are good at technology.

I’m going to take that one step further:

The very best technologist – the one most able to solve your problem for the lowest cost – may be someone who personally irritates you and who charges an hourly rate higher – perhaps multiples higher – than the competition.

Allow me to be (more) blunt:

  1. Did the company in the (true) account above believe they were going to save money?Yes, they did.
  2. Did the company in the (true) account above save money?No, they did not.

In my post titled A True Story of Value vs. Hourly Rate I share another true story of how our awesome, experienced team at Enterprise Data & Analytics was able to deliver value to a client. Even though we charged more per hour than the competition we delivered more value. How do we know?

The client did the math.

I’m writing this post to beg you to also do the math.
I’m writing this post to ask you to buy consulting services (if you need them) and not just be sold consulting services (whether you need them or not).
I’m writing this post to let you know that the cloud is awesome. But the cloud is not free.

“How Much Does Azure Cost?”

I hear this question a lot and it is difficult to answer. You can certainly browse over to the Azure Pricing Page to learn more. The difficulty is estimating how many __U’s you are going to use (see what I did there?).

Brent asked a really good question on Twitter and got this awesome answer:

The website is azureinstances.info and it. is. awesome.

How to Fix Being Charged for an Unused ADF Integration Runtime?

Stop ADF IR:

Delete unused cloud resources:

Peace.

Learn More:
Free Webinar – The Azure Data Factory Controller Design Pattern – 28 Jun at noon EDT

Honored to Present an SSIS and ADF Precon at Data Platform Summit 2018!

Expert SSIS – Live, Online, 2.5 Days – 10-12 Sep 2018 – delivered in cooperation with Brent Ozar Unlimited

Honored to Deliver Intelligent Data Integration – a PASS Summit 2018 Precon

ADF Execute SSIS Package Activity

The good people who work on Azure Data Factory recently added an Execute SSIS Package activity. It’s pretty cool. Let’s tinker with it some, shall we?

First, you will need to create an Azure Data Factory SSIS Integration Runtime. If you don’t know how, that’s ok – I’ve written a post titled Lift and Shift SSIS Part 0: Creating the ADF Integration Runtime that describes one way to set up ADFIR.

0. Connect to Azure Data Factory

You can connect to the Azure Data Factory (ADF) dashboard here. After you connect…

1. Navigate to the Author Page

The “author” is indicated by the pencil icon on the left side of the ADF dashboard:

On the Author page you may add ADF resources by clicking the “+” symbol beside the search textbox. The dropdown menu contains three items at the time of this writing: Pipeline, Dataset, and Copy Data. Select Pipeline:

2. Drag an Execute SSIS Package Activity Onto the Surface

Click  the new Execute SSIS Package activity to edit it, clicking the tabs shown below the Azure Data Factory Integration Runtime (ADFIR) surface. Configure the Execute SSIS Package activity name on the General tab:

3. Configure the SSIS Package to Execute

Configure the Execute SSIS Package activity’s SSIS Package on the General tab. Set the Catalog Package Path property to <folder>/<project>/<package> as shown below (click to enlarge):

I can hear you thinking, “But Andy, what if I don’t know the Package Path?”

Fear not. The next release of Catalog Browser – a free utility from the DILM Suite (Data Integration Lifecycle Management Suite) – displays the Catalog Package Path when you click on an SSIS Package:

Please note: Catalog Browser displays backslashes in the Catalog Package Path and ADFIR’s Execute SSIS Package activity expects forward slashes.

The next version of Catalog Browser is scheduled for release on or before 1 Jul 2018.

That’s it. A quick Debug execution will verify the ADF pipeline executes:

Use SQL Server Management Studio (SSMS) to connect to the server hosting your Azure Data Factory Integration Runtime. View the All Executions report and confirm the execution happened and succeeded:

For more information about using SSMS to connect to ADFIR, please see the post titled Lift and Shift SSIS Part 1: Deploy Integration Runtime SSIS.

Learn More:
Free webinar: Introduction to Lifting and Shifting SSIS to the Cloud
Andy Leonard
21 Jun 2018 at noon EDT
Register today!

:{>

Lift and Shift SSIS Part 1: Deploy Integration Runtime SSIS

What if I told you, “You can quickly and easily deploy Integration Runtime SSIS projects to Azure Data Factory?” You can! As shown in the image, you can simply right-click the SSIS project name in SQL Server Data Tools (SSDT) Solution Explorer and click Deploy to deploy the project to the Azure Data Factory (ADF) Integration Runtime. You can also start the Integration Services Deployment Wizard by navigating to an ISPAC file using Windows Explorer and double-clicking it.

In Part 0 we learned how to create and configure an Azure Data Factory Integration Runtime. In this post we connect to ADFIR, deploy an SSIS project, and examine options for viewing the contents of the ADFIR SSIS Catalog.

First, Deploying SSIS

Follow the Wizard… the Integration Services Deployment Wizard, that is. Connect to the Destination and create a Catalog folder if needed:

Next comes Validation. Be sure to address any errors that appear here:

Review is next:

And then, finally, Deploying Integration Runtime SSIS projects is as easy as deploying SSIS projects to an SSIS Catalog on-premises:

There’s really very little difference between deploying an SSIS project to an on-premises instance of an SSIS Catalog and deploying an SSIS project to an instance of the ADF Integration Runtime.

That. Is. Awesome!

What Happens When You Deploy Integration Runtime SSIS Projects?

Remember all the waiting back in Part 0? ADF was creating a copy of the SSISDB database – which contains all the goodies for the SSIS Catalog. that’s right – there’s a real, live copy of an SSIS Catalog stored in the database on the server you created in Part 0. You can view it in the very same way you would view an SSIS Catalog that resides on-premises: using the Integration Services Catalog node in Object Explorer in SQL Server Management Studio (SSMS).

Viewing the Catalog in SQL Server Management Studio (SSMS)

Open SSMS and begin connecting to a database:

Before you click Connect, click the Connection Properties tab and change the “Connect to Database” property to SSISDB:

You should now be able to view the ADF SSIS Integration Runtime using the SSMS Object Explorer’s Integration Services Catalogs node:

If you do not change the “Connect to Database” property to SSISDB, you can still connect to the Azure SQL DB instance, but you will not see the Integration Services Catalogs node. Instead, you will see nodes for Databases and Security:

All is not lost. Simply reconnect – this time remembering to change the “Connect to Database” property to SSISDB on the Connection Properties tab.

Viewing the Catalog with SSIS Catalog Browser

You can also use SSIS Catalog Browser (it’s free!) from DILM Suite to surface contents of the SSIS Integration Runtime. Simply connect to the ADFIR server:

Catalog Browser surfaces rich SSIS Catalog in a unified view:

Cool, huh?

Viewing the Catalog with SSIS Catalog Compare (v3 Preview)

One last thing…

SSIS Catalog Compare version 3 is in Preview (want a copy? Contact me!) and facilitates direct lift and shift from on-premises SSIS Catalogs to ADF Integration Runtime SSIS Catalogs:

Data engineers can use SSIS Catalog Compare to deploy directly to ADFIR or to generate scripts for:

  • Entire Catalogs
  • Catalog Folders
  • SSIS Projects
  • Literal Overrides
  • Catalog Environments
    • Catalog Environment Variables
  • References
    • Reference Mappings

For more information, visit DILM Suite SSIS Catalog Compare.

Conclusion

Don’t forget to tear it down when you are done!

Lastly, some Azure resources are expensive. You may not desire to leave ADFIR running, especially if you’re done tinkering and learning.

Learn More:
Free webinar: Introduction to Lifting and Shifting SSIS to the Cloud
Andy Leonard
21 Jun 2018 at noon EDT
Register today!

:{>

Lift and Shift SSIS Part 0: Creating the ADF Integration Runtime

Let’s face it: All the cool kids are playing in the cloud. And why wouldn’t they be? Microsoft Azure gets new features every week… sometimes every day! One shiny feature is Azure Data Factory Integration Runtime (ADFIR) which is new in Azure Data Factory version 2. In this post I will show you one way to create the Integration Runtime.

Connect to Azure and Create a SQL DB

First, connect to the Azure Portal. Click the “Create a resource” link and add a new SQL DB and server. I like to create a new resource group as well because a resource group allows me to group together a bunch of related resources. It’s also easier to tear down once I’m done. (Who wants to pay extra for leaving things running after you’re done tinkering? Anyone? Bueller? Bueller?)

Create an Azure Data Factory

Next, click the “Add a resource” link, select Integration from the topics on the left, and then click Data Factory to create an Azure Data Factory:

Walk through the configuration steps, selecting unique names where required. I tend to choose “lesser” options when they are presented. Again, I do not wish to pay for resources unless and until I need them.

Once your Azure Data Factory is configured you can browse to the Author & Monitor site:

When you reach the ADF Author & Monitor site, click the image and link to “Configure SSIS Integration Runtime”:

Configuring the SSIS Integration Runtime

There are three steps to configure the SSIS Integration Runtime:

1) Set the Name, Location, Node size and number (click to enlarge):

2) Configure SQL Server settings (click to enlarge):

3) Configure Maximum Executions per Node and Advanced Connections settings (click to enlarge):

Wait…

At the time of this writing it takes 20-30 minutes for the Azure Data Factory Runtime to be created:

It will finally start. Promise:

Conclusion

That’s it! You’ve configured Azure Data Factory’s Integration Runtime.

See Part 1 to learn more about deploying SSIS projects to your shiny new ADFIR!

Learn More:
Free webinar: Introduction to Lifting and Shifting SSIS to the Cloud
Andy Leonard
21 Jun 2018 at noon EDT
Register today!

:{>