Lift and Shift is Not a Strategy…

Lift and shift is a bridge.

If your enterprise is considering migrating enterprise data and data integration workloads from on-premises to the cloud, lifting and shifting is not the end of the story. Lifting and shifting is, rather, the mechanism – the conversion project – that positions your enterprise to leverage the economies of scale afforded by cloud technology.

Andy’s Lift and Shift FAQ

“Should We Lift and Shift?”

I hear this question often and my response is, “When it makes sense to do so, yes, you should lift and shift.” This begs the next question, which is…

“How Do We Know It’s Time to Lift and Shift?”

My engineer-y response to this question is, “You will know it’s time to lift and shift to the cloud when you want to leverage functionality available in the cloud that is not (or not yet) available in on-premises versions of the enterprise platform(s) in use in your enterprise.”

“What Are Some Advantages of Migrating to the Cloud?”

The biggest advantage of lifting and shifting enterprise data to the cloud is the ability to efficiently scale operations. By efficiently, I mean quickly and easily – especially when compared to the time and expense (don’t forget opportunity cost when calculating expense) to scale up systems on-premises.

The ability to scale up and scale down on demand is a huge advantage for some business models which experience “spike-y” demand for operations at different times of the year, quarter, or month. But even if that’s not the case, all data scales. It’s very handy to be able to connect to the Azure Portal and move a slider (as opposed to purchasing and provisioning more hardware…).

There’s a brand new (in my opinion) “knob” exposed by cloud-enabled efficient scaling. As I wrote in my post titled Time and Money in the Cloud:

Let’s say you pay $100 to incrementally load your data warehouse and the load takes 24 hours to execute at the scale you’ve selected in the cloud. Prior to thinking in DTUs, engineers and business people would think, “That’s just the way it is. If I want more or faster, I need to pay for more or faster.” But DTU math doesn’t quite work that way. Depending on your workload and DTU pricing at the time (FULL DISCLOSURE: DTU PRICING CHANGES REGULARLY!), you may be able to spend that same $100 on more compute capabilities and reduce the amount of time required to load the same data into the same data warehouse to minutes instead of hours…

The fact that the cost/performance curve can be altered in seconds instead of months meta-changes everything.

“Are There Disadvantages of Migrating to the Cloud?”

It depends. (You knew that was coming, didn’t you?)

Enterprise Data & Analytics helps enterprises migrate data, data integration, lifecycle management, and DevOps to the cloud. In some cases (~30%), the enterprises spend a little more money in the near-term. There are two reasons for this:

  1. When upgrading, it’s always a good idea to operate new systems in tandem with existing systems. In a lift and shift scenario, this means additional expenses for cloud operations while maintaining the expense of on-premises operations. As cloud operations are validated, on-premises operations are shut off; thereby reducing operating expenses. In truth, though, this dynamic (and expense) exists whether one is lifting and shifting to the cloud or simply upgrading system on-premises.
  2. “Standing on the bridge” (more in a bit) can cost more than remaining either on-premises or lifting and shifting the entire enterprise workload to the cloud.
  3. Regulatory requirements – including privacy and regulations about which data is allowed to leave nation-states – will constrain many industries, especially government agencies and NGOs (non-governmental organizations) who interact heavily with government agencies.

Standing On The Bridge

One option we at Enterprise Data & Analytics consider when assisting enterprises in lift and shift engagements is something we call “standing on the bridge.” 

Standing on the bridge is present in each lift and shift project. It’s one strategy for implementing hybrid data management, which almost every enterprise in the cloud today has implemented. Hybrid means part of the enterprise data remains on-premises and part of the enterprise data is lifted and shifted to the cloud. 

Hybrid is implemented for a variety of reasons which include:

  • Mitigating regulatory concerns; and
  • As part of the normal progression of lifting and shifting enterprise data and data workloads to the cloud.

Standing on the bridge for too long is a situation to avoid. 

“How Do We Avoid Standing on the Bridge For Too Long?”

Planning. Planning is how an enterprise avoids standing on the bridge too long. Your enterprise wants advice from experienced professionals to shepherd the lift and shift operation. 

Enterprise Data & Analytics can help.

Helpful Tools

Enterprise Data & Analytics has been delivering, and writing and speaking about Data Integration Lifecycle Management for years. 

We’ve built helpful tools and utilities that are available at the DILM Suite. Most of the DILM Suite tools are free and some are even open source. 

Enterprise Data & Analytics Welcomes Shannon Lowder!

I am honored and thrilled to welcome Shannon Lowder (@shannonlowder | blog | LinkedIn) to the Enterprise Data & Analytics team!

Shannon is a data engineer, data scientist, BimlHero (though not listed on the page at the time of this writing), and shepherd of the Biml Interrogator open source project. If you use Biml to generate SSIS projects that load flat files, you need Biml Interrogator.

Shannon, Kent Bradshaw, and I are also co-authoring a book tentatively titled Frameworks. (Confession: Kent and Shannon are mostly done… I am slacking…)

Shannon brings a metric ton of experience to serve our awesome clients. He has years of experience in data analytics, serving recently in the role of enterprise Lead Data Scientist. Shannon’s experience spans supply chain management, manufacturing, finance, and insurance.

In addition to his impressive data skills, Shannon is an accomplished .Net developer with enterprise senior developer experience (check out Biml Interrogator for a sample of his coding prowess).

Shannon is a regular speaker at SQL Saturday events, presenting on topics that include Business Intelligence, Biml, and data integration automation. He is a gifted engineer with experience in framework design, data integration patterns, and Azure who possesses a knack for automation. Shannon is an avid “tinkerer” who enjoys learning. He has experience implementing Azure Machine Learning and applying AI to predictive analytics using sources classified Big Data. Shannon is also a practitioner of data integration DevOps with SSIS. In other words, he fits right in with our team here at Enterprise Data & Analytics!

As Shannon writes on his LinkedIn profile:

I am a data guy with a passion for partnering with clients to solve their database and technology issues. Over my career, I’ve played all the roles: database developer, administrator, business intelligence developer, and architect, and now consultant. I’m the guy you call in when you have the impossible problem and everyone tells you it cannot be solved. I automate solutions in order to free your current staff to do the higher value tasks. I bring solutions outside of traditional relational database solutions, in order to find the shortest path between you and your goals.

As an accomplished Microsoft SQL data professional, recognized BimlHero, and practicing Data Scientist, I’m the resource you need to extract the most value from your data.

I’m humbled and thankful and excited to watch Enterprise Data & Analytics continue to (quietly) grow – adding cool people (another announcement is forthcoming) and service offerings like Data Concierge. It’s very cool to watch!

Welcome Shannon! I’m honored to work with you, my brother and friend.

For more information, please contact Enterprise Data & Analytics!

It’s Biml 2018 Release Day!

Scott Currie and the team at Varigence announce the release of BimlFlex 2018, BimlStudio 2018, and BimlExpress 2018!

This much-anticipated release includes awesome features for experienced Biml professionals as well as for data engineers new to automating processes!

BimlFlex 2018

BimlFlex is a mature and complete data engineering automation framework. Out of the box, BimlFlex automates several data models, including:

  • Staging Database
  • Persisted Staging Area (ODS)
  • Raw Data Valut
  • Business Data Vault
  • Dimensional Data Warehouse
  • Data Marts
  • Cubes
  • Tabular

The benefits of BimlFlex:

  1. Flexible Data Framework
  2. Robust Metadata Mapping Tool
  3. Avoid Data Debt
  4. Shorter Delivery Times
  5. Simplified Maintenance
  6. Upfront Pricing

 

I absolutely love the BimlFlex Savings Calculator near the bottom of the page (click to enlarge)!

BimlStudio 2018

BimlStudio 2018 includes support for all Azure Data Factory (ADF) version 2 items (as of 06/20/2018). Which ADFv2 items does BimlStudio 2018 support?

  • Linked Services
  • DataSets, Sources, and Sinks
  • Pipelines and Activities (including control flow)
  • Triggers

Also included is support for BimlScript PreCompiled Assembly Package (BSPCAP) files, which include the preprocessed binary assets for all of the BimlScript files in your project.

BSPCAP promises faster processing – especially with large codebases – for builds in the interactive designer and for command line builds. I’m looking forward to learning more about this feature!

There are too many new features to list here – go to the BimlStudio 2018 page and check them all out!

BimlExpress 2018

Varigence keeps giving away cool stuff! Nowhere is Varigence’s commitment to community more evident than in the feature list for BimlExpress 2018. The previous version – BimlExpress 2017 – included the Preview Pane. BimlExpress 2018 includes the ability to Convert SSIS Packages to Biml:

How cool is that? And it’s in the free (FREE!) version!

As with BimlFlex and BimlStudio, there are too many cool features to list here. Head over to the BimlExpress 2018 feature page to learn more.

And Happy Biml’ing!

:{>

Completed: Microsoft Big Data Professional Program Capstone Project

I’m excited to announce that I’ve completed – with help from my friend and brother and co-host of the Data Driven podcast, Frank La Vigne (Blog | @Tableteer) – the Microsoft Big Data Professional Program Capstone project!

The capstone is the last course requirement (of 10 courses) to complete the Microsoft Professional Program in Big Data. The official Professional Program certificate won’t be available until next month, but I’m excited to complete both the capstone and the professional program.

Although there was some data analysis included in the courses and capstone, I found a lot of data engineering was covered in the curriculum. For people wanting to learn more about Azure offerings for data engineering – including HDInsight, Spark, Storm, and Azure Data Factory – I highly recommend the program.

You can audit the courses, gain the same knowledge, and pass the same tests Frank and I passed – and even participate in a similar capstone project – all for free. You only have to pay to receive a certificate.

:{>

Achievement Unlocked: Certified in Implementing Real-Time Analytics with Azure HDInsight

I learned a lot completing Implementing Real-Time Analytics with Azure HDInsight. It’s the most fun Microsoft Virtual Academy / edX course I’ve taken to date!

In completing this course, I finished the requirements for the Microsoft Azure HDInsight Big Data Analyst program. I completed Implementing Predictive Analytics with Spark in Azure HDInsight and Processing Big Data with Hadoop in Azure HDInsight late last year.

What’s Next?

I started the Microsoft Professional Program in Data Science early last year, but I’m only a few courses into it. Anything worth starting is worth finishing.

I really got a lot out of the Big Data Analyst XSeries program. I’m eyeing the Microsoft Professional Program in Big Data program. I’ve already completed some of the courses as part of the Data Science program and the Big Data XSeries program. Some that I haven’t completed look very interesting – like Processing Big Data with Azure Data Lake AnalyticsOrchestrating Big Data with Azure Data Factory, and Analyzing Big Data with Microsoft R Server.

It’s time to update LinkedIn

:{>