Data Engineering: Collecting, rationalizing, and compiling disparate data to make it dashboard-ready

The Solution
SPK and Associates started by listening to the customer, and understanding the business needs which underpinned the data needs. We scrutinized the existing environment, identifying available resources as well as gaps that needed filling. And we employed an Agile approach to the endeavor, in order to make fast, incremental changes toward the final solution. Leveraging our knowledge of cloud, databases, and data engineering, SPK and Associates devised a “data transformation pipeline” which would ingest all of the disparate data sources, and automatically rationalize them into dashboard-friendly data. The “pipeline” analogy is apt, for our solution fed the different data sources, incrementally, into a data warehouse, where it could be easily manipulated to create visualizations and reports for the different business stakeholders. Specifically, the SPK solution employed four different platforms, each of which utilized specific configurations and custom code that we created for them:

• Amazon Web Services provided both server-based and server-less computing resources for the project.
• Snowflake provided data-warehouse functionality; this was where all of the normalized data was stored.
• Matillion, an ETL (extract, transform, load) tool, allowed SPK to create a pipeline of transformation steps; it was especially helpful because it integrates into third-party computing resources. Matillion provided the data import, modification, and output steps. Our code was executed on the Matillion platform.
• Grafana is a dashboarding application, but in this instance, it served as one of our sources of data.

Thus all of these components worked in relay-race faction: Data in from Grafana, processed through Matillion, output to Snowflake for consumption by the business.

The Results
Company stakeholders now have full access to a wide assortment of business data. They can easily create reports that are meaningful to each of their teams, allowing them to make better business decisions as a result. Among
the new at-their-fingertips data are KPIs such as:

• How many servers are in their fleet
• How many are underutilized
• Which services they are attributed to
• The costs associated with each

Importantly, SPK freed up the company’s engineering team to focus on developing their own product, rather than expending valuable time trying to develop this internal tool.

The Savings
As noted above, the earlier vendor’s attempt at a solution—never completed—had an estimated timeframe of six months.

SPK and Associates was able to create this entire solution, from scoping to implementation, in just one month.  That’s six times faster than what the other vendor had promised (yet never delivered).

There were savings in cost, too. The SPK solution was vastly simplified, compared to the previous attempt; our solution saved our client approximately 75 percent in computing and licensing costs alone.