IT transformation is inevitable, and the technology refresh cycle is becoming more and more aggressive and competitive. Open source has not only gained trusts of public sector enterprises but also into more regulated businesses and organizations. CIO’s office is constantly pushing for more innovative ideas, cost savings, and auditing their existing systems. Their guiding principles focus on evaluating open source solutions, cloud first approach, avoiding vendor lock-in, being agile, scalable, and all-encompassing security. The Big Data ecosystem has been following these principles very closely. The contributors to this ecosystem have seen the transformation as a necessity to solve for their specific needs and helping others in similar situations.

Change is driven by the consolidation of IT assets and legacy investments that have occurred in the past. The value proposition is even more attractive when organizations have control over licenses, users, and application teams using the right and preferred technology.

When should an organization realize a need for IT transformation?

The answer is, it should be an ongoing process where evaluating technologies, existing systems, and infrastructure must occur constantly. However, it should occur in alignment with the overall Business priorities. The priorities not just being “save me money” but also that maps to where the business wants to grow, customer experience, and new market discovery.

Here is an example from my experience with an insurance firm. Being a highly regulated business, the leverage of introducing new products and changing existing products was always a topic of scrutiny. This was not just from the customers but also from each of the state governments and other regulatory institutions. For us to succeed and maintain our current business, it was necessary first to understand what kind of data assets we had, especially being a company that was in business for many decades. During our discovery phase, it was important to map each of the components to a business process and identify what we called as the Business Improvement Opportunities (BIO’s). The result was a scalable business data lake catering to various lines of businesses, and with an ability to help consolidate existing commercial systems and software technologies.

The strategy we followed for multiple customers has now matured into a solution accelerator, helping us jump start similar engagements with our customers. Categorizing each of the discovery phases into an Enterprise Architecture framework, as shown in the picture. It is a top-down model that is inclined towards a “Business Domain Driven Architecture” model. Each of the BIO’s map to its relevant data attributes. These can be in the form of some complicated data architecture models like Data Vault or a more traditional data warehouse based on Kimball, or Inmon. The application layer enables presentation, processing, and maintenance of these data structures based on the business needs and priorities. Technology layer is what hosts these systems; that is the data center. This is where encryption and security setup occurs. Skilled technicians spend time on performance and monitoring options to optimize applications and recommend how data structures can or should be designed to gain better business results.

For each of these architecture layers, a multi-phase approach is recommended. Phase one is to conduct an early assessment of existing systems and IT assets. Gather vision statement from the business stakeholders, list out current pain points, challenges and future enhancements via workshops and user interviews. Phase two is to combine these discoveries into workable units of work. Areas that require an evaluation of technologies translate into proof-of-concepts, a list of “nice-to-haves” go into a feature list, and business use cases become user stories. Phase three combines everything together, takes these results and produces recommendations and an implementation plan that is backed by facts and results.

Of course, each customer engagement for us has been very customized and different. The use cases have varied from a Big Data Hadoop data lake to a zero-loss micro services platform. In each of these situations, the success was realized upon results and proof-of-concepts, showcasing the value behind such transformations.