Our DevOps Journey is Incomplete without Data

Data, Data Management, Data Architecture need our attention. If we could go as fast as only our weakest link, then Data needs our love.

Every company is a "Software Company", "Software is eating the world", in these similar lines I recently heard every company regardless of size is a "Data Company". True, somewhere or other every organisation produces, consumes, analyses, reports Data & makes decisions based on Data.

DevOps momentum has seen a rapid growth rate of new tools in the space of CI/CD, ARA (Application Release Automation), frameworks for enabling application delivery at pace. When it comes to Continuous Delivery, modern architecture patterns, practices like Microservices, our delivery teams face challenges with data.

If we could go as fast as only our weakest link, then Data, Data Management, Data Architecture, and associated practices need our attention and love.

Who owns the Data?

The typical mentality in any Enterprise is that the "Data Team" or "Data Management Team" owns the data. The question is, how did we get into this situation? The term "Silo" in DevOps hits me hard, as I see that the Data Team is the biggest silo in the entire Organisation and Value chain.

The problem, is it Data, the Data People or both?

The "Enterprise Data team" inherits a lot from their predecessors — the process, standards, procedures, planning, execution, operations and management. The challenge is recognising and accepting the opportunity to improve, modernise, and simplify.

Breakthrough Challenges

It is not easy, but it is not impossible.

  1. Naming Conventions: Having meaningful data model, schema, table, column names are important for developers and Ops.

  2. Domain, Boundary, Schema: If we are breaking the monolith and taking the route of DDD, Bounded Context, then we need to move to database/service boundary.

  3. Flow — Requesting Data model changes: We moved from SharePoint forms to PR-based workflows.

  4. Adopting modern practices: We helped our DBA to adopt Source Control systems and automated database deployment via pipelines.

  5. Relational or NoSQL? We pushed the agenda for JSON schema & payloads.

Learnings and Advice

Problems are not unique when it comes to common themes. Spread and Share — you need a platform to share learnings. Keep challenging: you are not alone, do not give up. Be sensible in Data architecture — architect/design systems in such a way that we could do "Continuous Delivery."