Skip to main content

8 posts tagged with "Engineering"

View All Tags

· 6 min read
Mike Schlosser

At Optum we believe healthcare data is one of the most powerful tools that can drive innovative clinical solutions and improve patient experience. We work with petabyte scale data sets to do work on some of healthcare’s more impactful problems, such as predictive analytics, end of end patient views for individuals, and robust digital health platforms that allow care anytime from anywhere.

Historically one of the biggest challenges we have faced is around data access, governance, standardization, and movement. Accessing datasets securely and quickly is one the biggest drivers of unlocking engineering teams. At Optum we are in the middle of a transformative transition from centralized data and data ownership to a decentralized model based on principles from Data Mesh. By moving to a decentralized data ownership and governance model, teams are encouraged to expose quality datasets in a secure, auditable, and discoverable manner via Optum’s implementation of Data Mesh.

This post is the first of a 4-part blog and describes our evolution towards Data Mesh as well as our experience leveraging and extending a popular open-source metadata platform, DataHub. In follow up blog posts we will expand on our usage of DataHub and go into depth on

  • How we have extended the metadata model within DataHub
  • Building custom metadata ingestion plugins to populate our data mesh
  • Building event-based triggers to act on real time on metadata changes

History: Healthcare Data Platform – Standard Streams & APIs

hcp-old

Our initial iteration on healthcare platform largely revolved around 3 access patterns – standard APIs, standardized data streams and a common big data ecosystem. Data was to be sourced from one of the three above patterns and no direct access to data was discouraged.

This largely manifested technically with a change data capture process reading transaction logs on data bases and emitting events to Apache Kafka. Those records were then denormalized under a given key and joined together to build the standardized output Kafka topic. The standardized Kafka topic was then stored in modern data platforms such as Elastic and Cassandra which were then queried to serve API traffic.

This pattern proved successful for most use cases. However, a downside was that this approach relied heavily on a set of standardized schemas, which were difficult to identify across lines of business with the vast scope within Optum. Governance to determine schemas and access to data grew in difficultly and complexity as the platform expanded. Because everyone had slightly different use cases, this occasionally meant teams still needing access to the source data rather than the standardized version.

Now: Data Forge, Data Mesh & DataHub

Today, Optum is taking a data mesh approach to data governance, access, discovery, and use.

mesh-overview

Our current iteration of data platform is one that connects data producers and consumers across the enterprise, utilizing their preferred patterns. Data Mesh provides a new way of thinking about data at Optum. It is a fundamental shift away from aggregating data in a single lake or database and trying to centralize data governance. By enabling data mesh, we enable governance and ownership are approached in a decentralized pattern. This allows the stewards who know their data, access requirements and restrictions to make decisions on what data is exposed and who can access it.

Data Forge focuses on three personas to drive adoption of Data Mesh:

  1. Producers: Data Producers can create data products from 1 or more data sets which involves adding additional metadata aspects. These additions have proven to increase discoverability and provide necessary context around data products. This encourages quality datasets and incentivize data products to expose the most up to date and accurate data sets.
  2. Consumers: Data Consumers can discover and subscribe to a data product via the Data Forge UI. Data is quickly discovered and understood, and engineering and analytics teams at Optum realize faster time to value through streamlined access review and provisioning. Additionally, Data Forge provides a built-in data movement capability to move data to and from various platforms depending on use case (more info here).
  3. Stewards: Stewards ensure appropriate data access and usage that aligns to security, privacy, and contractual policies. Typically, stewards can approve or deny data subscriptions to grant access to data products.

data-as-a-product

We leverage the idea of data products internally - which vary from data sets. Each of the three personas plays a unique role in the creation, usage, and governance of data products. Data products require additional metadata around the data and encourage usage of preferred data sets (more up to date, more accurate, etc.).

Datahub

Data Forge is built on top of Linkedin's DataHub and extends it with various metadata model enhancements and aspects.

datahub-logo

Data Forge heavily leverages Datahub, a modern open-source metadata platform originally from Linkedin. This documentation provides an architectural overview. Datahub has provided us an excellent base from which to build on Data Mesh, and leverage much of its functionality out of box, such as its ingestion framework, search capabilities, and built-in metadata aspects.

Extendibility

One of the main benefits we have leveraged from DataHub is its extensibility. We can easily use low code methods to extend the metadata model to add various aspects (pieces of metadata) as well as top level entities (think datasets, pipelines, and machine learning models). This allows us to extend to support internal representations of data and add additional business specific context as necessary.

We also have built plugin extensions to support custom internal platforms to ingest metadata. This was a significant upgrade over many metadata platforms previously used that weren't as extendable and could only support ingestion via out of box platforms.

Finally, DataHub’s event driven architecture provides us a mechanism to act on any metadata changes in real time. This allows us to perform various actions like provisioning access to a data product, notifying consumers on any schema changes that may affect their application or triggering data movement jobs to move data from source to sink platforms.

Our engineering team will elaborate on these in more detail in future posts with specific tutorials and walk throughs!

Community

The DataHub community is active and growing and we are excited to continue to invest in it. The community has significant say on its roadmap and the core maintainers have been extremely supportive of our use cases. We are thrilled to be listed as adopters on DataHub's official documentation site.

The community has made it seamless to begin contributing back to DataHub. As a result, we have begun contributing fixes and enhancements back to upstream. Examples of upstream contributions include support for TLS when connecting to Schema Registry from the metadata service and various enhancements to the Snowflake ingestion plugin.

Community support is available in the DataHub Slack channel. The community is active and growing rapidly. It has provided our team a fantastic way to have influence on road map, collaborate with the community and get exposure to how others are managing data in a timely manner.

· 5 min read
Murugappan Sevugan Chetty

Knative is an open-source community that values end-user contributions/interactions highly, this is evident in PRs, monthly meetups, slack interactions, etc. To top it all, recently Knative opened an end-user seat in the steering committee. This is very significant for a couple of reasons

  • Instills confidence in end-user organizations to use Knative in production
  • Very few OSS communities have end-user representation in the steering committee

I am pleased to share that I have been elected as the first end-user representative in the steering committee. In this blog, I would like to share my organization’s Knative experience so far.

Why Knative?

First things first, Knative was not picked for being the next shiny object. The whole process was organic. The following factors led to the need for a Kubernetes abstraction.

  • Mature container ecosystem.
  • Serverless workloads
  • Need for better developer experience than vanilla Kubernetes (deployment + service + ingress + ...)

The situation was ripe for building a new platform on top of Kubernetes. Before doing that we wanted to check if we could leverage an existing open-source solution, there were a handful of them at various maturity levels, Knative was the one that stood out for us

  • Best scaling performance compared to all its peers.
  • Developer experience (single knative service resource yields Kubernetes deployment + service + ingress + certificate + custom domain).
  • Robust Primitives for building a developer platform.
  • OSS Community with a good governance model
  • Not backed by a single vendor
  • Great support for an open and free project

Implementation

To cater to developers with varying skill levels we built an abstraction on top of Knative. This layer kept the barrier of entry low for new users and could be bypassed for advanced users. We also built components to integrate with our enterprise applications. The below pictures show our deployment model and integrations we had built in-house.

With the above deployment model and integrations, we were able to give clean code to URL experience for our internal users. With just a few params, developers were able to deploy their services and receive a TLS enabled secured URL in seconds.

Interaction with Community

Two years since our first Knative service deployment, our platform has grown at a rapid pace serving hundreds of developers. This would not have been possible without the Knative community support. There are various avenues to interact with the community.

Slack

To this day, there is no slowing down the momentum in the slack channels, most concerns get addressed almost immediately. Slack is the place to get started on Knative.

Hacky Hours

If you ask me to choose the best experience of Knative so far, weekly hacky hours would be on the top. These casual/virtual meetups used to be on Friday afternoons. Every week there were several 'aha' moments with demos by Knative maintainers/contributors/end-users. In these meetings, I could get 1 on 1 with the who's who of Knative and validate our implementations.

Monthly Meetups

Hacky hours might have been paused, for now, monthly meetups do happen every month. This is a great place to know the project TLDR and some neat use case demos. I had the privilege of showcasing our Knative implementation here. There is no better way to validate an implementation than giving a talk at a meetup/conference. For end-users, I definitely recommend demo'ing the solutions here.

Contributions

During our implementation, we did hit some hiccups and some missing features, which gave us the chance to open PRs and contribute upstream. The maintainers were receptive to our issues and most of them got merged, for the ones that didn’t there were valid reasons and good discussions around them to learn. I have contributed to most of the repos, but some of my significant contributions are to client repo where I was eventually elevated to a reviewer.

Apart from upstream contributions, we have built custom Knative channel implementation and Knative event sources and open-sourced under our GitHub org.

Our Knative implementation gave me the opportunity to speak at conferences. Most significant being Kubecon and IstioCon. Now with Knative in CNCF, it should open more doors for the end-users to share their story at Kubecon.

Closing Thoughts

In closing thoughts, I would like to share what Knative has been for me. A couple of years ago, I started experimenting with Knative not knowing that I would be on the Knative steering committee one day. I have not contributed to every single OSS project out there but have been part of various communities, raising issues, providing feedback, being on the slack channel, etc., with that experience I can confidently say being part of this community has been rewarding on every front. It wouldn’t be an exaggeration to say that Knative has been one of my driving forces during these covid times.

· 4 min read
Russ Rogers

Low-code / No code systems are touted as faster ways to do development but it's important to understand what they do well and to avoid the shortcuts that bypass good engineering practices.

Let's clarify what "no code" is. These systems allow coding in a declarative "configuration" rather than a traditional programming language. These configurations may take the form of JSON, YAML, XML, properties files, or proprietary formats. They work to simplify the coding tasks by eliminating some of the complications of syntax, module structure, constructs like functions, flow control, and variables. I include Domain Specific Languages in here as well. Nevertheless, when these configurations are interpreted at runtime, the result is a "program". Programming languages, whether compiled or interpreted, are themselves simplifications on top of lower-level machine instructions. It's a matter of how much abstraction vs. control provided to the user.

Conversely, "everything as code" emphasizes the idea that most aspects of a system benefit from the same treatment as traditional "source code". This includes the deployment of the infrastructure, application configurations, automated code deployment, and testing / validation. There are also some process elements like version control, release management, and code reviews that should be considered as part of an "everything as code" philosophy too.

These are not incompatible paradigms. Terraform configurations are a form of "low code" for infrastructure specification.

Expanding the pool of "developers"

By enabling SMEs in particular subject areas to provide input directly as "executable", they can eliminate a step in the translation of the specification into the instantiation of a solution. It may also enable use of lower-cost resources, providing a way of scaling out development work.

Don't skip good engineering practices

With the inclusion of non-engineering members into the development process, it's important NOT to also throw out engineer practice when using a low-code/no-code solution. When "no code" is touted for its "speed", there can be pressure to loosen process controls at the same time, especially if they are not part of the team's development culture.

Version Control

Because the configurations are often text-based, they work well in version control systems, allow for differencing, versioning, and structured collaboration. Release planning, branching, and change management, etc are still valuable regardless of the programming environment. It can serve as a central knowledge store for process control audits.

Code reviews

In good engineering practice, a code developer makes a change and submits a request (PR) for the change to be accepted. The change is reviewed and may require "approval" by someone other than the original developer, to catch unintended issues and enforce coding standards. This is true whether the change is C++ code or a no-code configuration file.

Unit and Integration Testing

The ability to build tests to verify that the code works as expect is an important tenant of agile development. It may not be applicable in all cases, especially where the systems to not support reuse but it should not be discarded out of hand.

Deployment

Automated, hands-off deployments, especially to production environments, is a good way to ensure rigorous progress has been followed and avoid unintended changes making their way into production. Again, this can be important for audits.

Take-aways

Don't skip good engineering practices when employing a low-code/no-code solution. Think about the "no code" configuration as another piece of the "everything as code" ecosystem and be sure the "non-coders" on your team understand the benefits of good engineering. If you're already using Terraform, you have a no-code / low-code component in your stack and you'd never consider not using the good practices above for that piece of your application.