Healthcare and data analysis

Healthcare is one of three domains we selected for initial applications of our framework for analysis of time oriented data (the other are security and finance). This is undoubtedly the most challenging domain, but also one with the biggest opportunities for delivering meaningful changes and positive impact. Healthcare is currently going through a radical revolution. And due to the nature of this domain, the consequences of the changes will affect everybody. We believe that data analysis is one of the key elements of the next generations of healthcare. But this also works in reverse – requirements and scenarios from this domain are great driving forces for innovation in data analysis.

Healthcare on the eve of a revolution

Healthcare is going through a gale of creative destruction, which will fundamentally change the landscape of the life science industry. It is primarily caused by scientific and technical progress: the revolution in sensors, always connected devices and capabilities to process and store massive amount of data. But it is not only about technologies themselves; it is also about how they have already changed users’ behaviors and expectations. Healthcare cannot be disconnected from the digital world that patients are used to. It is interesting to see that while many research efforts are still driven by the life science industry (e.g. genomics becoming more available), there are also strong initiatives originating from companies traditionally involved in information processing (IBM Watson, Microsoft Health or Apple CareKit). Revolution in healthcare is happening, even if there is significant resistance to change among medical professionals.

In our work we obviously focus on data analysis, which is actually very natural in the case of healthcare, as this domain is all about data. It always has been, long before the first signs of the digital revolution. Even when all doctor-patient interactions were direct and 1-on-1, they were based on insightful observations, and the doctor’s knowledge and experience to process them, to form a diagnosis and propose a therapy. Now we have more data, much more data, that can be available to patients and physicians. The data include genomics, anatomical imaging, physiological metrics, environmental records, patients’ (and physicians’) behaviors, decisions and observations.  There are many functional, technical and business challenges, but these data will eventually flow smoothly in networks of patients, doctors and AI systems. And then we’ll have to focus on different, but already familiar challenges: what to do with the new data that are within our reach and how to use them to help solve old problems? 

Data analysis will change healthcare

It should be clear now that we’re not talking here about data analysis understood as a basic process with some data on input and some results on the output. Data analysis in healthcare must be understood in a much broader scope, not limited only to selected, even if very useful, tasks, like analysis of anatomical images. Technical elements like data processing, extraction of information, creating models, learning from historical data, or providing practical decision support will always remain essential. But since data are expected to flow from multiple sources, and in many directions, data analysis should become one of the foundations, based on which required goal-oriented tasks can be effectively executed. And like any technology designed for digital healthcare, it must be extremely user-centered.

The goals of medicine should be always focused on the well-being of the patient. But in the digital world data are processed automatically, so the scope of data analysis can be much broader, without weakening that focus. What is more important, such expansion will actually increase the effectiveness of individual therapies. There are (at least) 3 levels at which data analysis should be considered in healthcare:

• Individual healthcare will be radically changed by personalized medicine based on individual characteristics and situation. The context of decision making will be vastly expanded by the technology beyond the knowledge and experience of medical professionals that are directly involved. On the other side, patients will become more active participants, also benefiting from decision support mechanisms delivering information in highly usable form.
• Healthcare relationships will become more recognized as essential for healthcare experience. There are concerns about reduced frequency of face to face interactions, but technology actually has the potential to help with establishing data flows and building strong trust-based connections. These should be the foundations for long-term relationships that are simple and convenient in good times, but remain very effective and natural in a case of a medical emergency.
• Population research will be redefined by the availability of detailed data about individual differences and the ability to select subsets of population with a high degree of similarity. Data about individual treatment, after proper processing, will be submitted for population analysis and help with detection of trends and patterns on a local or global level. More importantly however, results from population analysis will also be used in individual diagnosis and treatment.

Healthcare of the future will be very different, though the details still remain unknown. One thing however is certain, it will be based on various types of data automatically collected, shared, processed and analyzed in many ways we are not yet able to foresee.

A pattern case study

Data analysis is not only about big data, as there is also big value hidden in small data and simple methods that can be easily adopted to solve big problems. Figure 1 presents a very basic example of visualization generated using our framework. This simple scenario integrates data streams of different types: physiological metrics of body temperature, and two streams covering medicine intake below the main chart: one with a predefined schedule (MED1), which could be supported by adaptive reminders in a mobile device, and second with a basic record (MED2). The chart also includes analysis artifacts showing target temperature range, notifications when the ranged was reached (for the 1st time, and for continuous 24h), and the expected path of temperature change within selected timespan, assumed to be associated with scheduled treatment.

Figure 1. Tracking temperature and medicine regime with associated analysis artifacts for target value range, expected change, and detected relevant events (SVG)

This example is based on test data, but it is a useful illustration of the analysis & visualization pattern that could be applied in many medical scenarios. This pattern is based on streams with metrics of current state, streams with records of selected actions (including a plan), and artifacts generated from analysis of these data in the context of related models (possibly constructed based on analysis of similar cases). Healthcare is a very practical domain and it is fundamentally focused on a change. It is about learning from the past, understanding current conditions, and planning for the future, with emphasis on available options and the probability of their outcomes. This applies to the context of an individual patient, relationships between patients and physicians as well as populations of different scale. Healthcare is therefore not only about data in general, but about time-oriented data! This makes it a perfect application domain for our framework.

This is our first post about healthcare and it is the beginning of the longer story. We plan several more healthcare-related posts covering topics like practical security and privacy requirements, specific decision support scenarios and design patterns for interactions between patient and physicians. All posts in this series will be tagged as healthcare. In the meantime, we are always interested in healthcare-related research data we could use for template development or experiments with our framework. If you have time-oriented data and you are interested in their analysis or visualization, please don’t hesitate to contact us.

If you want to learn more about challenges and opportunities for modern healthcare, you may want to start with a great book Creative Destruction of Medicine by Eric Topol.

Everything is a stream

There is a nice consequence of our fixation on time-oriented data - we can consider everything as a data stream. Any input stream processed by our framework is required to include a time variable, from a simple timestamp field to more complex time structures. We use this variable to organize the data, but also to establish a shared dimension for connecting data elements from different sources. Such an approach gives us the opportunity to operate on multiple streams and implement interesting scenarios with the help of various stream transformations (e.g. time aggregation) and analysis methods.

We define data streams as sequences of events (or state snapshots/changes) that are time ordered, i.e. an event with a timestamp earlier than a previous one is flagged as a potential error. The simplest example of such a data stream is a univariate time series consisting of a timestamp and a data point (a single value), often with some additional requirements, like equal spacing between two consecutive data points. More complex objects in a data stream can include several variables describing an event, often closely related (e.g. time and distance of a running workout, enabling calculation of average speed). In many practical application scenarios, these objects can be models (e.g. snapshots of a system’s attack surface) or collections of unstructured data (e.g. articles published on a subject). 

The beauty of a time variable being present in all our data is that it can become a shared dimension enabling integration of streams from different sources and building a bigger (or just cleaner) picture from seemingly disconnected elements. Let’s illustrate it using a case study based on very simple historical rent data. The input stream includes monthly rent payments and additional variables like location of a rented place and tags indicating lease renewal events. The interim visualization of the rent series is presented in Figure 1, with location and renewal events added as annotations (Point Marker artifacts).

Figure 1: Monthly rent payments with locations and renewal events as point annotations (SVG)

The history of rent payment is an obvious example of time-oriented data, but in this case study more interesting are variables for location and lease renewal events, all connected by time (of monthly granularity). Let’s start with the location variable - we can detect when its value changes and automatically create an annotation (Time Marker artifact) indicating an event of moving between apartments. The lease renewal variable on the other hand can be used to identify critical data points when rent is most likely to change the most, and produce a new series with percentage difference between consecutive renewals. Figure 2 includes the results of these operations.

Figure 2: Monthly payments with markers for moving and differences between lease renewals (SVG)

The requirement of time variable is one of the foundations of the framework  we are creating. Time gives order to a stream of events, but also provides a frame of reference for data analysis. We can use time variables to effectively manage multiple different streams, create useful streams based on multiple input sources or extract key information from streams of unstructured data. Obviously there are challenges related to time granularities, irregular samples or overlapping of intervals and data points. All these problems however can be solved with help of the right technology.

Time aggregation case study

Whenever possible, we will use case studies and experiments to illustrate general concepts and specific features of our framework. We will try to create such examples based on different types of data. In some cases, we will use data streams within the context of a related application domain (e.g. healthcare or security). In other cases, we will use them as generic data samples, without considering their sources or possible goals of analysis. This post belongs to the second group, as we will use a stream of stock prices in order to illustrate some general challenges related to the nature of time dimension and the characteristics of time-oriented data.

In this post we look at time aggregation which is a transformation of data from a higher-frequency time granularity into lower-frequency statistics. This is actually a very common and practical scenario, applicable when we need to calculate aggregated statistics over intervals, for example a sum of weekly sales or a count of server responses per minute. The stock price data we use are also a product of time aggregation - they are last values for daily intervals (specifically adjusted close). The chart with the input stock price stream before aggregation is presented in Figure 1 (with horizontal lines for maximum, average and minimum values across the stream).

Figure 1: Input data stream with prices of MSFT stock since beginning of 2015 (SVG)

Time aggregation is a simple transformation that can be executed with a single command or a few intuitive gestures. It is usually parametrized with an aggregation interval and a summary function (e.g. count, average or standard deviation). The chart in Figure 2 includes our original data stream with a background of new data series created by aggregating the stream by MONTH intervals and with MAX/AVG/MIN functions. It is worth noting that the data series in Figure 2 are using both point (stock prices in navy) and interval (aggregated values in orange) time models. Since the base time granularity is determined by stock price (a day), we are using a stepped line for monthly summaries.

Figure 2: Input data stream aggregated by month with MAX/AVG/MIN summary functions (SVG)

The first visible challenge with such a transformation is a different number of days in a month, and thus a variable duration of aggregation intervals. The mapping between different time granularities can be problematic also on different levels, for example in business scenarios, where analysis is focused on workweeks that are affected by irregular holidays. Analysis scenarios can become more complex if we deal not only with the time dimension but also with locations. Aggregating multiple data streams from different geographical points (e.g. sales across the country) can be affected by time zones, especially if short intervals as selected (e.g. daily statistics).

When using time aggregation, we need to pay attention not only to the time dimension, but also to other data elements, especially during selection of summary functions. Applicability of these functions depends on characteristics of analyzed data, e.g. calculating an average will not work with values of nominal or ordinal scales. But it also depends on context of data. In our case it wouldn’t make sense to calculate sum of prices per an interval, since our data points are related to a state (a price of an asset). However, if exactly the same values (including currency) would be related to events, like sale transactions, calculating average price per interval could be very useful.

We will discuss more detailed challenges related to analysis of time-oriented data in future posts. We will also show how technology can help with solving these problems.