Meeting the Challenges of Modern Data Management

Not long ago, the primary challenge of the laboratory industry was simply making the transition from paper records to digital storage.

Not long ago, the primary challenge of the laboratory industry was simply making the transition from paper records to digital storage.

Remember that?

Different factors have made things far more complicated since then – from regulations involving data integrity and security concerns to an exponential explosion of data, rapidly growing file sizes, increasingly complex interfaces and more.

Today’s Laboratory Information Management Systems (LIMS) and Scientific Data Management Systems (SDMS) must evolve to meet these challenges. The good news is your lab can also benefit from significant opportunities for efficiency and profitability by keeping pace with four key trends in data-handling technology.

What’s driving the data management changes?
To understand how the demands of lab data are changing, let’s compare it to something we all can’t live without — our phones.

“My phone?” you say.
Yup – your smart appendage. (Honestly, when was the last time you put it down?)

Nokia 8210Twenty years ago, the Nokia 8210 mobile phone was a marvel. Not only could this little device make phone calls, but it replaced most functions of a personal digital assistant, or “PDA”. (Remember the PalmPilot?) Unfortunately, your data, contacts, and notes were locked inside that phone. Sure, the data was yours – but it was limited by both capacity and accessibility.

Now, what was amazing then seems woefully inadequate just two decades later. If you have a smartphone in your pocket or handbag today, you have more than seven million times the data storage than the Apollo 11 astronauts possessed during the first human moon landing. And that’s just the capacity of your phone. Your ability to access internet-based storage — what we call “the Cloud” — has freed your data from many of its former limitations, including capacity and accessibility.

Four keys to mastering your data
The data throughput challenges of modern laboratories are a lot like your relationship with your phone.

  • You need it (utility).
  • You don’t want your data exposed or hacked (security).
  • You need more storage space (capacity).
  • You need a way to access and analyze the data that’s most relevant to you (analytics).

Here’s how today’s LIMS and SDMS need to work to address all four of your lab’s needs:

Remember when it was enough for your LIMS to store data without being too concerned about how it got there? Those days are long gone. Ever-changing regulations from government agencies are continually updating what is considered an acceptable level of data integrity. These apply not only on LIMS data, but to the source and collection of raw data as well.

To make all of this information accessible and useful, a modern LIMS should provide the functionality to capture both source and raw data. The easiest way to accomplish this is to capture live data directly from networked instruments. LabVantage, for example, is able to interface RS-232 devices with network converters such as balances and scales, live-streaming appliances like freezers, or even complex software and systems that communicate with REST and SOAP web services.

Another option is to capture data from instruments and systems that produce output files in a SDMS. The new LabVantage SDMS, for example, can grab raw data files, parse them to store LIMS-specific data, index them with optional meta data and store everything in a fully-searchable, secure and audited repository.

It’s also important to consider how your data might be shared and reused outside the boundaries of your lab. This can provide insights and value to other business units within your organization, and also has the potential to streamline collaboration with your strategic partners.

While data security is well understood in modern laboratories, it has also evolved significantly as throughput has increased. It’s no longer enough just to know who can enter or approve data. The immense volumes of data modern labs now handle require security to be considered on a much larger scale.

LabVantage, for example, enables multiple security methods for determining which data can be viewed by any given individual and which users are authorized to act upon it.


Your lab also needs to comply with regulations like HIPAA and GDPR. Data privacy features are needed to ensure compliance by default while giving your team fast access to the information they need.

It’s one thing to understand that 1,000 data points might be entered in your LIMS. It’s another thing entirely to have an instrument produce multiple 2GB files a day, each requiring storage in the SDMS. This is several orders of magnitude beyond the typical data volumes your standard LIMS system needed to manage only a few years ago.

Now every lab has capacity challenges. One of the strategies we found to overcome the capacity issue was to use data capture engines to collect and stream data into the repository. Installed both on the application server (or cluster of servers) and remotely on laboratory servers or workstations, this solution enables users to increase the throughput of both large and numerous data files.

While it’s critical to have access to your information, new analytical methods are increasingly needed to handle vast amounts of data. These analytical tools overcome the “needle-in-the-haystack” challenge of finding the information you need. They change that data pool sitting in storage into actionable insights that deliver measurable value to your lab, or interestingly, giving you the ability to predict the future based on historical information.

LabVantage Analytics, for example, uses advanced technologies like Elasticsearch to analyze data from a variety of sources, including LIMS, SDMS, ELN, LES, ERP, MRP, and many others. Included technologies like machine learning (ML) and artificial intelligence (AI) can even be used to launch the analysis of data far beyond business intelligence, with the ability to train the train the system to intelligently look for patterns, behaviors, opportunities, risks, and much more.

data slow snailA Modern LIMS and SDMS Keep You in Control
Working with a LIMS and SDMS system that can handle these four needs isn’t just a useful practice. It’s rapidly becoming essential to any lab that hopes to cope with the rapidly expanding level of data.

In addition, all four capabilities have the potential to offer you competitive advantages over labs that continue to rely on technologies developed for a world with the data needs of ten or twenty years ago.

That would be like trying to stream 4K video from Netflix on a PalmPilot.

Imagine that!