Turning unstructured content into clinical insight

Scientific publishing sits on a goldmine of clinical insight, but most of it is trapped in unstructured text. This publisher knew their journal back catalogue was hugely valuable to pharmaceutical clients—yet turning that content into usable, analysable data was slow, manual, and difficult to scale.

Pharma customers were asking sharper questions about clinical trials, real-world evidence and therapy areas such as oncology and respiratory disease. The publisher wanted to answer those questions with high-quality, machine-readable data products, not just PDFs, and saw an opportunity to create a new revenue stream in the process.

When I joined as an Agile Delivery consultant, the strategic direction was clear: build on the existing Entellect data platform and create a robust AI big data pipeline that could fetch, organise, structure and search journal article content at scale. What they needed was someone to turn that ambition into a working product and a sustainable way of delivering change.

I led the development of a new pipeline built around more than ten reusable microservices. These services ingested unstructured articles, applied enrichment and identifiers, grouped content into subject domains such as oncology and respiratory, and exposed it through search and discovery capabilities that data scientists could actually use.

By moving from raw content to structured, subject-based data offerings, we made it possible for life science clients to run deeper analysis, improve the ROI of their clinical trials, and buy exactly the slices of data they needed—rather than wading through entire journal archives.

The technology only worked because we invested just as much in people and ways of working. I assembled and coached a cross-functional team of developers, DevOps engineers, QA and a product owner—over ten specialists across onshore and nearshore locations—and introduced agile and lean practices tailored to their context.

We started with simple, timeboxed experiments on the Entellect platform, using short sprints and regular showcases to keep stakeholders close to the work. This helped the team build confidence with microservices, AI-driven enrichment and new data models without getting paralysed by the scale of the back catalogue.

As delivery matured, we tightened the feedback loop with commercial and product stakeholders, so everyone could see how technical decisions affected the eventual Text Journal product. That collaboration ensured we were building services that were not only elegant technically, but directly aligned to how pharma clients wanted to buy and use data.

Over the course of the engagement, agile best practice became part of the publisher's product culture rather than a side project. The microservices were designed to be reusable, so future products could be accelerated instead of starting from scratch—turning this initial investment into a genuine platform play.