There are always reasons for asset managers and owners to modernize their technology stacks. But for the growing number of market participants investing in private markets, the situation is becoming urgent. And it is all because of data.

According to BNY’s Future of Asset Management report, asset owners and managers identified reliance on data and analytics as the top trend in asset management for the next three to five years.

Francisco Ceballos, product manager for Platform Experience and Core Capabilities, said the drivers of this trend can be divided into three categories: the pursuit of greater operational efficiency, the need for data-driven decision-making and the demand by investors and regulators for greater transparency and increased reporting.

“Firms spend a significant amount of time gathering, cleaning and normalizing data before it can be used,” says Ceballos. “This often involves working across siloed and fragmented legacy systems.”

Despite tech stack inefficiencies, firms are trying to maintain momentum, especially those investing in alternative assets. But with capital allocations to alternatives rising, the growth in private markets is straining traditional investment workflows, which suffer from limited disclosures, unstructured reporting formats and subjective performance measurement, says Ceballos.

“Private market investors expect more granular data and detailed analysis, not just a simple PDF report. They need this for due diligence, to validate their investment thesis, understand borrower credit, analyze loan performance and so on,” he says.

But to meet rising demand, asset managers and owners need modern systems that work in harmony so they can centralize and validate data. “Relying on spreadsheets and emails is no longer enough,” says Ceballos.

The Challenges of Data Management in the Private Market

One of the biggest challenges private market asset managers and owners face is the complexity of the valuation process, says Ceballos. The illiquid nature of private assets and the absence of publicly traded data make it necessary to use third-party verification. Firms also need a centralized and holistic view of their portfolios, known as the golden source of truth.

However, there are several barriers — technical, structural and cultural — slowing adoption of centralized data platforms, says Ceballos. The most noticeable impediment is the inconsistent quality and availability of the data, much of which is unstructured. This difficulty is exacerbated by disparate data-capturing processes across different departments.

Firms mired in fragmented, legacy and outdated systems face an uphill battle to integrate new technology. “Plus, you have to consider the additional expense of recruiting new staff, changes to contracts with existing vendors and the need to train existing staff on the new systems and processes, as well as a resistance to change,” adds Ceballos.

Unlocking the Potential of Unstructured Data

And then there is the inconsistency of the data itself. According to MIT, unstructured data accounts for around 80% of all enterprise data.¹ Unstructured data is especially common in private markets.

“In the absence of publicly traded data, valuable insights are often trapped inside PDFs, emails and spreadsheets, all of which are notoriously hard to process,” says Ceballos. “As a result, firms can end up spending their time in search of data in the rough.”

Fortunately, there is technology widely available that can help. Natural language processing (NLP), machine learning (ML) and generative AI (gen AI) can identify anomalies in reports. And large language models (LLMs) and gen AI can be used by firms to generate content from unstructured data, for example, by analyzing regulatory filings and producing reports.

Consequently, firms can benefit from better data-driven investment insights and enhanced risk management. In BNY’s survey, 72% of respondents said their organization is looking to expand the use of data analytics and insights and digital capabilities/technologies for front-office risk management.

“Real-time, proactive compliance allows portfolio managers to identify issues before they arise and fulfil compliance at speed,” Ceballos says. “By digitizing the regulatory filing process, AI and ML can be applied much more easily, enabling the technology to detect useful patterns and maintain a digital audit trail.”

The Role of Data Governance

But when applying AI to data management, firms should be cognizant of governance and regulatory issues, says Ceballos. “While some regulatory frameworks have emerged, such as the EU’s AI Act, it is an area that is still developing. Firms should set standards for data integrity.”

There also needs to be strong data governance around the use of AI for data collection, storage and distribution. “You need to explain the logic and transparency in terms of how it is being used. And you need to assign roles, monitor AI risk for bias and stay up to date with regulations.”

In addition, companies need to continuously check the output of their AI tools and refine the model,” says Ceballos. “Increasing the diversity of the model’s training data will also be critical to help the model learn from a broader range of examples, mitigate biases, improve performance and enhance real-world relevance.”

Every Cloud Has a Silver Lining

The development of cloud computing, which has helped to facilitate the explosion in data and cloud technology, will also be integral to managing this data, says Ceballos. “Cloud-native pipelines help firms quickly ingest information so that transactions like trades, prices and reference data arrive in near real-time, keeping everyone working off the latest information.”

“The platforms typically have centralized data feeds, while the use of API-first interfaces allows multiple users to interact and make that data available to clients,” says Ceballos.

Quality controls such as validation, schema checks and anomaly detection are built into automated pipelines, so every new data source or update passes through the same rigorous tests, says Ceballos. “Meanwhile, metadata catalogues, lineage tracking and real-time health dashboards give full visibility into every ingestion, transformation and delivery step. Access controls and audit logs ensure data is trustworthy, compliant and easy to govern.”

The Path Forward

So how will things develop over the next three to five years? Predicting the future of technology is a dangerous game, says Ceballos. Who, for example, foresaw the rise of AI just a few short years ago.

Some market shifts are not only inevitable, but also critical to the development of private market data management. “I firmly believe that the shift to cloud-native platforms is crucial to support the computational demands of AI and to manage the ever-increasing volume of data,” says Ceballos.

Outside of AI, predictive modelling is one likely development, while distributed ledger technology (DLT) will continue to be important for the development of tokenization — which could in turn bring much-needed liquidity to private markets, making the asset class even more accessible to investors.

And as more participants enter private markets, the demand for transparency will increase, as will regulatory scrutiny, says Ceballos. “This will mean greater focus on the transparency and potential bias of algorithms, as well as privacy issues in connection with the EU’s General Data Protection Regulation (GDPR) and a host of other regulatory challenges.”

Industry trends are likely to be amplified by another important market shift that is currently underway — the integration of environmental, social and governance (ESG) data as firms grapple with the various disclosure requirements under sustainability rules.

The final consideration, says Ceballos, is less about technology or data and more about a talent shift. The days of manual data collection are likely numbered as roles increasingly involve AI supervision and management.”

While the focus over the past decade was on collecting and cleaning data, organizing this data properly will be the paramount task in the next ten years. As private markets scale up, technology will not be the sole differentiator. Instead, the people and roles that supervise, govern and apply AI across centralized platforms will assume an enhanced role. The manual work that once dominated our workflows is giving way to higher value responsibilities: stewardship, product management and real-time compliance oversight that blends quantitative rigor with domain judgement and expertise.

Related Platforms