Along with the emergence of generative artificial intelligence (GenAI) has come a surging demand for data and data center capacity to host growing AI workloads. And more and more organizations find themselves in the race to build the infrastructure and data center capacity capable of supporting the current and future use of AI and machine learning (ML).
For finance functions, high-quality, well-organized, and trustworthy data is essential in the development of effective AI-driven operating models. And while speed is a big factor, trust and safety are even greater concerns in a technology environment where there are few guardrails for AI risk management. Just think of the internet with no rules around e-commerce, privacy, or business and personal safety.
So where does a management team get a handle on the critical issues around an AI approach that is both highly efficient from an operations standpoint and optimized for risk management? We believe in this case that the past can be the prologue: consider a principle known as the “medallion architecture” — a commonly used industry framework for managing large-scale data processing in cloud environments. For many of the same reasons it works so well there, we also find it applies well to data engineering. It’s particularly well suited for tax and finance operations, where data is one of the most valuable assets and for which flexible, scalable, and reliable data management is essential for regulatory compliance speed and accuracy.
A Layered Approach
The reality is that data and AI are essentially inseparable in our new digital era. While data has existed for a long time without AI, AI does not exist without data. By extension, a solid data strategy is required for achieving meaningful returns on AI value, and medallion architecture is a highly effective data management tool that helps get the most out of an organization’s AI investment. As a data engineering model, it organizes information into three distinct tiers of bronze, silver and gold “medals.” Each layer has a specific role in the data pipeline, designed to facilitate clean, accurate and optimized dataflows for downstream processes:
Bronze: This is the raw data layer. The data is ingested from various sources, including structured, semi-structured and unstructured formats. At this stage, the data is stored in its original form without any significant transformation. This serves as a robust foundation, providing a full audit trail and allowing businesses to revisit the raw data for future needs.
Silver: In this intermediate stage, data from the bronze layer is cleaned, filtered and structured into a more usable format. This involves applying necessary transformations, removing duplicates, filling in missing data and applying quality checks. The silver layer acts as a reliable data set that can be used for analysis, but it’s still not fully optimized.
Gold: This is the final stage of the data pipeline where the silver data is further refined, aggregated and structured for direct consumption by analytics tools, dashboards and decision-making systems. The gold layer delivers highly curated, trusted data that’s ready for use in real-time reporting and advanced analytics.
Applying the Benefits of Medallion Architecture in the Finance Sector
For financial institutions, data management needs are highly complex. Banks, trading firms and FinTech companies process enormous amounts of data daily, with requirements for accuracy, speed and regulatory compliance. Medallion architecture addresses the following needs.
1. Improved data quality and governance. Financial institutions must ensure data accuracy and completeness in alignment with strict regulatory requirements, such as Basel III, the Sarbanes-Oxley Act (SOX) and MiFID II. The multilayered features of medallion architecture support data quality checks that can be applied at each stage. By moving from the bronze to gold layer, data undergoes multiple transformations and validations, improving accuracy and reducing errors. It also facilitates better data governance and traceability, allowing for easier auditing and compliance reporting.
2. Scalability for large data volumes. The financial sector often deals with massive data sets — from transaction histories and market feeds to customer data. The layered approach makes it easier to scale these data pipelines. Since the raw data in the bronze layer is stored in its original form, it can handle the ingestion of high volumes of data without requiring immediate transformations. As data moves to the silver and gold layers, the architecture supports scalable processing frameworks that enable financial institutions to efficiently process large data sets.
3. Faster time to insights. In fast-paced financial markets, speed is essential. Trading firms, for example, need real-time data to make decisions on market movements. The medallion structure allows financial institutions to separate raw data ingestion from data analytics. Analysts can start working on silver and gold layers for immediate insights, while engineers refine and clean the data in the background. This results in quicker access to actionable insights, essential for high-frequency trading or real-time fraud detection.
4. Flexibility and agility. Medallion architecture offers flexibility in handling diverse data sources and types — an essential feature in the financial industry, where data comes from numerous channels. The bronze layer’s ability to store raw data in its native form makes it easy to adapt to new data types or sources without needing immediate transformations, while the silver and gold layers can be adjusted to reflect new business requirements, market conditions or regulatory changes.
5. Cost efficiency. Processing large volumes of financial data is expensive. Separating the raw data from the processed data helps reduce unnecessary data transformations and storage costs. Financial institutions can optimize their compute resources by running complex transformations only when needed, thus lowering operational costs.
6. Enhanced security and risk management. Raw data in the bronze layer can be heavily restricted, with only authorized personnel able to access it, while the curated gold layer can be more widely available for analysis. This segmentation of data access allows for tighter security controls and reduces the attack surface.
7. Advanced analytics and machine learning. From algorithmic trading to fraud detection and credit risk analysis, ML and AI are very important to the financial industry, and this approach facilitates advanced analytics by providing high-quality, structured data in the gold layer. Additionally, having access to both silver and bronze layers enable data scientists to work with both historical and refined data, both of which are essential for building accurate predictive models.
Medallion architecture is an effective framework for financial sector data management and processing in the digital era. Its layered approach offers financial institutions the capability to handle vast volumes of data efficiently, while providing data quality, compliance and scalability. Using this layered approach, financial firms gain better control over their data pipelines, reduce costs and drive innovation through advanced analytics. As data management plays an increasingly crucial role in contemporary business, this framework helps position financial firms for success in a data-driven world.