Member-only story
Demystifying the Modern AI Stack
The rise of artificial intelligence (AI) means that it is more important than ever for developers and engineers to deploy AI projects more quickly and at greater scale across an organization. At the same time, there has been a boom of AI tools and services designed for different purposes, which has made it challenging to evaluate all of them in the quickly evolving environment.
To achieve this fast and efficient deployment of AI projects, it is crucial for your organization to possess the “modern AI stack,” which is a collection of tools, services, and processes implemented with MLOps practices.
The modern AI stack enables developers and operations teams to construct machine learning (ML) pipelines efficiently, improving utilization, end-user experience, team collaboration, maintenance activities, and more.
>>> Before reading this piece on the modern AI stack, make sure to check out my previous article on “What is MLOps?.”
The modern AI stack can first be broken down into three distinct phases:
- Data Management
- Model Training/Evaluation
- Deployment
Let’s take a look at each one of these phases and what they entail.
Phase 1: Data Management
The first phase of the modern AI stack is data management, which includes data gathering, data transformation, data processing, data versioning, and data monitoring.
Data Gathering
When it comes to data gathering, which is key to having usable data, the process often relies on third-party tools and services that can be integrated into your company’s internal tools.
There are a few key components to data gathering:
- Data Collection: Involves web scraping, sifting through databases, and complex queries for extraction. Datasets can also be directly sourced from various third-party services and sites (OpenML, Kaggle, Amazon Datasets).
- Data Labeling: After data have been collected, they must be…
