Serverless Data Engineering
“Serverless” is an architecture which uses an abstraction where the system designer or software developer doesn’t have to worry about the computing infrastructure underneath that is performing the work. As an example, Amazon’s AWS Lambda, cloud-based product could be using Kubernetes or Docker containers to perform all the heavy lifting in terms of computational tasks. Serverless allows for a level of abstraction that forgoes the requirement of having to know the infrastructure on the underlying platform. A software developer instead can focus on writing the Python function that performs some task and is perhaps attached to a “trigger”.
Other AWS server less technologies include AWS Athena, an serverless interactive query service which can monitor an AWS S3 bucket and combine a number of data and do SQL queries on it. In summary, Serverless allows you to simply focus on the business logic of the problem a developer or system designer is trying to solve. Serverless is the highest level of abstraction, and sits above the “Platform as a Service” (PaaS).
Examples of Serverless products available today:
Serverless Functions using AWS Lambda
The whole foundational concept of Serverless computing is that of a “function”. AWS Lambda is an example of what could be termed “Function as a Service” (FaaS). Given Python’s prevalence in the ML community, one could argue that the core of anything that is built using the serverless paradigm is the Python function.
DRAFT — MLOps framework overview chapter v3 ( Bruce Haydon ACM Chapters (Buffalo/New York(NYC)/Toronto)/ HSB C CA 22 /
Copyright © 2020–2022 Bruce Haydon