Introduction

Design of a storage layer is a challenging task. Fortunately data management is a mature discipline with a variety of solutions available to accommodate various use cases. Unfortunately there is no single solution that can fit all use cases well, hence we need to understand why certain solutions are better than others in dealing with particular data management problems.

The two types of storage layers that can be discussed in this question are:

• Ephemeral: meaning that data will not be available in case of system restart. Data that is stored in an ephemeral layer is the data that an organisation can afford to lose - typically cache for the data stored in a persistent storage layer.
• Persistent: meaning that the data must always be available or recoverable. All data that has a value for an organisation is stored in a persistent layer.

By the end of this tutorial you will learn about the following:

• The challenges of storage layer design and how to address them.
• The types of workloads and arguably good fit solutions for those workloads.