Challenges

• Ensuring correct operation - We need to ensure that the storage layer deals correctly with concurrent operations, and takes the necessary steps to ensure that the data is consistent and durable during restart scenarios.
• Performance - Later in the tutorial you will learn about key performance indicators, at this stage it is enough to understand that storage layer performance depends on the types of data stores that are being used and the configuration that is being applied.
• Accessing data - Often our use cases would involve requirements for retrieving data that matches specific characteristics. For example: a client would like to know how much time on average customers spend using a platform, hence we need a way to provide such functionality.
• Dealing with large data volumes - Today businesses deal with hundreds of terabytes of data, which can not be handled by a single server. Large scale storage systems are available for such use cases, however these systems require compromises. Such data stores use the concept of sharding to distribute data across a number of servers.
• Ensuring business continuity - What happens to our storage layer if the server that is hosting the primary data store breaks down? What happens if the data centre gets flooded? When we design a storage layer we need to make sure that our data can be accessed in case of disaster.