Sharding

The act of sharding or horizontal partitioning is the process of breaking up large tables in a database into smaller chunks called shards. These shards are spread across multiple servers (or multiple database instances). To reference which server has what data, a sharding key is generated, which is a key for a specific server, and it specifies which data is stored where.

Each shard will have the same columns and schema as the original table, but the data stored will be different for each created shard.

These shards are useful as they allow for faster and easier management of (if the database is too large). By spreading the data across multiple servers, the shards can store more information and handle a larger number of queries.

You may have already heard about some popular sharded databases, without knowing that they employ the concept. These databases are sometimes also known as distributed databases. MongoDB and Cassandra are examples of these sharded or distributed databases. Many popular databases are also not sharded, these include SQLite, or Redis.