Challenges in Data Partitioning

Some considerations when deciding data partitioning schemes:

• Hotspots and skew when some partitions have excessive load need intelligent assignment of records to partitions.

• Rebalancing existing data when nodes are added/removed requires redistributing data.

• Joins and data consistency become harder across shards.

• Physical network partitions can isolate data partitions, so fault tolerance must be built-in.