There is a famous Cassandra PPT at SigMod 2008. In one slide, they presented several interesting lessons:

1. Add fancy features only when absolutely required
2. Many types of failures are possible
3. Big systems need proper system-level monitoring
4. Value simple designs

These lessons are interestingly useful.