I’d design a distributed training system using model parallelism (e.g., tensor/model sharding) and data parallelism across GPUs/TPUs. I’d optimize with mixed precision, gradient checkpointing, and efficient data pipelines, leveraging cloud-based autoscaling for cost-effective scalability and fault tolerance.