Meta Interview Question

Design a distributed web crawler that avoids revisiting duplicate pages