In Plaka and the villages around it you will find most of the attractions of Milos. In Plaka there is a museum with a copy of the Venus of Milo. The old city of Tripiti is worth a visit. It is situated on a slope and it was capital of the island from around 1000 BC to the Byzantine period. The best preserved are the walls and a Roman theater. This place is also a memorial where the Venus of Milos was found. The lower section of the city stood behind the ancient port and has been dominated by a small hill with a temple on it (a small church built with stones of this temple is now in its place).
The theater can be reached if you follow the path down just before the small parking place (walk back a short distance). Most people take the path leading upwards from the parking but that doesn't go anywhere - it will give you a nice walk along the coast with beautiful views though.