You don't particularly like the idea of crossing any ghosts that might have decided to haunt the inn. You stay the night, but you check out first thing in the morning.
You head on to the castle ruins.