The First Doctor’s new companions Ben and Polly arrive with him in the TARDIS on the coast of seventeenth century Cornwall. They meet a worried churchwarden named Joseph Longfoot, who lives in fear of "Avery's boys" and, in thanks for the Doctor’s kindness in relocating a dislocated finger, imparts a cryptic message he calls "Deadman's secret key": "Smallwood, Ringwood, Gurney". While the time travellers head off to settle at the local inn, Longfoot has another visitor. This one is Cherub, Longfoot's former shipmate under pirate Captain Avery on the Black Albatross. Cherub and his master, Samuel Pike, who captains the Albatross since Avery died, want to recover Avery's accursed gold. Pike is convinced that Longfoot has the treasure or knows where it is hidden. When the churchwarden does not co-operate, Cherub kills him – but not before revealing he saw the three travellers who visited Longfoot earlier.
Hours later the discovery of the church warden’s body leads the locals to suspect the three strangers at the inn. The local Squire is called to intervene and adjudicate, and ends up charging Ben and Polly with the murder. Employing trickery to obtain their freedom, they split up. Ben hides at the church until Josiah Blake, a revenue man tracking the local smugglers, disturbs him.
In the meantime Cherub and some pirates have kidnapped the Doctor and taken him to the Albatross. The Doctor attempts to bargain with Pike, and finds himself kept aboard ship while the captain goes ashore. Pike decides to try and make an alliance with the Squire as well to protect himself while he searches for Avery’s treasure. The greedy Squire is the organiser of the local smuggling ring and offers to cut Pike and his pirates in. They are interrupted by Polly, who has come to implore the Squire to help her find the Doctor and is shocked to see him in the company of the kidnapping pirate Cherub.
Pike, Cherub and the Squire take Polly to the church, where they attempt to convince Blake that Ben and Polly are the true smugglers. Knowing the truth but lacking the manpower to arrest the pirates, Blake pretends to arrest Ben and Polly. The Doctor has meanwhile escaped from the ship and meets up with his friends in the churchyard. Blake works out a smuggling drop is due soon and heads off for more revenue men to break the smuggling ring.
The smuggling alliance has by now fallen apart: the Squire has realised he is dealing with a ruthless pirate who will not honour any bargains with him while Cherub has decided to locate Avery’s gold for himself. The Squire too sets off to find the gold, as do the time travellers since the Doctor is convinced the rhyme of the churchwarden is the key. He works out the names Ringwood, Smallbeer, and Gurney pertain to graves in the crypt but before he can find the treasure the other seekers arrive. Cherub wounds the Squire, and then forces the Doctor to confess the rhyme. Cherub concludes that Deadman too is a name of one of Avery’s former pirates, but is slain by a vengeful Pike, who now threatens to pillage the entire village in his search for Avery’s treasure. The Doctor bargains with Pike for the lives of the villagers if he shows him the treasure and, with this agreed, they find the gold at the intersection of the four graves.
No sooner does Pike have the treasure than Blake and an armed patrol of revenue men arrive. Aided by the injured Squire – who repents of his sins – Blake kills Pike, and the pirate force is routed. As the battle dies down, the Doctor and his companions slip away to the TARDIS.