Tradition gives Henry Clifford credit for building Barden Tower in the fifteenth century. It is said that he preferred to live near the River Wharfe to his family seat at Skipton Castle and he was known as the Shepherd Lord because of these pastoral leanings. Constructed for the same reasons as Hellifield Peel it is of a similar size.

In 1658 Lady Anne Clifford restored the building as it had decayed to a ruin and the result is a complicated blend of fifteenth and seventeenth century architecture. It is difficult to discern between the different builds, however a fifteenth century fireplace has been cut in half by a seventeenth century wall.

At one time the tower had a courtyard, surrounded by a wall with two entrances. One of these gateway arches still survives intact; of the other only the jambs remain.