While Hermitage Abbey and Church were created for the book, Hermitage Castle is a very real place. Considered by many to be "The Most Haunted Castle in Scotland", the Castle is a semi-ruin today.
Currently under the care of Historic Scotland, Hermitage has an incredibly creepy history.
Also known as the "Strength of Liddesdale" and the guardhouse of the "bloodiest valley in Britain," Hermitage was supposedly build by a man named Nicholas de Soulis around 1240. It stayed in the de Soulis family until 1320 when his Nicholas' descendant William was forced to give it up. Why? Well, apparently William was suspected of witchcraft, and the story is that he was involved in a plot to attempt to kill Robert the Bruce. There's a folk tale that William de Soulis eventually met his end boiled to death in molten lead by his own tenants, who claimed that de Soulis was also kidnapping young people in the area and sacrificing them in the castle. In reality, de Soulis died in Dumbarton Castle. De Soulis is also supposed to have had a particularly nasty familiar, who is rumored to roam the halls today.
These aren't the only ghosts in the Castle. Mary, Queen of Scots, also has ties to this place; her ghost is said to wander here as well. There are also stories of prisoners being intentionally starved to death in the dungeon, and it's said that you can still hear their desperate cries echo among the stones at Hermitage.
In addition to the de Soulis family, other clans that have owned the castle are Douglases, the Hepburns, and yes, the Scotts. in 1594, castle was granted by the Crown to Sir Walter Scott of Buccleuch, a notorious Border Reiver who would later gain additional fame by assaulting Carlisle Castle in England to free fellow Reiver Kinmont Willie Armstrong.
The Castle's notorious history and links to the Stuart and Scott families made it the perfect place to set multiple scenes during the series. Hermitage Castle is mentioned multiple times in Wyldwood, and is a major location in its sequel, Circle of Three.
Want to see Hermitage for yourself? There's a great video (not narrated, but has lovely Celtic music in the background) on Youtube that I really enjoyed: