Places that elves live(d) in are supposed to be protected from evil, right?
"Protected"... Well... Maybe not.
Don't forget that Rochine and "her children" were in a valley where Elves used to live...
In volume 14, the Skull Knight said that the cave was a good place to hide...
Maybe those places can hide branded persons, not exactly protect them by repelling Specters and the like...
Well, didn't the shadow beast appear to Gatsu in the springs at Godo's, that was enchanted?
The Beast of Darkness may be "materialized" by Specters or "shown" by Incubi, but I think it's still a part of Guts himself... So the cave being "special" or not may not be that important...