Castleby David Macaulay is supposedly for kids - but pretty much every gamer of any age I have ever known has a copy on their shelves. It's a grand book full of marvelous illustrations, and what it does best is explain how and why castles were built the way they were. From the perspective of gamers and gaming, this book is a treasure trove of little design details that can be added to castles and dungeons and fortifications, to add background detail and a few little mysteries, e.g. why are there holes tracing a shallow angle along the castle wall? To provide anchor points for posts for a ramp to transport rocks.
There's a larger question of why would castles in a fantasy setting with flying monsters and destructive magic be built in any way similar to that of our ordinary Middle Ages. It's up to the referee to determine the answer to that question. If you want a decent base to work from in castle design, however, this is a good book. You can get it here.