They are the documents left behind by the people who actually lived in a particular time and place (as opposed to stuff written by historians). They're often a window into weird corners of thought and fact that you might never see in a secondary text.
For instance, there was this nice rabbi, Benjamin of Tudela...
Benjamin traveled from Northern Spain all the way to Persia and back about a hundred years before Marco Polo did his thing, and he left a record of the communities he visited. (This was not uncommon for travelers in the Middle Ages; their descriptions of other lands and carrying of news and letters was often the currency they used to pay for their food and lodging.)
So...if you read Benjamin's account, you eventually arrive here:
From thence it is two day's journey to Gebal, at the foot of (Mount) Lebanon. In that neighborhood dwell a people called Al-Hashishim. They do not believe in the religion of Islam, but follow one of their own folk, whom they regard as their prophet, and all that he tells them to do they carry out, whether for death or for life. They call him the Sheik Al Hashishim, and he is known as their Elder. At his word these mountaineers go out and come in. Their principal seat is Kadmus, which is Kedemoth in the land of Sihon. They are faithful to each other, but a source of terror to their neighbours, killing even kings at the cost of their own lives. The extent of their land is eight days' journey. And they are at war with the sons of Edom who are called the Franks, and with the ruler of Tripolis, which is Tarabulus el Sham.
Benjamin was talking about the very real and historical Order of the Assassins (which will show up in the sequel to The Excalibur Trick.) As it happens, he was wrong about them not being Muslim; they just weren't Sunni Muslim. (They were Nizari Isma'ili Shi'a Muslim, for anyone interested.) But this is like...straight from the horse's mouth. A man a thousand years gone speaking to you...
Gives me chills.