Corey Robin has some choice anecdotes from the recent book Speer: Hitler's Architect, by historian Martin Kitchen. The book deflates what's left of Speer's rep as the "good Nazi."
As Minister of Armaments, Speer relied extensively on slave laborers from concentration camps to work in the factories. In 1944, he fell ill for an extended period of time. Himmler seized on the opportunity of Speer’s absence to remove those laborers from the factories -- at the pace of roughly 40,00 per month -- and send them back to the camps. Back at the office several months later, Speer complained about the “kidnapping” of his workers.
Robin also found in Kitchen's book a 1944 quote from German exile journalist Sebastian Haffner:
[Speer] symbolises indeed a type, which among all the belligerents has become increasingly important: the pure technician, the classless, brilliant man without a background, who knows no other goal than to make his way in the world, purely on the basis of his technical and organisational capabilities….This is his age. We can get rid of the Hitlers and the Himmlers, but not the Speers. Whatever may be the fate of each individual man, they will be with us for a long time.
Update: I read Kitchen's book on Speer and highly recommend it. The first part closely follows the architect's day-to-day life as an ambitious Nazi scum, based on Kitchen's careful study of the available documentation, and then the second half explores Speer's self-mythologizing after his release from prison. The first part effectively undercuts the claims in the second, and then the reader gets to enjoy watching the Speer myth begin to fall apart in the last few years of his life, as evidence he assumed was buried or destroyed began to surface.