Preheat the oven to 225 degrees F.
In a shallow pan combine the eggs, heavy cream, sugar, whole milk, vanilla extract, vanilla bean paste, cinnamon, nutmeg, and a pinch of salt. Whisk well.
Cut the brioche or challah bread into 3/4 inch thick slices. Place the sliced bread in the mixture, allow to soak for at least 2 minutes per side. (Push down on the bread if necessary. Add more time if needed. The bread should be saturated, but not soggy).
Heat 1 tablespoon of butter and 1 tablespoon of oil into a large skillet over medium-high heat. When the pan begins to sizzle and foam it's go time!
Carefully lift the bread slices out of the custard mixture. Letting the excess drip back into the shallow pan. Cook in the skillet until golden brown and the center springs back when pressed. Approximately 2-3 minutes per side.
Transfer the cooked pain perdu to the oven until ready to serve.
Continue cooking the remaining slices of bread with butter and oil. Serve hot.