Preheat the oven to 350 degrees F. lightly Butter a 13x9 inch baking pan.
Evenly sprinkle some of the brown sugar over the bottom of the pan.
In a small pot, gently simmer the cream, milk, rosemary, garlic, salt and black pepper.
Slice the potatoes as thinly as possible.
Place a shingled layer of potatoes on top of the brown sugar. It doesn't need to be perfect or fancy.
Sprinkle some of the Parmesan and cheddar over the potatoes.
Place another layer of potatoes on top and sprinkle with more brown sugar, Parmesan and cheddar.Repeat these layers about 3-4 times or until you run out of potato slices.
Carefully pour the cream mixture over the potatoes and let sit for 5 minutes.
Use a wooden spoon to gently move the potatoes around making sure the liquid is getting into all of the layers.
Place the pan into the oven and bake until golden on top.
Check them after 30 minutes and gently pat the top of the potatoes down with the spoon to keep the top layer covered in liquid as much as possible.
Let sit for 10 minutes before serving.