Preheat the oven to 350 degrees. Line 2 baking sheets with silicone mats or parchment paper.
In a medium sized mixing bowl, combine the flour, baking soda, baking powder, and salt. Set aside.
In a large bowl or the bowl of an electric mixer, beat together the butter and peanut butter on high speed until light and fluffy, about two minutes. Add both sugars and beat until smooth. Add the egg and egg yolk, beating well after each addition. Beat in the milk and vanilla extract.
With the mixer on low speed, slowly add the flour mixture to the peanut butter mixture. Once it's mostly combined, turn the mixer up to medium speed and beat until cohesive.
Scoop rounded Tablespoons of dough onto the prepared pans. Place a marshmallow in the center of each one, using your palm to press it into a thick disc.
Bake for 10-11 minutes, or until the edges are set and the marshmallows is faintly browned. Whisk together the sugar and cinnamon if using and sprinkle it lightly over the top of the cookies. Allow them to set up on the pan for at least 10 minutes, or until they're sturdy enough to be moved to a cooling rack to cool completely.