On a ribbon, adding more entry to QAT will squeeze caption area to a point before it puts additional QAT in what I call "extended QAT area". Is there a way to set that point (minimum caption area)?
No, there is no way to set a minimum for this. The area available for the QAT is dynamic and depends on a few different factors, such as the ApplicationMenu visibility, the caption size and available area, and whether any contextual tab group headers are present in the caption.