F1 to H4 COS applied but F1 expiring soon

Hi, my h1b was not picked up and in anticipation of that i applied for F1 to h4 COS. But my F1 OPT is expiring soon. Can I continue to stay while my COS is in progress? Or do I need to leave the country because I will be out of status once my OPT expires and H4 approval notice is not received ?

Yes.

While your change of status application is processed, you are considered to be in ‘period of authorized stay’ status, so no issues.