The short answer
Yes to the flesh — but never the pit, which is toxic.
Why
Peach flesh is a sweet, juicy treat, but the stone (pit) contains cyanide compounds like other stone fruits. Keep your budgie well away from the pit.
How to feed it
Remove the pit completely and offer a small piece of ripe flesh; washed skin is fine.
How much & how often
A small piece occasionally.