The short answer
Yes — celery is safe, but slice it thin to remove the strings, and offer the leaves too.
Why
Celery is hydrating and harmless, but its tough stringy fibres can be a choking hazard for a tiny beak. The leaves are actually more nutritious than the stalk.
How to feed it
Slice the stalk thinly across the grain to cut the strings, or simply offer the leaves.
How much & how often
Frequent is fine.