It is dangerous water but you just want to be prepared (plenty of gas, oil, food, water, working safety equipment, rain gear, towel, change of clothes, etc.), have a really good chart, don't run anywhere real fast that you aren't familiar with and watch the weather.
Never rely on one single source for navigation or seeking help either. GPS and cell phones can fail when you need them for many reasons. Have a compass. Make sure your flares aren't expired.
I've literally broke and lost more stuff on the Bay than everywhere else outside of my two outboard losses - they did not occur there.
The danger in Saginaw Bay is that it is shallow AND big - a higher risk combination. You can bottom out in swells. The winds that push water one way or another can make a safe area unsafe for a time.
Don't run at high speed anywhere inshore you don't already know to be safe. Even if someone else does it nearby. I have 2 busted lower units from 1 experience of watching boats run a small bay and thinking if they can do it, we can do it. Right. But only if you know there is a small opening blasted through a rock bar AND you know exactly where the opening is!!!

We could have saved some equipment back then if we had all the satellite imagery and Internet information available now, along with GPS instead of nothing and then Loran for a while.
On calmer days, Saginaw Bay in the fall can be awesome! Like all fall fishing, the bass may move a lot and be spotty. But find some of those spots and it will be worth it. You don't have to go to the Charities either. Also like much fall fishing, bigger bass become more available shallower more often in early to mid-fall.
I can't speak personally for the specific areas you mention since I'm not sure I ever fished around Port Austin and I never fall fished near Caseville. Wildfowl Bay and nearby islands however can be real good in the fall though I haven't been there in quite a few years and it is shallower now. I do see a few fishing reports that indicate the fishing can still be good.
If you decide to run to the Charities from that side, see if you can find someone who has done it recently (I haven't) since that side is more tricky now than it was when the water was higher.
I've always wanted to fish more around Port Austin but never found the chance with so many options. It would be interesting to go exploring slowly under favorable conditions. It is very complex looking from Pt Austin all the way to Grindstone. Haven't seen or heard many recent fishing reports from that area.