I used to have a 10' stacer starfish (3.1m). First trip out from warneet in fog as thick as you like and we wound up around crawfish rock in perfectly calm waters and a false sense of security set in. A few trips later and we were out of quail bank when the wind started to get up a bit, from a slight breeze to about 15 knots in 15 minutes and we had to punch into the wind and waves to get back in, we were taking on lots of water over the bow and bailing as we tried to come back at a speed not much over 5kph (top speed was 10kph with a 4 HP). Thinking I had learned my lesson I only took the boat out in more sheltered waters, one day at shallow inlet the girlfriend and I were caught out again by a fast rising wind that agin caused us to punch into it to come in. The gf was white knuckled the whole way, wet and cold. She never came out again in that boat.
My only point is that a small boat has limitations. The right weather can very quickly turn into the wrong weather. Yes I would take a small boat out again into the bay but I wouldn't head too far out and I would have an incredibly keen weather eye. I would head home at the first hint of the weather turning anything but perfect.
Until you have experienced the fear of being in a small boat in nasty conditions it's hard to comprehend how things could get so dangerous when you're "just in the bay"!
I now have a much bigger boat and know that it can handle pretty bad weather in the bay so am more relaxed. My wits are on heightened alert when outside the bay. My mate has a 4.2m tinny that we regularly go out in WP. It feels much bigger and safer than my old starfish but I still wouldn't like to get caught out in 20knot winds.
If you want to take your kids, girlfriends, whoever, out on the boat be sure you keep very safe and have an eye to the weather at all times and head in as soon as things look like they might sour, that's if you ever want them to go out on the boat again!