Go for it! Soon, it may be too cold to swim in the ocean, but swimming pools can be analternative - if you want to swim all year long.
I do swim, and my stoma is located below my waistline, therefore it's not visible when only wearing shorts, however, when I do swim, I wear a pair of speedo style trunks, and on top, a pair of patterened lose swimming shorts.
I have only ever used Dansac Invent bags, and when I had my surgery, I was given some mini Dansac bags, for use when swimming, however, I have never needed the mini bags, but I believe that Dansac do still offer a mini bag, and it's possible to obtain free samples of Dansac products.
Also, swimwear styles could be an option for you, have you checked out some of the mens singlet type swim suits that are currently on trend?