So after a couple years of on and off pain I finally got in to a doctor and got diagnosed with gallstones.
A bit to my surprise it seems like the only real recommended solution is surgery to remove the gall bladder. I've talked with my doctor and had a surgery consultation and both doctors have basically been treating this as not a big deal. The surgeon in particular said not to worry too much about long term consequences I've read about and that I shouldn't have to really change anything (diet, etc) after the surgery.
I'm not a fan of permanently removing any organ but since that seems to be the only recommendation I'm willing to go with it. They would do it all with robotic laparoscopy using the DaVinci system, which seems minimally invasive with 1 incision in the belly button at least.
I think several people around these parts have had similar problems and/or gone through the surgery, so I guess I'm just looking for any advice or feedback or warnings.