Well, that sucks. In trying to play RB3, I found out my TV a) doesn't go loud enough to hear it well over my drums, and b) no matter how much I play with the calibration settings, I can't get it to sync up right. It's fine during the actual song, but anytime there is a fill section, it's about 1/2 second off and completely screws up my timing. I knew there was a reason I gave up on this genre.
If you're only having problems during the freeform fill sections, then you may not have a problem. It can't adjust your freeform inputs in the same way it can adjust the prerecorded audio, so the freeform drum sound bites are going to sound a bit off. If you just listen to the actual analog sound of your drumsticks hitting the drum heads instead of the fill section representations, your drumming should be just as in-time with the music as it is for the non-freeform sections.
I probably worded that poorly, sorry.
tl;dr version: as long as the calibration works for the actual note gems, you're fine. The fill sections don't matter aside from activating star power.