There are several things that can cause this. It sounds like you have eliminated most of the configuration issues. This may or may not be your problem, but I learned the hard way you have to be anything but late in completing your loops. If you have the Beat Buddy connected by MIDI cable to the Infinity and you close your loop just a tad late, the Infinity Looper will try to adjust the speed of the loop (speed it up) during playback to make it complete in time for the next “cycle” of the loop. This results in a playback that is normal speed for the majority of the loop period but then it will do a quick chirping sound when it briefly speeds up the loop in order to complete it in time to keep it in sync with the BB.
If you think this might be your problem, here is an experiment to try. Record a short loop and make sure you complete the loop early. Try completing the loop early by as much as a full count or more. If the above explanation is your problem, then this loop will never speed up. In fact, what you will hear it stop at the end of the loop. You should hear silence while the loop waits for the clock signal from the BB before it starts the next cycle. You should never hear the loop change in speed, just a gap at the end each time it stops playing while it waits for the last beat of the measure.
If you try this and you still hear the speed of the loop changing, then you have a parameter set wrong in either the Infinity Looper or the BB. Most likely in the Infinity. Or, you have a hardware/cabling issue.
I was having this problem periodically and I traced it to three things that must be set correctly. I had to turn MIDI notes off in the Infinity, I had to make sure that the time signature for each song in the Infinity was set the same as the Time Signature in the BB and I had to make sure that I was closing each loop I recorded as close to beat four (in a 4/4 time sig song) as I could get without going a millisecond past beat four into beat one of the next measure.
Hope this helps. It can be a bit of pain to get the two units synced, but once you do, it is a blast the way the two work so well together.