You have chosen a difficult problem.  You might even have to go to an extended Kalman filter.  But before you do that, you might try some tricks to improve the signal going into your smoother.  What is you phase detector?  You might try a quadrature detector, mixing against a cos and -sin term instead of with just a reference.  But it sounds like you essentially have tat if you mentioned a Costa's style loop.  The key is to get more performance out of your phase detector.  Possibly making a non linear selector.  Try 2, 3 or even more phase/frequency combinations as your reference signal.  You can run multiple PLL's, and choose the best one.  Or even better, choose the results of the best one to jump the poorer ones.  This contest method can triple your work, but it is a regular structure so it is not complicaed, and you will not have to do any Kalman tuning and limit control.  The Kalman limit control is always the tricky part.
Good Luck!
John Solar

jsolar,


Phase detector is just Err = Q*sgn(I) - I*sgn(Q) which is the maximum likelihood solution for QPSK. The fundamental problem is that you would like to average to get a lot of noise rejection into the loop.  The issue is  averaging implies delay, which is always bad in a feedback loop.  Smoothers tend to average the PLL output {NCO output to rotate the signal after the loop filter (which follows the phase detector which generates the error signal)}.  The ML solutions tend to try brute-force approaches {try every freq/phase combinations and use logic to select the "correct" one to use as the error signal into a loop designed to track particular Doppler characteristics (order of the response)}.  Using decoded data in our case has way to much delay to train the loop.

 
Turns out the extended kalman isn't that great.  Solved problem by recycling training symbols from a preamble by using a delay line to effectively recycle preamble data after the nondelayed data is used to estimate timing, phase,frequency to steer the loop in the right direction (no costas loss due to known data), and then since the phase and freq are very good at the start of the random data portion the loop can handle it.

The smoothers I discovered don't help with loss-of-lock problems, they just improve jitter and are actually applied in a savitsky-golay configuration to the NCO output controlling th phase rotator which is smoothed and then a secondary path derotates on the smoothed NCO output with appropriate delay to account for the smoother observation delay.  Raising the order of the loop from 2 to 3 also helps quite a bit under high Doppler rates.  

I find the Kalman type solutions will just converge to the best PLL loop filter coefficients--which are a function of SNR only.  The Kalman doesn't care about what the input signal looks like at all and in fact is not really adaptive at all (unless you adapt the P-matrix which has SNR info in it).  Kalman approaches seem to be good in these one dimensional signals only when they can be applied linearly (like for timing recovery) where there really is no nonlinearity and you don't need to do the extended variation.  

The smart move is to introduce Pilot symbols with a density that allows for traditional loop structures and doesn't waste too much BW/Energy by keeping the ratio of symbols dedicated to training small. (maybe by puncturing the coding to allow the punctured outputs to be dedicated to pilot symbols).

Can't use Cray's to demodulate signals yet!!

Thanks

johnwiss,
Good work.  The recycled preamble sound like the key to your innovation.  Averaging (of any sort, Kalman or otherwise) does not really help when you are below threshold (the point where the output SNR drops drastically with dropping input SNR).  The cycle slips averaged give bad data.  So finding a way to identify the cycle slips is the key.  In an FM demodulator (PLL etc), the literature warns against this.  Unfortunately, I have not seen many ways of actually doing much about it.  Keeping a pilot tone of some sort does address the problem.  Do you have control over the format?  If so, you have the keys to solving the problem.  A warning though.  The voice guys did not like the puncturing (flag stealing) in GSM.  If you can add the overhead and not actually steal, you might be better off.  If you have a final phase (not sine wave phase, but last step in the modulation process) of CDMA like chopping you could use that as your pilot.  The 3G/GSM world is doing that quite well.  But they have the best experts around the world thinking about all the details.

Good Luck!
Best Regards,
John Solar