Sorting out noisy signals
Sorting out noisy signals
(OP)
I am a Civil Engineer who's just started in the field, so please bear with me here. It's quite simple stuff.
How can we separate different frequencies in a signal in such a way that the information modulated into a given frequency is preserved? I mean, if you use a passband filter around a target frequency, you average out many wave cycles of that frequency and all the information stored there is lost.
At least that's what happens with a FIR filter (I've tried) and I suppose IIR filters would have the same effect. So, is there a different kind of frequency filter that keeps modulation intact? I tried a PLL as well but the net effect is pretty much the same. How is that problem solved in real life?
Thanks in advance!
How can we separate different frequencies in a signal in such a way that the information modulated into a given frequency is preserved? I mean, if you use a passband filter around a target frequency, you average out many wave cycles of that frequency and all the information stored there is lost.
At least that's what happens with a FIR filter (I've tried) and I suppose IIR filters would have the same effect. So, is there a different kind of frequency filter that keeps modulation intact? I tried a PLL as well but the net effect is pretty much the same. How is that problem solved in real life?
Thanks in advance!
RE: Sorting out noisy signals
This is not true; otherwise, FM radios wouldn't exist.
A bandpass filter that just gets your carrier and modulation through is what you should be designing to.
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
There is a limit to signal contents and carrier frequency. Your filter, however steep, sets the limit for upper modulation frequency.
A practical application is in AM radios. Google superheterodyn, intermediate frequency and band-width.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Sorting out noisy signals
Now, I've seen the math of a FIR filter and also made a spreadsheet out of it. Apparently, it works fine. I tested it by submitting a signal with 6 different sine frequencies and after it goes through the filter, only what's within the passband remains.
But the math of the TIR filter is exactly what I described: the filtered signal intensities are calculated as a weighted average of previous points of the unfiltered signal, therefore mixing up intensities in different times.
For instance, let's think of a very simple AM encoding: half a cycle in full amplitude means 1 and half a cycle in 0 amplitude means 0. If I modulate 10101010101010 etc., my signal will be the positive halves of a sine. If I put that through the filter, it will average full amplitude semicycles with zero amplitude semicyles. The result is that all zero semicycles will be gone in the filtered signal. Not only that, lots of negative values also show up (there are both positive and negative weights, represented as bi in the formula above).
Check out the graph. In the example I used 44100Hz as a sampling rate, a signal of 4900Hz modulated like above (just the positive halves of a sine, the rest is 0). In the filtered signal, all areas of "silence" disappeared, as the maths should have you expect.
Am I doing anything wrong?
RE: Sorting out noisy signals
As I understand, the superhet make sense if you have more than one frequency. But the problem I mentioned arises for any number of arrier frequencies, including just 1. Or am I wrong here? Anyway, I do not intend to use more than one carrier frequency, and he problem remains. Do you have any idea how to solve it?
Thanks!
RE: Sorting out noisy signals
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: Sorting out noisy signals
You are right. Maybe that was not the best example I could have come up with. But anyway, whatever are the changes I make to a pure sine wave in the modulation process, they are going to introduce alien frequencies. I tried alternating full waves and blanks as well. Look what happenned.
I tried alternating frequencies. No mater what the signal is, the FIR summation averages out different wave cycles, thus scrambling the data.
So, how is this problem solved in practice?
RE: Sorting out noisy signals
No, most FIRs are digital models of analog filters. Analog filters have phase delays that are dependent on the frequency; that's what you are seeing. "calculated as a weighted average of previous points" is not equivalent to "average out many wave cycles of that frequency and all the information stored there is lost." If you do the math correctly on each individual frequency you will see that the signal is preserved, but delayed.
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
Yes, my description "average out many wave cycles" is not very accurate. I just meant that the weighted average of several input points across many cycles will inevitably smooth things out and scramble different cycles into a pastiche, and that is exactly what I see in my results.
Now, the math is very, very simple but of course I still could have made a mistake. Yet the filter does remove frequencies outside the passband, I checked. And if the signal is a pure sine wave, unaltered, it really goes unchanged through the filter (apart from some expected change in amplitude). If there is some mistake in the spreadsheet, are there any other tests I could perform to check that hypothesis?
RE: Sorting out noisy signals
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
I think the thinking there is on the wrong track. If that statement is fundamental to your problems with this then you need to start over.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: Sorting out noisy signals
The point is, ONLY a pure sine in a frequency f has only content in that frequency. Whatever modulated data is inserted into that pure sine creates higher frequencies. There is no way around this. So, the problem remains... How are different frequencies in a signal sorted out from each other? How do cell phones and tvs do it? They have several frequencies relatively close to each other.
RE: Sorting out noisy signals
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
And lower too. Called sidebands. Lower and Upper Sidebands.
Amplitude Modulation is quite straightforward (regarding sidebands), but Frequency and Phase Modulation just isn't.
Also, look up 'aliasing' in terms of sampling.
Signal processing, both analog and digital, is extremely non-trivial. It's not easy to provide little answers.
RE: Sorting out noisy signals
Nope.
Mike Halloran
Pembroke Pines, FL, USA
RE: Sorting out noisy signals
More detail of what signal you are specifically trying to 'pull out of the noise' will be helpful here.
Z
RE: Sorting out noisy signals
Yeah, there are the sidebands! But do you think they interfere with the problem I have in hand?
Yes, sidebands might distort my signals if my filter passband is too narrow. But tat doesn't change the fact that the signal is distorted as it goes through the filter, right? Because that is the problem I am facing, no matter what kind of modulation I use.
RE: Sorting out noisy signals
I don't need the frequency content. I just want to demodulate a signal that comes with noise. Now, that is trivial... when tere is no noise. But the only way I know to eliminate noise is to use a filter (a passband one in this case, since I have noise both above and below the carrier frequency). And unless the signal is a pure sine (it never is when there is modulated data into it), the filter distorts it (see the FIR filter equation and the graphs above) and thus the data is lost.
RE: Sorting out noisy signals
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
In the RF world, one of my quips is that, "If RF were purple, then it wouldn't be considered magic." In other words, once you can see it, it's easy.
Same sort of thing applies when trying to do signal processing. You *must* work in the frequency spectrum domain most of the time, only occasionally referring to the amplitude versus time domain as required (example, pop-noise filtering where they're more visible in the amplitude plot).
RE: Sorting out noisy signals
Ideally you'd give us a sample of the data you have in numerical form and what you hope to extract from it.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: Sorting out noisy signals
Right again, IRstuff! So my modulation of choice was not fit to my filter (or vice-versa). That is a very useful piece of information, thanks!
Let me give you some more info. My sampling rate is 44100Hz and I chose somewhat arbitrarily the carrier at 4900Hz (exactly 1/9 of the sampling rate). I'm dealing with sound being broadcast across a few meters around people and typical day-to-day noises, so I'd like the frequency to be as high as possible to be removed from the most important interferences, but at the same time I'd like to be as far away as possible from the 22050Hz limit so as not so sacrifice resolution. For now, 4900Hz is the compromise number, which we are free to change.
Back to our problem. I chose the amplitude rectangular modulation because it is easy to implement. But if my passband covers the 10x span from 4500Hz to 45000Hz (ok, as high as 22050 max), then I'll get all the unwanted noise above the carrier frequency, and I can't afford that.
All things said and done, it seems I need a modulation with the narrowest possible range of content in the frequency domain, right? Now, what would that modulation be?
RE: Sorting out noisy signals
Yep, that's great advice, thanks! I just started some tests on Excel, which is my weapon of choice.
If I had done that from the start, I wouldn't have needed the insights IRstuff just gave me.
RE: Sorting out noisy signals
Yeah, maybe I should change. But I've been an Excel guy for such a long time...
The data is going to be just a few bytes, a positive integer up to, dunno, 10^9.
RE: Sorting out noisy signals
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
Yeah, I used the 33kbps phone lines. But remember we had very little noise there! The need for filters is what makes this case more difficult and much closer to radio: an environment with a lot of noise as loud as your signal and in VERY close frequencies.
So, fair enough: I would like to send some 30 bits in less than 0.02s (or so) every couple of seconds across some 10 or 20m using a sampling rate of 44100Hz. That makes 1.5kbps, which seems pretty reasonable. The times I set are first approximations deriving from the fact that there could be multiple senders (maybe 10 or 20) which are impossible to coordinate, so their messages are spaced out across time (probably at random intervals up to a couple of seconds) in a way they don't superpose (too often). It's the only time-share solution I can envision in a situation you can't sync the senders.
So, I didn't do the math but I figured that if you send 30 different signals that are 0.02s each at 2 or 3s varying intervals, there is little superposition and in 5s you probably had at least one instance of most signals not superposed to another. The limiting factor here is obviously how long I am willing to wait to get most signals right, which I set in 4 or 5s. Of course, we could use 0.03s or even more if we have to. But I am hoping we don't.
RE: Sorting out noisy signals
There are plenty of off-the-shelf solutions for wee little radio links to perform such functions. The applicable modules are typically $1 or $2 from sellers on eBay. There's a whole ecosystem of this within the embedded "Arduino" community.
Edited to add: Andreas Spiess "The guy with the Swiss accent" on YouTube seems to specialize in this topic.
RE: Sorting out noisy signals
You could assign a different channel to each sensor, or you could increase the complexity and actually ping each sensor in turn and get it to answer back.
I have got half a dozen Arduino wifi boards, but the project I bought them for is fairly low priority, so apart from a quick look to see what they were like (they need aerisls for my application) I can't tell you much about the practical side.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: Sorting out noisy signals
But I'm doing it over sound. How could I apply that?
RE: Sorting out noisy signals
But it is supposed to be a two way communication. All parties should be able to communicate with all others.
RE: Sorting out noisy signals
... i.e., much of the hard networking stuff is already done.
Over sound links, maybe you could use time division multiplexing.
Divide each second into, say 32 periods.
Within all but one period, only one of up to ~30 transmitters can make noise; all listen.
Each sent message/squeak includes a few bits to designate the intended receiver,
a few bits to encode a message,
and a few bits to encode a checksum.
One address is for broadcast, message to everyone.
One timeslot is silent to synchronize everyone.
Everyone must send _some_ message within their designated time slot,
to show that that time slot is occupied and active,
so there must be a 'null message' for that purpose.
Identities 0..30 are preassigned to each unit.
You may need to add some fudge code to get all the transceivers into sync at startup,
like changing time delays from an arbitrary starting point until everyone receives
a complete cycle of messages with no checksum errors, i.e., no one is transmitting
when he shouldn't be.
Or equip everyone with radio receivers and use WWV for sync.
The above outline represents the core of something that might work,
BUT, if you are doing this among people,
expect complaints of annoyance about the buzzing if they can hear it,
and complaints of general malaise if they can't.
It's also going to be more expensive and more difficult than using stuff you can buy.
Mike Halloran
Pembroke Pines, FL, USA
RE: Sorting out noisy signals
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: Sorting out noisy signals
Interesting ideas, Mike. But I'd have to be able to sync everybody with a given clock first, which I'm not sure is possible. Anyways, time multiplexing is solvable. But I'll never get there if the signal doesn't make it through the filter, which is where I am right now.
RE: Sorting out noisy signals
Well, if y'all wanna help me along the whole project, that would be great! But I wouldnt ask that. All I'm asking is how to make a signal survive going through a filter.
I'm sorry, what are you talking about?
I didn't talk about the whole project before because most of its features are irrelevant and I thought those information would throw people away from the one thing I came here asking for help - and that's exactly what is happening now.
The fact that it is 2way com doesn't change the problem at hand: how to extract a signal in a noisy environment since filters tend to distort it. I'm guessing that TVs, cellphones and radios have filters with very little distortion because you can afford to use thousands of wavecycles to produce a single bit of information, so modulation functions can be very, very gradual, with derivatives very close to zero, thus being almost perfect sines at short intervals, and those kinds of signals have almost no sidebands and will go pretty much undisturbed through a filter.
In other words, I tried rectangular modulation functions with frequency once or two times the carrier frequency. That created huge sidebands, far outside my passband. On the other hand, TVs, cellphones and radios have gradual modulation functions with frequencies that are at least 2 or 3 orders of magnitude inferior to the carrier. Right now, I'm guessing that's where my problem lies. If that is true, the solution is to use more gradual modulation functions (and lose A LOT of bandwidth, of course).
I was hoping I could have 1 or two bits per wave. But apparently that is too much to ask.
RE: Sorting out noisy signals
No, that's not what they do. They do a systems engineering analysis, come up with rational and sound design requirements, and THEN, they design.
You keep harping about noise, but you have produced zero information about it. How can you expect to come up with a workable solution if you don't provide ALL the relevant design specs and environments?
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
Read my msg again; I outlined two distinct ways to get the clocks in sync.
Mike Halloran
Pembroke Pines, FL, USA
RE: Sorting out noisy signals
What relevant info is there about this that is needed to solve the problem? I really don't know!
RE: Sorting out noisy signals
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
A few (quite a few, actually) tried to help you on the right track. But not easy with the very vague specifications that you presented. A moving target, as it turned out to be. My answer: "Shannon and others formulated that a long time ago. It is about band-width of your filter. Modulation causes sidebands. Simple in AM and somewhat more complex in FM". Was an attempt to make you understand the basic concepts. Did you google Shannon and his basic work?
If you think that "It's quite simple stuff" then you may have to rethink.
BTW, I think that I shall pick up bridge designing. There can't be much to it. Seems to be quite simple stuff. The Romans built the Pont du Gard and medieval masonrys built Cathedrals. So why shouldn't an electrical engineer like me start designing bridges? I wonder.
You may have noticed the little flag "Report" in the lower right corner. Well this thread is very close to being just that. And disappear.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Sorting out noisy signals
Dan - Owner
http://www.Hi-TecDesigns.com
RE: Sorting out noisy signals
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Sorting out noisy signals
thread815-437029: Miami Pedestrian Bridge, Part V
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Sorting out noisy signals
Sigh...
Time and effort > /dev/null 2>&1
Perhaps we shouldn't bother at all.
RE: Sorting out noisy signals
VE1BLL, I am very liberal when it comes to impossible threads. But this one has not brought anything but confusion and frustration. We all learn, but the OP's conviction that "It's simple stuff' makes every effort go to the bit bucket.
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.
RE: Sorting out noisy signals
The OP cannot fail to be learning from this thread. As well, some of the participants might have picked up a tidbit or two. And perhaps lurkers too.
RE: Sorting out noisy signals
Gunnar Englund
www.gke.org
--------------------------------------
Half full - Half empty? I don't mind. It's what in it that counts.