How to determine & monitor changes in frequency of an irregularly occuring event over time?
How to determine & monitor changes in frequency of an irregularly occuring event over time?
(OP)
My question sounds fairly basic, but as statistics is not my strong suit I'm asking it here.
I have an event that happens irregularly over time, similar to, but not exactly like hurricanes. I know the time and date of each event, and thus the "time gap" between events. After each new occurrence I want to determine how frequently events are occurring, and thus if they are happening more or less often at that time. Then, after each new event, I'd update a graph and trends should emerge.
Do I use something like the exponential moving average (EMA) approach for an irregular time series to somehow calculate the average "time gap" after each new event, or is this way off? Another thought would be to instead use a fixed length of time and, after each new event, calculate the number of events per this fixed length of time to get a "running" frequency?
I have an event that happens irregularly over time, similar to, but not exactly like hurricanes. I know the time and date of each event, and thus the "time gap" between events. After each new occurrence I want to determine how frequently events are occurring, and thus if they are happening more or less often at that time. Then, after each new event, I'd update a graph and trends should emerge.
Do I use something like the exponential moving average (EMA) approach for an irregular time series to somehow calculate the average "time gap" after each new event, or is this way off? Another thought would be to instead use a fixed length of time and, after each new event, calculate the number of events per this fixed length of time to get a "running" frequency?
Talk To Other Members
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
f(t)=(119*f(t-1)+s(t))/120
where f is the ema and s is your time history of events. The advantages of ema over simple moving average f(t)=ave(s(t),s(t-1)...s(t-119)) is that it has calculable effects on the frequency response, and responds immediately to a spike at t even if there is a spike at t-120, and generally looks more like a real system. SMA is fine where there are a lot of events in the sample time, but in signal analysis it would be unusual to use a uniform weighted SMA - I'd only deliberately use it in strange circumstances.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
Isn't the unknown term introduced alpha (0 < alpha < 1)? With the corresponding equation something like Link:
where p would be the "time gaps" going back in the series. If so, how do you go about determining alpha?
To me this equation looks odd on the surface, because you're not dividing the sum of all of the "time gaps" by the total number of events.
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
Just try it out, it works.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
EMA(t) = EMA(t-1)*(119/120) + s(t)/120
where s(t) = latest time gap in question (what I've noted above as "p").
From that link I mentioned above, an alternate form of the above equation (using similar terms) is:
EMA(t) = EMA(t-1) + [s(t) - EMA(t-1)]/120
So, if you'll indulge me a little longer, a couple more questions:
1) These two equations appear to be slightly different. Any chance you have a reference for yours?
2) Hmm. Seems like I'm still a little vague on what equation to use for EMA(t-1). Is it included within the article I've linked to above?
3) What variables go into choosing an alpha? And, can I use this alpha from the very beginning of the event series, or do I need to wait until a certain time elapses before I can start calculating EMAs?
Thanks
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
I'd have thought it was obvious that the two equations are identical, the one is just the other expanded.
No, no refs. Oh, well wiki has it all https://en.wikipedia.org/wiki/Moving_average#Expon..., including the expansion.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
There are other options as well, depending on what is the nature of the problem. One possibility is a maximum likelihood estimator (MLE) like a full-blown Kalman filter, although your problem might only require an alpha-beta filter.
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
As far as the statistical behavior of the events, I'm not exactly sure. I'm guessing they are random, or should be random. I want to know if they are in fact random, or are the frequencies changing over time, and by how much. What does "supposed to be trendable" mean? Are some events non-trendable? Are say hurricanes (as noted above), or say murders in a city, trendable? I guess I wanted to avoid the event type from the discussion and simply look at frequency.
And I did look at a chapter Link about "maximum likelihood estimator (MLE) like a full-blown Kalman filter", but that's a little beyond my pay grade. I hope I don't have to go there.
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
http://www.aoml.noaa.gov/hrd/tcfaq/E23.html
For example, were there the same number of hurricanes 1900-1950 as 1967-2017? if so then any trend in frequencies is pretty weak or inconsistent. If, as is the case, the answer is that there were fewer hurricanes in later years then torturing the data to find some increase in frequency would need to use some bulletproof method.
I've attached a somehwat cleaned up version of that data, showing decimals of a year and the hurricane strength.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?
RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?
Top line is cat 1 and higher, next is cat 2 and higher, and so on (Octave's legend box is cantankerous). The averaging period is 30 years. The greyed out section is the first 30 years. The exponential average is misleading in my opinion because it shows the effect of the zero data right through until 1950. These are trailing averages, that is the date is that of the last year in the average. Normally I'd shift the x axis so that the midpoint is used.
Cheers
Greg Locock
New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?