Continue to Site

Eng-Tips is the largest engineering community on the Internet

Intelligent Work Forums for Engineering Professionals

Register Log in
  • Congratulations KootK on being selected by the Eng-Tips community for having the most helpful posts in the forums last week. Way to Go!

How to determine & monitor changes in frequency of an irregularly occuring event over time?

Status
Not open for further replies.
TheGreenLama

TheGreenLama

Structural
Mar 21, 2018
155
My question sounds fairly basic, but as statistics is not my strong suit I'm asking it here.

I have an event that happens irregularly over time, similar to, but not exactly like hurricanes. I know the time and date of each event, and thus the "time gap" between events. After each new occurrence I want to determine how frequently events are occurring, and thus if they are happening more or less often at that time. Then, after each new event, I'd update a graph and trends should emerge.

Do I use something like the exponential moving average (EMA) approach for an irregular time series to somehow calculate the average "time gap" after each new event, or is this way off? Another thought would be to instead use a fixed length of time and, after each new event, calculate the number of events per this fixed length of time to get a "running" frequency?
 
Replies continue below

Recommended for you

Sort by date Sort by votes
Funnily enough Hurricanes was my first thought. EMA is ideal for this. I'd use say 120 months as a first guess. That is,

f(t)=(119*f(t-1)+s(t))/120

where f is the ema and s is your time history of events. The advantages of ema over simple moving average f(t)=ave(s(t),s(t-1)...s(t-119)) is that it has calculable effects on the frequency response, and responds immediately to a spike at t even if there is a spike at t-120, and generally looks more like a real system. SMA is fine where there are a lot of events in the sample time, but in signal analysis it would be unusual to use a uniform weighted SMA - I'd only deliberately use it in strange circumstances.



Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376
 
Upvote 0 Downvote
I'm still a little fuzzy on the terms. You've introduced a set time, 120, when talking about EMA. I thought EMA was an infinite series going back to include all terms.

Isn't the unknown term introduced alpha (0 < alpha < 1)? With the corresponding equation something like Link:

EMA1_m4bbb2.jpg

where p would be the "time gaps" going back in the series. If so, how do you go about determining alpha?

To me this equation looks odd on the surface, because you're not dividing the sum of all of the "time gaps" by the total number of events.
 
Upvote 0 Downvote
f(t-1) includes some memory of all the preceeding values . alpha is 1/120 in the case I chose.

Just try it out, it works.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376
 
Upvote 0 Downvote
OK, so if I'm understanding your equation correctly, this tranlates to:
EMA(t) = EMA(t-1)*(119/120) + s(t)/120
where s(t) = latest time gap in question (what I've noted above as "p").

From that link I mentioned above, an alternate form of the above equation (using similar terms) is:
EMA(t) = EMA(t-1) + [s(t) - EMA(t-1)]/120

So, if you'll indulge me a little longer, a couple more questions:
1) These two equations appear to be slightly different. Any chance you have a reference for yours?
2) Hmm. Seems like I'm still a little vague on what equation to use for EMA(t-1). Is it included within the article I've linked to above?
3) What variables go into choosing an alpha? And, can I use this alpha from the very beginning of the event series, or do I need to wait until a certain time elapses before I can start calculating EMAs?

Thanks
 
Upvote 0 Downvote
last bit first, yes, you have to do something nasty until you have 1/alpha samples.

I'd have thought it was obvious that the two equations are identical, the one is just the other expanded.

No, no refs. Oh, well wiki has it all including the expansion.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376
 
Upvote 0 Downvote
What exactly is the statistical behavior of these events? Are they supposed to be trendable?

There are other options as well, depending on what is the nature of the problem. One possibility is a maximum likelihood estimator (MLE) like a full-blown Kalman filter, although your problem might only require an alpha-beta filter.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! faq731-376 forum1529 Entire Forum list
 
Upvote 0 Downvote
Chuckle, chuckle. Greg, you are correct. The equations are the same. My only defense is that it was late at night when I posted.

As far as the statistical behavior of the events, I'm not exactly sure. I'm guessing they are random, or should be random. I want to know if they are in fact random, or are the frequencies changing over time, and by how much. What does "supposed to be trendable" mean? Are some events non-trendable? Are say hurricanes (as noted above), or say murders in a city, trendable? I guess I wanted to avoid the event type from the discussion and simply look at frequency.

And I did look at a chapter Link about "maximum likelihood estimator (MLE) like a full-blown Kalman filter", but that's a little beyond my pay grade. I hope I don't have to go there.
 
Upvote 0 Downvote
I think "number of events per decade" or whatever is a pretty good place to start. EMA will handle that nicely, so would SMA. If you torture the data with some fancy techniques you might find a trend that a simple approach doesn't reveal, but if it doesn't pass some basic sniff tests you'd be hard pushed to justify it.


For example, were there the same number of hurricanes 1900-1950 as 1967-2017? if so then any trend in frequencies is pretty weak or inconsistent. If, as is the case, the answer is that there were fewer hurricanes in later years then torturing the data to find some increase in frequency would need to use some bulletproof method.

I've attached a somehwat cleaned up version of that data, showing decimals of a year and the hurricane strength.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376
 
Upvote 0 Downvote
hurricanes_vhnifg.png


Top line is cat 1 and higher, next is cat 2 and higher, and so on (Octave's legend box is cantankerous). The averaging period is 30 years. The greyed out section is the first 30 years. The exponential average is misleading in my opinion because it shows the effect of the zero data right through until 1950. These are trailing averages, that is the date is that of the last year in the average. Normally I'd shift the x axis so that the midpoint is used.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376
 
Upvote 0 Downvote
Status
Not open for further replies.

Similar threads

MK_0202
  • Locked
  • Question
How to Increase Minor Axis Bending for Rolled-H section 1
Replies
7
Views
7K
Agent666
Agent666
Ocman76
  • Locked
  • Question
How is the Buoyancy Reduction Factor Equation derived?
Replies
1
Views
831
IRstuff
IRstuff
YungPlantEng
  • Locked
  • Question
Automation of coding historical failure data?
Replies
1
Views
731
IRstuff
IRstuff
naah
  • Locked
  • Question
Risk assessment/ Riskanalysis of design change
Replies
2
Views
828
GregLocock
GregLocock
Souvik Singh Rathore
  • Locked
  • Question
How to use Lame's parameter 'A' while converting x,y coordinate into zeta,eta coordinate (FEA)
Replies
1
Views
699
IRstuff
IRstuff

Part and Inventory Search

Sponsor

Back
Top Bottom