×
INTELLIGENT WORK FORUMS
FOR ENGINEERING PROFESSIONALS

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

How to determine & monitor changes in frequency of an irregularly occuring event over time?

How to determine & monitor changes in frequency of an irregularly occuring event over time?

How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
My question sounds fairly basic, but as statistics is not my strong suit I'm asking it here.

I have an event that happens irregularly over time, similar to, but not exactly like hurricanes. I know the time and date of each event, and thus the "time gap" between events. After each new occurrence I want to determine how frequently events are occurring, and thus if they are happening more or less often at that time. Then, after each new event, I'd update a graph and trends should emerge.

Do I use something like the exponential moving average (EMA) approach for an irregular time series to somehow calculate the average "time gap" after each new event, or is this way off? Another thought would be to instead use a fixed length of time and, after each new event, calculate the number of events per this fixed length of time to get a "running" frequency?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

Funnily enough Hurricanes was my first thought. EMA is ideal for this. I'd use say 120 months as a first guess. That is,

f(t)=(119*f(t-1)+s(t))/120

where f is the ema and s is your time history of events. The advantages of ema over simple moving average f(t)=ave(s(t),s(t-1)...s(t-119)) is that it has calculable effects on the frequency response, and responds immediately to a spike at t even if there is a spike at t-120, and generally looks more like a real system. SMA is fine where there are a lot of events in the sample time, but in signal analysis it would be unusual to use a uniform weighted SMA - I'd only deliberately use it in strange circumstances.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
I'm still a little fuzzy on the terms. You've introduced a set time, 120, when talking about EMA. I thought EMA was an infinite series going back to include all terms.

Isn't the unknown term introduced alpha (0 < alpha < 1)? With the corresponding equation something like Link:


where p would be the "time gaps" going back in the series. If so, how do you go about determining alpha?

To me this equation looks odd on the surface, because you're not dividing the sum of all of the "time gaps" by the total number of events.

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
OK, so if I'm understanding your equation correctly, this tranlates to:
EMA(t) = EMA(t-1)*(119/120) + s(t)/120
where s(t) = latest time gap in question (what I've noted above as "p").

From that link I mentioned above, an alternate form of the above equation (using similar terms) is:
EMA(t) = EMA(t-1) + [s(t) - EMA(t-1)]/120

So, if you'll indulge me a little longer, a couple more questions:
1) These two equations appear to be slightly different. Any chance you have a reference for yours?
2) Hmm. Seems like I'm still a little vague on what equation to use for EMA(t-1). Is it included within the article I've linked to above?
3) What variables go into choosing an alpha? And, can I use this alpha from the very beginning of the event series, or do I need to wait until a certain time elapses before I can start calculating EMAs?

Thanks

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

last bit first, yes, you have to do something nasty until you have 1/alpha samples.

I'd have thought it was obvious that the two equations are identical, the one is just the other expanded.

No, no refs. Oh, well wiki has it all https://en.wikipedia.org/wiki/Moving_average#Expon..., including the expansion.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

What exactly is the statistical behavior of these events? Are they supposed to be trendable?

There are other options as well, depending on what is the nature of the problem. One possibility is a maximum likelihood estimator (MLE) like a full-blown Kalman filter, although your problem might only require an alpha-beta filter.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
Chuckle, chuckle. Greg, you are correct. The equations are the same. My only defense is that it was late at night when I posted.

As far as the statistical behavior of the events, I'm not exactly sure. I'm guessing they are random, or should be random. I want to know if they are in fact random, or are the frequencies changing over time, and by how much. What does "supposed to be trendable" mean? Are some events non-trendable? Are say hurricanes (as noted above), or say murders in a city, trendable? I guess I wanted to avoid the event type from the discussion and simply look at frequency.

And I did look at a chapter Link about "maximum likelihood estimator (MLE) like a full-blown Kalman filter", but that's a little beyond my pay grade. I hope I don't have to go there.

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

I think "number of events per decade" or whatever is a pretty good place to start. EMA will handle that nicely, so would SMA. If you torture the data with some fancy techniques you might find a trend that a simple approach doesn't reveal, but if it doesn't pass some basic sniff tests you'd be hard pushed to justify it.

http://www.aoml.noaa.gov/hrd/tcfaq/E23.html

For example, were there the same number of hurricanes 1900-1950 as 1967-2017? if so then any trend in frequencies is pretty weak or inconsistent. If, as is the case, the answer is that there were fewer hurricanes in later years then torturing the data to find some increase in frequency would need to use some bulletproof method.

I've attached a somehwat cleaned up version of that data, showing decimals of a year and the hurricane strength.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?



Top line is cat 1 and higher, next is cat 2 and higher, and so on (Octave's legend box is cantankerous). The averaging period is 30 years. The greyed out section is the first 30 years. The exponential average is misleading in my opinion because it shows the effect of the zero data right through until 1950. These are trailing averages, that is the date is that of the last year in the average. Normally I'd shift the x axis so that the midpoint is used.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members! Already a Member? Login


Resources

eBook - Manufacturing the Cars of Tomorrow
In this ebook, we'll explore how additive manufacturing is going to transform the way cars are made. This includes commentary from thought leaders such as Ford's CTO, Ken Washington, Customer case studies of ways 3D printing is being used today, and a variety of part examples where 3D printing is already impacting how automobiles are made. Download Now
White Paper - Smart Manufacturing for Semiconductor
New technologies and approaches present great opportunities for semiconductor manufacturers to achieve high levels of innovation, yield and improvement. This white paper explores some of these cutting-edge technologies and how they can be applied effectively in the semiconductor industry. Read about how Smart Manufacturing is transforming the semiconductor industry. Download Now
White Paper - Analysis and Simulation in Aircraft Structure Certification
Organizations using simulation and analysis tools effectively see the benefits in their ability to achieve certification faster and with drastically less total cost than those who do not maximize these tools. Read this White Paper to learn about how digital tools such as analysis and simulation help in aircraft structure certification. Download Now

Close Box

Join Eng-Tips® Today!

Join your peers on the Internet's largest technical engineering professional community.
It's easy to join and it's free.

Here's Why Members Love Eng-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close