×
INTELLIGENT WORK FORUMS
FOR ENGINEERING PROFESSIONALS

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
  • Students Click Here

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Students Click Here

Jobs

How to determine & monitor changes in frequency of an irregularly occuring event over time?

How to determine & monitor changes in frequency of an irregularly occuring event over time?

How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
My question sounds fairly basic, but as statistics is not my strong suit I'm asking it here.

I have an event that happens irregularly over time, similar to, but not exactly like hurricanes. I know the time and date of each event, and thus the "time gap" between events. After each new occurrence I want to determine how frequently events are occurring, and thus if they are happening more or less often at that time. Then, after each new event, I'd update a graph and trends should emerge.

Do I use something like the exponential moving average (EMA) approach for an irregular time series to somehow calculate the average "time gap" after each new event, or is this way off? Another thought would be to instead use a fixed length of time and, after each new event, calculate the number of events per this fixed length of time to get a "running" frequency?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

Funnily enough Hurricanes was my first thought. EMA is ideal for this. I'd use say 120 months as a first guess. That is,

f(t)=(119*f(t-1)+s(t))/120

where f is the ema and s is your time history of events. The advantages of ema over simple moving average f(t)=ave(s(t),s(t-1)...s(t-119)) is that it has calculable effects on the frequency response, and responds immediately to a spike at t even if there is a spike at t-120, and generally looks more like a real system. SMA is fine where there are a lot of events in the sample time, but in signal analysis it would be unusual to use a uniform weighted SMA - I'd only deliberately use it in strange circumstances.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
I'm still a little fuzzy on the terms. You've introduced a set time, 120, when talking about EMA. I thought EMA was an infinite series going back to include all terms.

Isn't the unknown term introduced alpha (0 < alpha < 1)? With the corresponding equation something like Link:


where p would be the "time gaps" going back in the series. If so, how do you go about determining alpha?

To me this equation looks odd on the surface, because you're not dividing the sum of all of the "time gaps" by the total number of events.

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
OK, so if I'm understanding your equation correctly, this tranlates to:
EMA(t) = EMA(t-1)*(119/120) + s(t)/120
where s(t) = latest time gap in question (what I've noted above as "p").

From that link I mentioned above, an alternate form of the above equation (using similar terms) is:
EMA(t) = EMA(t-1) + [s(t) - EMA(t-1)]/120

So, if you'll indulge me a little longer, a couple more questions:
1) These two equations appear to be slightly different. Any chance you have a reference for yours?
2) Hmm. Seems like I'm still a little vague on what equation to use for EMA(t-1). Is it included within the article I've linked to above?
3) What variables go into choosing an alpha? And, can I use this alpha from the very beginning of the event series, or do I need to wait until a certain time elapses before I can start calculating EMAs?

Thanks

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

last bit first, yes, you have to do something nasty until you have 1/alpha samples.

I'd have thought it was obvious that the two equations are identical, the one is just the other expanded.

No, no refs. Oh, well wiki has it all https://en.wikipedia.org/wiki/Moving_average#Expon..., including the expansion.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

What exactly is the statistical behavior of these events? Are they supposed to be trendable?

There are other options as well, depending on what is the nature of the problem. One possibility is a maximum likelihood estimator (MLE) like a full-blown Kalman filter, although your problem might only require an alpha-beta filter.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

(OP)
Chuckle, chuckle. Greg, you are correct. The equations are the same. My only defense is that it was late at night when I posted.

As far as the statistical behavior of the events, I'm not exactly sure. I'm guessing they are random, or should be random. I want to know if they are in fact random, or are the frequencies changing over time, and by how much. What does "supposed to be trendable" mean? Are some events non-trendable? Are say hurricanes (as noted above), or say murders in a city, trendable? I guess I wanted to avoid the event type from the discussion and simply look at frequency.

And I did look at a chapter Link about "maximum likelihood estimator (MLE) like a full-blown Kalman filter", but that's a little beyond my pay grade. I hope I don't have to go there.

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?

I think "number of events per decade" or whatever is a pretty good place to start. EMA will handle that nicely, so would SMA. If you torture the data with some fancy techniques you might find a trend that a simple approach doesn't reveal, but if it doesn't pass some basic sniff tests you'd be hard pushed to justify it.

http://www.aoml.noaa.gov/hrd/tcfaq/E23.html

For example, were there the same number of hurricanes 1900-1950 as 1967-2017? if so then any trend in frequencies is pretty weak or inconsistent. If, as is the case, the answer is that there were fewer hurricanes in later years then torturing the data to find some increase in frequency would need to use some bulletproof method.

I've attached a somehwat cleaned up version of that data, showing decimals of a year and the hurricane strength.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

RE: How to determine & monitor changes in frequency of an irregularly occuring event over time?



Top line is cat 1 and higher, next is cat 2 and higher, and so on (Octave's legend box is cantankerous). The averaging period is 30 years. The greyed out section is the first 30 years. The exponential average is misleading in my opinion because it shows the effect of the zero data right through until 1950. These are trailing averages, that is the date is that of the last year in the average. Normally I'd shift the x axis so that the midpoint is used.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376: Eng-Tips.com Forum Policies http://eng-tips.com/market.cfm?

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members! Already a Member? Login


Resources

White Paper - The Criticality of the E/E Architecture
Modern vehicles are highly sophisticated systems incorporating electrical, electronic, software and mechanical components. Mechanical systems are giving way to advanced software and electronic devices, driving automakers to innovate and differentiate their vehicles via the electric and electronic (E/E) architecture. As the pace of change accelerates, automotive companies need to evolve their development processes to deliver and maximize the value of these architectures. Download Now
White Paper - Model Based Engineering for Wire Harness Manufacturing
Modern cars, trucks, and other vehicles feature an ever-increasing number of sophisticated electrical and electronic features, placing a larger burden on the wiring harness that enables these new features. As complexity rises, current harness manufacturing methods are struggling to keep pace due to manual data exchanges and the inability to capture tribal knowledge. A model-based wire harness manufacturing engineering flow automates data exchange and captures tribal knowledge through design rules to help harness manufacturers improve harness quality and boost efficiency. Download Now
White Paper - Modeling and Optimizing Wire Harness Costs for Variation Complexity
This paper will focus on the quantification of the complexity related costs in harness variations in order to model them, allowing automated algorithms to optimize for these costs. A number of real world examples will be provided as well. Since no two businesses are alike, it is the aim of this paper to provide the foundational knowledge and methodology so the reader can assess their own business to model how variation complexity costs affect their business. Download Now

Close Box

Join Eng-Tips® Today!

Join your peers on the Internet's largest technical engineering professional community.
It's easy to join and it's free.

Here's Why Members Love Eng-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close