Extrapolation of a measurement data 1

[-amir]

I would like to extrapolate the results of a measurement. Following picture shows the measurements results. The best I can do is that to calculate the average values for 10 measurement and I use it for approximation, but the error is too large. In the following figure the blue graph is for measurement and the red graph is the average for 10 steps of measurement. The dots show the measurement points.
Does any one knows a better method for extrapolation so that one can get a smaller error?

 

[drawoh]

-amir,

Is extrapolation a good idea? Do you have some underlying mathematical theory that justifies you claiming that your lines can be extended off either end of your graph? Can you extract accurate information from your highly inaccurate data? Do you understand why your data is all over the place?

--
JHG

 

[MFJewell]

Is extrapolation a good idea?

I remember my numerical methods professor telling us extrapolation is a terrible idea.

 

[IFRs]

Sure you can extrapolate. Just not at the vertical scale you have. Change the scale so instead of 0 to 900 it goes from 0 to 9000. I suspect it will not give you what you wanted...

 

[IRstuff]

There are two issues with least squares (averaging): slow convergence and lag.

You might consider some maximum likelihood approaches, such as simplified Bayesian or Kalman estimators.

TTFN (ta ta for now)
I can do absolutely anything. I'm an expert!

https://www.youtube.com/watch?v=BKorP55Aqvg

faq731-376 forum1529 Entire Forum list

http://www.eng-tips.com/forumlist.cfm

 

[zdas04]

I always approach extrapolation with a great deal of fear and trepidation. If I have to do it, I require more than 10 data points (10,000 is OK, but I'd rather see a half million or more). If you ever want to see how worthless 10 data points is (it is no better for interpolation than for extrapolation) capture some really short interval measurement data (say 400 data points per second or so) and look at the variability. It will make you cringe.

[bold]David Simpson, PE[/bold]
MuleShoe Engineering

In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual. Galileo Galilei, Italian Physicist

 

[Denial]

I think you are trying to make a silk purse out of a cow's ear [sic].[ ] That data is all over the place.[ ] Even if you try to extrapolate, exactly WHAT are you going to use as the basis for the extrapolation?[ ] Your orange line?[ ] How?[ ] Even that line does not show a usable trend.

If you are going to make any heroic assumptions about how that messy data is behaving, you are going to need strong a priori reasons to back up the assumptions.[ ]

On top of that, I don't believe your orange line is actually a "rolling average" of 10 data points.[ ] The 10 successive data points that end with the third last point on your graph are (approximately) 590, 430, 540, 380, 520, 440, 550, 610, 500 & 400.[ ] The average of these ten values is 496.[ ] But the corresponding "average" points you show on your orange graph trend steadily from 540 to 600.[ ] So your orange points cannot represent a 10-point average, whether that average is calculated retrospectively, centrally, or prospectively.

 

[GregLocock]

The first thing you have to decide is whether it is a stationary process, with a lot of noise, or some function of time (I assume the (failing grade) x axis is weeks). One trick is to split the data into two halves and overplot them. From that I'd take a wild guess that there is a weak underlying annual cycle. Your moving average is dodgy as pointed out above, I suspect you have wrapped around, consciously or unconsciously, or trusted Excel.

Fitting trend lines to noisy data without an underlying model of the process is very hard to justify. For instance a drunkard's walk of a very simple process could give you exactly the same plot, yet is useless for extrapolation. y(x+1)=y(x)+ (RND(1)-.5) where RND(1) is gaussian or uniformly distributed random numbers between 0 and 1 (doesn't matter much). That will go to plus or minus infinity, eventually, with a value proportional to approximately x^0.5

If you can't identify a model that can be used to guess at a trend line then even saying it is a stable process is dangerous. In process control your first job is to get a stable process, before bothering with any fancy maths.



Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[Denial]

GregLocock has astutely noticed that your horizontal axis runs from 1 to 52, which strongly suggests that the data is annual.[ ] Overlooking the fact that you didn't bother to tell us this, if the data IS annual then you probably have a priori reasons to assume that it will tend to repeat annually.

For data that repeats annually, you should investigate fitting a sinusoidal trend line.[ ] For an example of this, there was a recent discussion on Eng-Tips' companion website Tek-Tips, in its "Microsoft: Office" forum.[ ] See its thread thread68-1785035, initiated by "bhujanga".[ ] (The OP had daily data rather than annual data, but the principles are the same.)[ ] He was trying to fit a quartic polynomial.[ ] I tried, unsuccessfully I fear, to convince him to use Fourier-type sinusoids.[ ] In that thread, see my posts of 06Mar18@11:41, 06Mar18@20:41, and 07Mar18@02:20.[ ] One of these includes an attached spreadsheet that does the sinusoid-fitting.

You are still going to have a helluva lot of noise, but this approach will probably reduce it slightly.

 

[-amir]

Thank you for all the comments.
That is true that the measurement has lots of noises, roughly about 20% of the average level. For next step approximation (extrapolation), I do not want to find an exact number and just best possible approximation with a smaller error can be OK. Unfortunately the data is not periodic also. For the picture,12 points for averaging is used.
Maybe one can reduce the error as followings:
If the measured data at a point is y then the average is y_av. Fluctuation around the average is z=ABS(y-y_av). Again I will find the average of the z that is z_av.
Now I can have two up and low boundary:
Up boundary is y_av+z_av
Low boundary is y_av-z_av
One of this up or down boundary seems gives a better approximation than only y_av.
I hope that I can extrapolate the average values because they are more smooth than the measured values.
To choose number of data points for averaging is not easy too. I want to get a smooth curve but I do not want to lost so information in averaging.

 

[MacGyverS2000]

I remember my numerical methods professor telling us extrapolation is a terrible idea.

He formed an opinion based upon the data he had...

Dan - Owner

http://www.Hi-TecDesigns.com

 

[GregLocock]

-amir, you seem to have ignored every single piece of advice or observation you have been given. Good luck.

Cheers

Greg Locock


New here? Try reading these, they might help FAQ731-376

http://eng-tips.com/market.cfm?

 

[Denial]

One final point for you to ignore.[ ] Whether it has arisen from random processes or not, your data DOES contain some periodicity.[ ] Allowing for it reduces the SSE by about 20%.

 

[rb1957]

when you say "extrapolation" what exactly do you mean ? If X is time then extrapolating into the future ? Doesn't sound unreasonable.

You have a lot of data with a lot of scatter. You could assume no change in mean over time (if X is time) or what appears to be a weak movement of the mean (like a cos wave with amplitude of 50).

You can't (or shouldn't) wish error away because its inconvenient.

How much does it matter what the reading is ?? If there's lots of $$ riding on the results then it serves to collect more data, or collect data more accurately.

another day in paradise, or is paradise one day closer ?