Reducing data set

Question

Reducing data set

dgallup (Automotive)

(OP)

30 Apr 15 13:47

I have a 4 column by 80,800 row data set. It's too much to graph. It's data recorded every 6 seconds, I could get rid of 90% of it and only keep one record every minute. How can I automate that?

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

IRstuff (Aerospace) · Answer 1 · 2015-04-30 14:15:12

HAve you tried the autofilter?

Note that Excel 2010 has removed that specific limit, but the older versions of Excel are only limited to 32k data points, so there's not that much decimation that you need to do.

TTFN
FAQ731-376: Eng-Tips.com Forum Policies

Need help writing a question or understanding a reply? forum1529: Translation Assistance for Engineers

Of course I can. I can do anything. I can do absolutely anything. I'm an expert!
There is a homework forum hosted by engineering.com: http://www.engineering.com/AskForum/aff/32.aspx

dgallup (Automotive) · Answer 2 · 2015-04-30 14:31:00

I'm using 2007 so it's only graphing about 40% of the data. I hate working with so much data anyway, it's just a lot of unnecessary overhead.

I tried filtering to only times that fit the pattern ??:??:00. The beginning data has points that meet that criteria but evidently the clock has a little skew so after awhile there are no data points that meet that criteria. I'll try adding a column with just sequential integers and filter on that.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

dgallup (Automotive) · Answer 3 · 2015-04-30 14:48:13

Can't get the column of integers to work. Filtering for values ending in 1 produces nothing. Every value is filtered out. Tried filtering for *1, same thing. Very strange. Meanwhile, my technician managed to download data from the machine with points every 3 minutes, a little coarser than I wanted but at least manageable. I would still like to understand why the filtering on the column of integers doesn't work.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

dgallup (Automotive) · Answer 4 · 2015-04-30 14:58:49

Replaced mu column of integers with a repeating sequence of 1 to 10 and that filtered just fine.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

IRstuff (Aerospace) · Answer 5 · 2015-04-30 15:16:45

Some form of the index function might work

TTFN
FAQ731-376: Eng-Tips.com Forum Policies

Need help writing a question or understanding a reply? forum1529: Translation Assistance for Engineers

Of course I can. I can do anything. I can do absolutely anything. I'm an expert!
There is a homework forum hosted by engineering.com: http://www.engineering.com/AskForum/aff/32.aspx

Denial (Structural) · Answer 6 · 2015-04-30 21:18:26

If you are going to add an extra column try using
=MOD(ROW(),10)
replacing the 10 with whatever sampling severity is to your taste.
(You could even use a variable instead of 10, so you could readily explore the graph's sensitivity to the value.)

yakpol (Structural) · Answer 7 · 2015-04-30 23:58:58

You may try VBA macro to reduce the size of the table.

CODE -->

Sub delete_rows()

Nsave = 3 ' Number of consequitive rows to save
Ndelete = 3  ' Number of consequitive rows to delete
Row1 = 6 ' Number of the first row of database
Nrows = 35 'Number of rows in database

Niter& = Nrows / (Ndelete + Nsave)

For i = 1 To Niter
    Row1 = Row1 + Nsave
    sRows$ = Row1 & ":" & Row1 + Ndelete - 1
    Rows(sRows).Delete
Next i

End Sub

Yakpol

IDS (Civil/Environmental) · Answer 8 · 2015-05-01 02:24:54

Quote:
Some form of the index function might work

There are many ways to do this, and there is nothing wrong with the filtered list or VBA approach, but I would probably rearrange all the data using the Index function.

Say we wanted to plot points at 1 minute intervals, which is convenient because that's every tenth point:

Suppose we want the rearranged table to start in cell B2
In row 1 above the output range enter the values 0 to 9 in starting in column B.
In Cell A2 enter 1
In Cell A3 enter =A2+10 and copy down as far as required.
In Cell B2 enter =Index(datarange, $A2 + B$1). Datarange is either a named range, or the absolute cell address of the range (must be a single column).
Copy across the 10 columns and down as far as required.
If desired, copy the entire table and paste as values, or if the data will change leave it as formulas.

You can now plot any one of the columns, or find the average of each row and plot that (or max, or min, or whatever).

Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/

dgallup (Automotive) · Answer 9 · 2015-05-01 12:58:44

Thanks all, that gives me several ways to skin this cat. I went with the filtered list then copied the reduced set. Produced a beautiful plot.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

zelgar (Civil/Environmental) · Answer 10 · 2015-05-01 13:33:53

IMHO, it's never a good idea to delete raw data since once it's lost, it can never be replaced. It would be better to have the macro copy the data from the original sheet to a 2nd sheet at the frequency you want like the following:

CODE -->

Sub points()
Dim freq As Integer, iRow As Integer, iCol As Integer, iRow2 As Integer
freq = 10 ' frequency of getting data
iRow = 2 ' starting row of data
iCol = 1 ' column of data
iRow2 = 2 'Starting row of data in new sheet

Do Until IsEmpty(Sheet1.Cells(iRow, iCol))
    Sheet1.Rows(iRow).Copy Destination:=Sheet2.Cells(iRow2, 1)
    iRow = iRow + freq
    iRow2 = iRow2 + 1
Loop
End Sub

Come Join Us!

Posting Guidelines

Reducing data set