INTELLIGENT WORK FORUMS
FOR ENGINEERING PROFESSIONALS

Log In

Come Join Us!

Are you an
Engineering professional?
Join Eng-Tips Forums!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!

*Eng-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

Posting Guidelines

Promoting, selling, recruiting, coursework and thesis posting is forbidden.

Jobs

Reducing data set

Reducing data set

(OP)
I have a 4 column by 80,800 row data set. It's too much to graph. It's data recorded every 6 seconds, I could get rid of 90% of it and only keep one record every minute. How can I automate that?

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

RE: Reducing data set

HAve you tried the autofilter?

Note that Excel 2010 has removed that specific limit, but the older versions of Excel are only limited to 32k data points, so there's not that much decimation that you need to do.

TTFN
FAQ731-376: Eng-Tips.com Forum Policies

Need help writing a question or understanding a reply? forum1529: Translation Assistance for Engineers


Of course I can. I can do anything. I can do absolutely anything. I'm an expert!
There is a homework forum hosted by engineering.com: http://www.engineering.com/AskForum/aff/32.aspx

RE: Reducing data set

(OP)
I'm using 2007 so it's only graphing about 40% of the data. I hate working with so much data anyway, it's just a lot of unnecessary overhead.

I tried filtering to only times that fit the pattern ??:??:00. The beginning data has points that meet that criteria but evidently the clock has a little skew so after awhile there are no data points that meet that criteria. I'll try adding a column with just sequential integers and filter on that.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

RE: Reducing data set

(OP)
Can't get the column of integers to work. Filtering for values ending in 1 produces nothing. Every value is filtered out. Tried filtering for *1, same thing. Very strange. Meanwhile, my technician managed to download data from the machine with points every 3 minutes, a little coarser than I wanted but at least manageable. I would still like to understand why the filtering on the column of integers doesn't work.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

RE: Reducing data set

(OP)
Replaced mu column of integers with a repeating sequence of 1 to 10 and that filtered just fine.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

RE: Reducing data set

If you are going to add an extra column try using
=MOD(ROW(),10)
replacing the 10 with whatever sampling severity is to your taste.
(You could even use a variable instead of 10, so you could readily explore the graph's sensitivity to the value.)

RE: Reducing data set

You may try VBA macro to reduce the size of the table.

CODE -->

Sub delete_rows()

Nsave = 3 ' Number of consequitive rows to save
Ndelete = 3  ' Number of consequitive rows to delete
Row1 = 6 ' Number of the first row of database
Nrows = 35 'Number of rows in database

Niter& = Nrows / (Ndelete + Nsave)

For i = 1 To Niter
    Row1 = Row1 + Nsave
    sRows$ = Row1 & ":" & Row1 + Ndelete - 1
    Rows(sRows).Delete
Next i

End Sub 

Yakpol

RE: Reducing data set

Quote:

Some form of the index function might work

There are many ways to do this, and there is nothing wrong with the filtered list or VBA approach, but I would probably rearrange all the data using the Index function.

Say we wanted to plot points at 1 minute intervals, which is convenient because that's every tenth point:

Suppose we want the rearranged table to start in cell B2
In row 1 above the output range enter the values 0 to 9 in starting in column B.
In Cell A2 enter 1
In Cell A3 enter =A2+10 and copy down as far as required.
In Cell B2 enter =Index(datarange, $A2 + B$1). Datarange is either a named range, or the absolute cell address of the range (must be a single column).
Copy across the 10 columns and down as far as required.
If desired, copy the entire table and paste as values, or if the data will change leave it as formulas.

You can now plot any one of the columns, or find the average of each row and plot that (or max, or min, or whatever).

Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/

RE: Reducing data set

(OP)
Thanks all, that gives me several ways to skin this cat. I went with the filtered list then copied the reduced set. Produced a beautiful plot.

----------------------------------------

The Help for this program was created in Windows Help format, which depends on a feature that isn't included in this version of Windows.

RE: Reducing data set

IMHO, it's never a good idea to delete raw data since once it's lost, it can never be replaced. It would be better to have the macro copy the data from the original sheet to a 2nd sheet at the frequency you want like the following:

CODE -->

Sub points()
Dim freq As Integer, iRow As Integer, iCol As Integer, iRow2 As Integer
freq = 10 ' frequency of getting data
iRow = 2 ' starting row of data
iCol = 1 ' column of data
iRow2 = 2 'Starting row of data in new sheet

Do Until IsEmpty(Sheet1.Cells(iRow, iCol))
    Sheet1.Rows(iRow).Copy Destination:=Sheet2.Cells(iRow2, 1)
    iRow = iRow + freq
    iRow2 = iRow2 + 1
Loop
End Sub 

Red Flag This Post

Please let us know here why this post is inappropriate. Reasons such as off-topic, duplicates, flames, illegal, vulgar, or students posting their homework.

Red Flag Submitted

Thank you for helping keep Eng-Tips Forums free from inappropriate posts.
The Eng-Tips staff will check this out and take appropriate action.

Reply To This Thread

Posting in the Eng-Tips forums is a member-only feature.

Click Here to join Eng-Tips and talk with other members!


Resources


Close Box

Join Eng-Tips® Today!

Join your peers on the Internet's largest technical engineering professional community.
It's easy to join and it's free.

Here's Why Members Love Eng-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close