Hard Drive File Search Tools
Hard Drive File Search Tools
(OP)
At work we have a library of reference data, which all of our 30+ engineers can access and refer to as they work.
It contains design manuals, regulatory guidance, vendor datasheets, industry standards, etc. The library has the following approximate dimensions:
120 GB, 170,000 files, 7200 folders
It grows significantly every year.
I have found that newer members of our engineering staff, not familiar with our knowledge base, have trouble using it.
For our younger engineers, it is a matter of believing the information exists and/or that we could possible know something that Google doesn't.
For our older engineers, like me, when things are stored on a path I wouldn't expect to follow, I have trouble finding other people's additions.
There was a time that Google Toolbar could be used to search and efficiently find information in the library, but GT has gone by the wayside.
To find things now, we rely on a rough system of organization... and the horrible windows search bar above the folder viewer.
I've been looking at file search tools and found some candidates - but I thought I'd ask if Eng-Tips members have any recommendations before I commit?
"Copernic" and "UltraFileSearch" are the two contenders I like.
It contains design manuals, regulatory guidance, vendor datasheets, industry standards, etc. The library has the following approximate dimensions:
120 GB, 170,000 files, 7200 folders
It grows significantly every year.
I have found that newer members of our engineering staff, not familiar with our knowledge base, have trouble using it.
For our younger engineers, it is a matter of believing the information exists and/or that we could possible know something that Google doesn't.
For our older engineers, like me, when things are stored on a path I wouldn't expect to follow, I have trouble finding other people's additions.
There was a time that Google Toolbar could be used to search and efficiently find information in the library, but GT has gone by the wayside.
To find things now, we rely on a rough system of organization... and the horrible windows search bar above the folder viewer.
I've been looking at file search tools and found some candidates - but I thought I'd ask if Eng-Tips members have any recommendations before I commit?
"Copernic" and "UltraFileSearch" are the two contenders I like.
STF






RE: Hard Drive File Search Tools
Attached is screen shot...
Should have added... you can do a dos (command) dir to list the contents by sorted filename only and can highlight some of the most important ones.
Dik
RE: Hard Drive File Search Tools
I find it is valuable to extract file textual contents for inclusion in pages, saving the time required to fire up some application, though the page still retains a link to the original document - this is where the embedded links are placed. I have had no problem loading the equivalent of several hundred MIL standard pages as a single Wiki page; I have found that for some it still makes sense to break them out by chapter.
Another wonderful feature is that a Wiki keeps history, eliminating having to create new file names for every version. The individual pages also get not only histories, but can also have difference reports to see what changed between any pair of versions (another reason for text extraction.)
RE: Hard Drive File Search Tools
Good idea.
Dik
RE: Hard Drive File Search Tools
https://www.pcmag.com/article2/0,2817,2399582,00.a...
RE: Hard Drive File Search Tools
Why do you find it horrible?
Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/
RE: Hard Drive File Search Tools
I'd forgotten that, but I blogged about it (with a solution) at:
https://newtonexcelbach.com/2017/03/04/indexing-pd...
Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/
RE: Hard Drive File Search Tools
IDS, thank you for the tip on the PDF searches. About 95% of the files in my library are PDF's.
3DDave,
I'm a bit confused by your suggestion - I am probably too ignorant to understand it. My mental image is that I would view a HTML page with a web browser, using local file addresses rather than an internet address, reading a bunch of subject-specific pages that I and my colleagues have spent time collectively writing and editing over a period of months or years. Like Wikipedia for our network servers. If you could convince me that this would be a productive pursuit then I'd be happy to start now. Unfortunately, I lack the charisma to convince anyone else in the department, so the effort to write all this would be entirely mine. If I have completely misunderstood, I hope you can forgive me.
Dik,
DOS list is pretty much what I do now!
STF
RE: Hard Drive File Search Tools
It is free from mythicsoft.com, for personal and commercial use.
It has a big brother paid app, FileLocator Pro, which is claimed to do even more, so it must be incredible.
My wife is a total computer Luddite, who got herself a job as a secretary, of all things.
I was able to walk her through:
Finding mythicsoft.com,
Downloading Agent Ransack,
installing it,
and using it to find a bunch of files that were mis-filed by someone else,
just talking to her over the phone.
I can think of no higher accolade.
Mike Halloran
Pembroke Pines, FL, USA
RE: Hard Drive File Search Tools
This is still an improvement, but there are still some quirks.
Text within a PDF can be found only if I have opened the file. If the file has not been opened, then the text string is not found.
For example, the word "toughness" appears in most of the 40 files I have in a particular material properties folder.
I know this because all of the files were generated by the same source (Boeing) and follow the same format, just for different materials.
I have opened 4 of the files to confirm that my memory is not faulty.
When I do a windows search in this folder, the only files that appear in the result are the 4 files I have already opened.
Checking with task manager/resource manager, I briefly saw the indexing service looking through the files as I accessed them, but the rest are ignored. In the time it's taken to write this up, the indexing service has stopped indexing again. So... the only way to index all of the PDF's is to open every. single. one.
I'm considering the need to "Rebuild Index", which is a button in the control panel menu that I previously ignored. Just have to give it the time it needs to run the whole index all over again.
STF
RE: Hard Drive File Search Tools
Wikipedia is a particular wiki. Most of them are in the CMS, Content Management System area. They manage content. You have content that you need to be managed so that users can find what they are looking for. My guess is that it's less than 1% of what you have now with no traceability to any projects you use it for.
If you create any documentation then that usually has some basis. So a bracket might have a material spec, a finish spec, a next assembly, a requirements document, a stress report, some meeting notes relating to the bracket, an outside supplier, and so forth.
Here's the typing that might be required for part XYZ:
((SAE AMS QQ-A-250))/11
((finish spec))
((next assy))
((XYZ Requirements))
((XYZ Stress Reports))
((XYZ Meeting Notes))
((link to PDF file of drawing))
Supplier: ((CAGE Code))
Responsible Engineer: ((Bob the Builder))
Contract: ((123xyz))
Everything in (( and )) is a link to a page. If the page doesn't exist, when it is selected it asks if you want to create the page. So it's easy enough to create scaffolding; unlike some systems which won't let connectors be created without a place to connect to.
Or, however you care to look at the data. Maybe you have an ERP system that does some of that. So skip it; this is an example of what it could be used for, not what you have to use it for.
The one I use is TikiWiki. https://tiki.org/Features There are others.
Perhaps, as Henry Ford suggested, you are just looking for a faster horse.
RE: Hard Drive File Search Tools
TTFN (ta ta for now)
I can do absolutely anything. I'm an expert! https://www.youtube.com/watch?v=BKorP55Aqvg
FAQ731-376: Eng-Tips.com Forum Policies forum1529: Translation Assistance for Engineers Entire Forum list http://www.eng-tips.com/forumlist.cfm
RE: Hard Drive File Search Tools
Dik
RE: Hard Drive File Search Tools
OK, but it's indexing of the content of pdfs and Office files that I find really useful, and once it is set up properly, Windows does an excellent job of that.
Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/
RE: Hard Drive File Search Tools
Forget it!
3DDave, I was thinking of an Access database, so thank you for giving more detail on the wiki. Now I'm picturing a project-specific or part-specific page of designer's notes, which cross-reference the source data (wherever it's stored the link can be made) that the designer used. OK now I think I have a more complete picture. I write notes like that all the time, often in Notepad, and including knowledge base links as I use them. No it's not clickable links, but cut-and-paste is so easy for a keyboard-oriented user like me. The wiki idea seems better as a tool to cultivate in all of my colleagues, the habits to make notes during their design process, and to do it in a way that others can benefit from... yes the philosophy of wiki's seems to be sinking in.
STF
RE: Hard Drive File Search Tools
I am in the process of upgrading to version 7, "In the process" because I have struck a few difficulties in getting it to activate its "extensions". My current thinking is that this is because my Windows-10 computer somehow acquired an invalid name when it was set up, and Copernic's licensing / security (extensively revised for version 7) makes use of a computer's name. I am getting good help from Copernic Support, but our diametrically opposed time zones mean we can only achieve one question / answer per 24 hours.
One thing I have already noticed about Copernic v7, as a consequence of my difficulties activating its extensions. Its free version is much less capable than was the free version two decades ago.
RE: Hard Drive File Search Tools
Does it indicate that indexing is complete?
Are all the folders that need to be indexed included in the list of indexed locations?
Under "Advanced" do all the file extensions that should have content indexed have an appropriate filter? e.g. "PDF filter" not "File Properties Filter"
If those three things are complete or set correctly, I don't know why the index would be incomplete.
Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/
RE: Hard Drive File Search Tools
There is also the consideration that should be given right from the outset, as to how many users will need to access the system at once. This is a great drawback to the typical Access DB, normally it gets stored as a file somewhere, and simultaneous access to the DB ends up breaking it. The better alternatives (e.g. MySQL, PostGreSQL) involve setting up some sort of server specifically to handle it, which involves much more effort, and much negotiation with IT. They come with a steep learning curve for beginners to DBs too.
Content / Document Management Systems can get around some of the initial setup of the DB, and can negate the issues of files shifting around providing they are set up to be the only means of locating the particular file. They don't work for every content type (SolidWorks / Inventor / SolidEdge are good examples of ones that don't) unless specifically configured to manage them, and can often come with their own issues, particularly if not set up properly. They will generally index far faster than a Windows Explorer interface though, and can allow for searching via more metadata than is exposed to the filesystem.
Having used a number of different configurations (user editable wikis, Sharepoint, Document Management Systems, and good old Windows Explorer) I am aware of a lot of the pitfalls of each system. Locking down file share systems greatly limits the potential damage due to cryptolocker issues too.
EDMS Australia
RE: Hard Drive File Search Tools
Freddy, and Doug, I will not ask for support from IT, given that they seem to be understaffed already.
Currently, I am trying out ideas on my home computer. It's a pretty good sandbox for testing, because I have a bit of a library of my own.
My home computer is Win7, the ones at work are Win10. That may make a difference in how the file indexing service works.
At work, we do have content management for our approved and controlled design data (Autodesk Vault). The issue at hand is for data not controlled by this system, and probably shouldn't be controlled by it either. Since the Vault is currently only configured to manage Autodesk and MS Office data files, I would be imposing more IT support to extend it to the zillion PDF files floating around.
Doug,
Yes, and yes, although I see no part of the dialog boxes that attempts to indicate either the status of the indexing, or whether it is complete or not. It does have a button that lets me pause it for 15 minutes. So far, I have been using Task Manager/Resource Monitor to determine if the search index process is even running. Usually it's minimal. Actually, it's funnier than that: whenever I access Resource Monitor and select the Search Protocol Host, it's usually running but not doing anything. If I leave it selected for a minute, it suddenly "gets busy"! There is a faint hope, that if I give it a few months, it will eventually index everything on my hard drive.
IRStuff,
Thank you. Immediate results, if only for the files that actually have descriptive filenames. A lot of them do, so it's a great stopgap until I can choose a content searcher.
I'm about to try UltraFileSearch and Copernic (free trials) next.
STF
RE: Hard Drive File Search Tools
Can even cross-search file name terms with file content terms, but it takes longer. After 15 minutes it's searched about 16,000 files and found plenty of correct hits.
With 170,000 files to search, it may take 2 hours to finish.
The search seems to have started with the oldest files, so the last files to be found will probably be my most recent additions.
I was thinking it might also be useful for my photo collection... but probably too slow for that, too.
RE: Hard Drive File Search Tools
RE: Hard Drive File Search Tools
I let UltraFileSearch run overnight with a file content search parameter, and received all of the expected results. It also turned up some results that I didn't expect, all of which contained material on the desired subject, which is EXACTLY the point, so I'm pretty happy with it.
STF
RE: Hard Drive File Search Tools
It would appear to me, that for 30+ engineers, the file indexing tool you're testing would need to be installed on each machine in order to work the way you're expecting. That would likely have licencing implications, as well as the possibility of increasing network traffic and inadvertently slowing down the file access.
EDMS Australia
RE: Hard Drive File Search Tools
RE: Hard Drive File Search Tools
Your examples of Vault is exactly what I was talking about when I mentioned being the only means of locating a file, if a user can't access the file storage, then they can't break the link.
EDMS Australia
RE: Hard Drive File Search Tools
RE: Hard Drive File Search Tools
Fortunately we also have a wiki that allows us to embed URLs to those Sharepoint docs. So the big MS docs aren't actually lost. We effectively hand-index them in wiki pages that mention them in context.
Steve
RE: Hard Drive File Search Tools
Reports that Windows search didn't index files that hadn't been opened, I did some checks on my system (Windows 10). I copied a .doc file to an un-indexed location, made some changes, then saved it with a new name as a .doc and .pdf, copied the new files to the original folder, observed the index update process, and did some searches. What I found was:
1. Reindexing the new files (about 5MB each) took about a second (see screen-shot below).
2. Searching for old content found it in all 3.
3. Searching for words only in the new files found it in both of them, even though neither had been open in an indexed location.
4. It looks like only single words are indexed. If you enter a phrase it will find all files containing any of the words in the phrase, but it doesn't seem to work for an exact phrase in "".
5. It indexes personal names, but not non-dictionary words. I have no idea how comprehensive its name list is.
6. You can sort the list by relevance, which seemed to work pretty well.
7. If it finds files with the search term it updates almost immediately, then continues searching; I presume in non-indexed folders.
I do recall being just as unimpressed with Windows search at one time as others here, but I think more recent versions are vastly improved. I don't recall if there was a big change between Windows 7 and 8, but I think the 8 version worked much as in 10.
Regarding monitoring the indexing, the Indexing Options dialog now looks as shown below. If you copy new files to an indexed folder it briefly notifies that it is indexing, then shows the revised number of indexed items.
Doug Jenkins
Interactive Design Services
http://newtonexcelbach.wordpress.com/
RE: Hard Drive File Search Tools
Oh, now I see. When you first suggested it, I thought the wiki operated like some kind of interface that writes the HTML code for me (like Wordpress or Frontpage) but just in the background. So I did not believe it could deal with the file organization. This isn't what you're talking about at all. I was already a little daunted by all the page writing to make a readable Wiki, but now I realize that this would definitely require IT support, in the creation of a separate server online for the users to access. Eventually it would randomize all of the semi-ordered file structures currently in place. Access to the data would require the index (though admittedly the index seems robust).
It might help to illustrate the file structure I currently use, which I've talked about but maybe haven't made clear to everyone:
For example, to find a PDF copy of Peterson's Stress Concentration Factors,
- ..\Engineering\Textbooks\Peterson\Peterson - Stress Concentration Factors (1st Ed).pdf
I don't really need to do a search for "Peterson" since I can just click on the textbooks folder, and pick the right book.Another; ESDU Paper 68002, SHAFTS WITH INTERFERENCE-FIT COLLARS,
- ..\Engineering\Design_Manuals\ESDU Papers\ESDU_by_Subject\Mech Systems\68002.pdf
This is a bit more of a challenge, since the ESDU papers themselves are not sorted in any particular order, nor given any useful names, except the files I have renamed. So I made the index easy to find first, and from there all other subjects and titles within the document numbers can be found.Hopefully, this give an idea that the documents have been organized in some fashion, even if calling it "curated" would be an exaggeration. A user can select a general subject, and then narrow it down.
Some might find my insistence on referring to DOS computer folders to be an obstacle. I may have to concede that using DOS file structure to organize documents whose subjects are interwoven is both old-fashioned and difficult for noobies to absorb. In my defense, my system of organization is modeled after two other similar reference file libraries that I was able to learn, and use efficiently on the machines of others. I later copied much of this system of organization for my own purposes.
Hi Doug,
No, your persistence is greatly appreciated!
I accepted long ago that I will be using Windows for maybe the rest of my life, so it behooves me to learn to use it as effectively as possible.
Here's a screen-shot to match yours:
Sorry, I didn't notice the status text at the top of the window, not until you posted your screen-capture.
"Indexing speed is reduced due to user activity."
Searching this subject turned up these links:
- https://www.neowin.net/forum/topic/613825-can-i-fo...
- https://social.technet.microsoft.com/Forums/window...
Which suggested this registry edit:- HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Search\Gathering Manager
- DisableBackoff = 1
The service (or the computer) has to be restarted after the registry change.Later comments on the Microsoft forum recommend turning it back OFF again once it's done, or it could unnecessarily slow the whole system down.
I'll give this a try, but it means I have to hit "Submit" before I reboot.
STF
RE: Hard Drive File Search Tools
"Indexing in progress..." Better than before.
To do this, the Regedit didn't work. Not sure what was really wrong but I suspect the HKEY was the WRONG key to edit for my version Windows 7.
Instead, after a bit more searching the 'net I found this: https://superuser.com/questions/234211/can-i-force...
The indexing service should be stopped while the Group Policy setting is changed, then restarted for the change to take effect without a windows re-boot.
The indexing service is now reading through about a MB per second so maybe I will have access to better search results soon.
STF