Dealing with lots of files

With a cluster such as Eddie it is very easy to generate tens of thousands (or millions) of files. The problem is that the standard tools for listing and manipulating files tend to have problems when the number of files exceeds approximately 1000.

Suppose on has a directory with a lot of output files in and you wish to list them 

$ ls /exports/work/mygroup/myspace/data

This command will either fail or take a very long time to run due to the shell globbing problems.

However, the following will work 

$ find /exports/work/mygroup/myspace/data

The ability within unix type systems to chain commands together allow rather useful tools to be made

To find the number of files in a directory use the following command 

$ find /exports/work/mygroup/myspace/data | wc -l

Dealing with lots of files (last edited 2007-06-15 13:45:28 by JohnBlairFish)