Since most of us grew up in a word of DOS and Windows, our desire for folders and tree-like hierarchical structures is vast. We just make folders and subfolders to categorize information it has become … natural. Most of us do not want to let folders go, even when migrating to a new world like document management and SharePoint.

SharePoint provides the functionality to create folders within document libraries and so the most natural impulse when migrating from that dusty file server, is to just recreate the folder structure.
But to really get the most out of MOSS one must utilize metadata (custom columns). Metadata create a dynamic categorization system within document pools and allow for infinite personalization through views. Metadata use also turn search from guesswork to an intuitive task.

To be frank the folders we all use in file systems are metadata for the files they contain, just single dimensional. The “Projects” folder denotes that all documents contained in it are project related documents. Since this is not enough we rely on subfolders (client name or project type perhaps) and info in the filename (specific project name f.e.) to add more “metadata” to the document. This process that can quickly be chaotic and as the number of files grows it becomes unusable.

This can easily be alleviated by using a true metadata structure in SharePoint (two custom columns “Client” and “Type” in the ProjectFiles document library would be enough for the above example). So the document visibility is increased but also the structure can be “re-organized” (through views and filtering) without any manipulation of the documents files themselves. Teams working on the same file-set can also benefit from this approach. So the Sales department can view documents sliced per client or budget, but the IT department can view the documents per Product or Project type. Metadata is the way to keep everyone happy while at the same time avoiding multiple copies of files (and emails flying around trying to find the correct version).
Another thing against folders is that even if they are there, they cannot be used for filtering.

Just to add a technical limitation to all the business and design ones, the maximum document url length in SharePoint is 260 chars. This can be easily reached if the existing folder structure is big or really descriptive.

So it is obvious. No folders in document libraries, just metadata. Well … there is one single detail that folders have on their side, you can define access rights on them. So if you want to reduce the number of required document libraries or you have trouble dealing with access rights maybe a single layer of folders is ok. You will have to add metadata anyway probably just to get the views your users will desire (yes you can add metadata to a folder through customization, but it is not copied automatically to the documents that are placed inside it).

To conclude, in my opinion try to avoid folders at all cost unless access rights management makes them an unavoidable necessity.