Rebuilding My Music Library

[jaswrx]

ok, then back to my point, creating one root folder to store all your music with a few subfolders is just fine. I see absolutely no performance loss and I don't consider to have "enumeration problems, even when there are hundreds of files in one place. I know some pretty big players, and they don't have any subfolders. NONE. Everything stored in one folder. Just saying.... Less of an issue than you claim.

Now regarding software indexing, I gotta say you are the first person that I have heard that has crashed Serato b/c of too many files. Not saying it has not happened, but I don't know anybody else having issues b/c of too many files.

Doesn't sound like any app will load your music so what do you do?

 

[ittigger]

dbstudios just posted that he has as well. This might be a little more common than you think.

Creating one folder with a few subfolders is still a problem - with the issue being if you have several thousand files within a folder, enumeration (from the OS) becomes an issue. The system simply cannot instantly list 10,000 files - that takes time to list and then draw. As I said, at 1000 files, the system begins to take a performance hit (which may or may not be noticeable. As you increase this number, the hit becomes larger.

 

[jaswrx]

It's not common. If it was, there would be outrage and people would be crashing all the time. I am telling you as of now, I am not seeing an issue putting on my files in one folder. I can search in the spotlight just fine, and do it all the time.

It's extremely common for a DJ to store music in one folder with many subfolders. People don't normally spread out folders that contain the same type of content all over the HD like you are suggesting. Talk about disorganized.

I will let you know when it becomes a problem, in that I see the performance slow down.

So what do you do since you cannot load a massive amount of files the system has to index? Don't DJ anymore.

 

[ittigger]

If two people are seeing it, I'm willing to bet alot more are. Maybe they're not saying anything about it. Maybe they've figured it out and play within the lines. I can't explain why others may not have said anything. I do know that until today, no one said it here either - and now you've heard of two. I can't imagine we're the only two.

According to some quick searches, OSX may not break at 10,000 items in a folder - but higher and it does. It has been seen to break at 50,000 items in one folder - in which case, it will never even list the files in the folder. Why is this you ask? Enumeration. It simply cannot list, draw and keep up with the contents of such a folder. Somewhere between 10k and 50k is the answer. Do you want to be the one that finds out - and according to Murphy's Law, you'll find out when you really don't want to.

I can't even fathom trying to navigate and locate 10,000 items in one single location.

Tips To make your Mac faster:
Finder:
By default, your Finder is set to search for files within All My Files view. When files on your Mac are too many, the Finder slows down and opening each new window becomes a burden.

Spotlight:
Just as we did with Finder, you can limit the search area for Spotlight. This should gain your Mac a few points in speed. Go to System Preferences > Spotlight. Now click the Privacy tab. Here you can exclude folders from Spotlight search and thus lighten its load. Just drag and drop folders onto the pane - starting from the most “heavy” ones. As a result, the Spotlight will not be indexing the specified folders and volumes. The less time it spends indexing - the better the performance of your Mac.

This looks like enumeration issues and performance hits to me.

 

[jaswrx]

I have 30,000 in my one folder right now. Don't worry, you will be the first to know if it crashes.

Like you said Serato doesn't work well (breaks) when it gets insanely high so obviously people are not storing 500,000 files to DJ with.

What do you mean by fathom trying to navigate in one location? You search... And it's all organized within Serato via crates.

Yes, changing the finder window to search within the folder makes logical sense, but not bc of enumeration. If I open a window, I want to only search that folder, not the whole HD.

I leave the Spotlight as is. I am not in any shape or form complaining about indexing time. Things are almost instantly accessed, maybe b/c I don't have 400,000 files like you. I am sure it helps that I had a SSD too.

You should spend some time in post productions houses. Granted the hundreds of thousands of files are not stored on the same as the OS ( but external), I have never heard them complain. I mean what you are going to do if you have that many files in one place. You don't have a choice. Move on, even if there is a slight performance hit like you say!

 

[Enchanted Sound]

I feel I have to explain something, I keep all of my karaoke files on an external 1TB HD, so everything runs from that, my computers cannot handle that much internally. I found that my older Dell does not have enough memory (only about 4gigs) which may lead to the issue, but even my newer HP (which has 8 gigs) still slows down considerably when trying to load the files into serato, not to mention a bit of slowing when trying to search. Now, I admit, it could be my computers, but even after serato makes it's log, it still takes time to load the crate every time, due to sheer size. Currently my karaoke file is about 1TB in size, that is a ton of information for any program to work through.

 

[ittigger]

I have 30,000 in my one folder right now. Don't worry, you will be the first to know if it crashes.

Like you said Serato doesn't work well (breaks) when it gets insanely high so obviously people are not storing 500,000 files to DJ with.

What do you mean by fathom trying to navigate in one location? You search... And it's all organized within Serato via crates.

Yes, changing the finder window to search within the folder makes logical sense, but not bc of enumeration. If I open a window, I want to only search that folder, not the whole HD.

I leave the Spotlight as is. I am not in any shape or form complaining about indexing time. Things are almost instantly accessed, maybe b/c I don't have 400,000 files like you. I am sure it helps that I had a SSD too.

You should spend some time in post productions houses. Granted the hundreds of thousands of files are not stored on the same as the OS ( but external), I have never heard them complain. I mean what you are going to do if you have that many files in one place. You don't have a choice. Move on, even if there is a slight performance hit like you say!

I've spent ALOT of time in production houses using all kinds of data. This is how I know about enumeration. I'm a network and system engineer by trade. Enumeration affects all systems. If you've ever seen Sharepoint, that's another great example. Sharepoint works GREAT with properly organized data. Try to store it all in one place and you'll wait for it to load and display. Try to imagine every thread ever posted here being listed on one long page. Do you think it would happen instantly - or do you think there would be a delay?

If you're referring to the index, I load and navigate a 100k song title index without issues within Serato. I have to wait for the index to load but once it's done, I'm ok. Storing in one location = storing in one physical folder. I'm speaking about the OS in this perspective. I'm not talking about Crates here. I'm talking about the physical location of the file.

Example - storing all rock files in the folder rock as:
C:\Rock\Song Title-Artist.mp3

30,000 files in one location like this is alot. The system has to inquire, then list and draw the contents each time you visit this folder. That's alot of processing - hence the performance hit.

The index is one large file - and mine is currently around 100k titles.

Most people that see the performance hit, do move on. Some also look at the way their doing things and see if they could benefit from better performance - and then apply what they think could help them get better performance.

 

[ittigger]

I feel I have to explain something, I keep all of my karaoke files on an external 1TB HD, so everything runs from that, my computers cannot handle that much internally. I found that my older Dell does not have enough memory (only about 4gigs) which may lead to the issue, but even my newer HP (which has 8 gigs) still slows down considerably when trying to load the files into serato, not to mention a bit of slowing when trying to search. Now, I admit, it could be my computers, but even after serato makes it's log, it still takes time to load the crate every time, due to sheer size. Currently my karaoke file is about 1TB in size, that is a ton of information for any program to work through.

Mine are also external as there may be times I need to move the library to a different machine.

 

[Proformance]

Mine are also external as there may be times I need to move the library to a different machine.

If both of you who report the issue use external drives, and the person not experiencing an issue has them on an internal drive - perhaps you should look to the link and ports to your external as a root cause?

 

[DJ Bobcat]

OK... One of the reasons for keeping my DJ libraries as small as possible is because keeping copies updated takes time. Smaller libraries will help a lot when copying a library onto a new PC or a new drive. But another thing I need to think about is reducing the number of copies of the full library. I currently have my main server library from which all copies are derived, and I have a backup server with an exact copy. And here’s the craziness... I have about 10 other copies. While a couple are offsite backups, most of the copies are on my DJ computers. Having smaller libraries will help a lot. I’m also considering putting shared copies of the server library on Small NAS’s, which will be shared by the main and backup computers in both my Large and Small Systems. THAT may not save any copies, though, as I’ll still have copies of the smaller library on each computer, but at least it will keep the number of full library copies down.[emoji4]


Sent from my iPhone using Tapatalk

 

[Enchanted Sound]

perhaps you should look to the link and ports to your external as a root cause?

First of all, the external drives, ports, and links are not an issue, the sheer size of my file is. If I wanted to speed it up, I will have to split my karaoke music into multiple drives, which will still slow Serato down, just possibly not nearly as much. With karaoke music, it's hard to know which songs people will request from one night to the next, so, unlike DJ music, trying to minimize the selection is impossible.

 

[ittigger]

If both of you who report the issue use external drives, and the person not experiencing an issue has them on an internal drive - perhaps you should look to the link and ports to your external as a root cause?

Bob, you're not an IT guy, so I expect you wouldn't understand some of these things.

First, if you do some simple searches regarding large amounts of files and any application, you will see crashing.

Additionally, if you did some more simple searching, you would see that DJ software in general (and any application really) has problems with large amounts of data. Serato specifically has been documented as having issues with large amounts of data - since .. well forever - as some users have always had large libraries. As time has gone on, the library limit has increased but it still has a working limit. This means that this problem is not exclusive to the both of us. I also currently use Virtual DJ - guess what happens with large libraries there and is a known issue? Go ahead, guess.

Everything has limitations.

In my instance, it is the sheer size of data, which is why my main library is not the entire library. Jas has not seen this problem simply because he does not have as many files - and he's admitted this. However, if he jams a large amount of files in one directory, he will see the enumeration issue which affects all systems. A system can only process so much information so fast - and when you tell it to process more than it can, the result is a performance hit.

Additionally, until they have reliable cost effective several TB internal drives, there is no way for you to store this amount of data on a laptop's internal drive - which would gain you minimal performance increases because the bottleneck is not the drive or the port - it's the processing of a large amount of data.

 

[ittigger]

OK... One of the reasons for keeping my DJ libraries as small as possible is because keeping copies updated takes time. Smaller libraries will help a lot when copying a library onto a new PC or a new drive. But another thing I need to think about is reducing the number of copies of the full library. I currently have my main server library from which all copies are derived, and I have a backup server with an exact copy. And here’s the craziness... I have about 10 other copies. While a couple are offsite backups, most of the copies are on my DJ computers. Having smaller libraries will help a lot. I’m also considering putting shared copies of the server library on Small NAS’s, which will be shared by the main and backup computers in both my Large and Small Systems. THAT may not save any copies, though, as I’ll still have copies of the smaller library on each computer, but at least it will keep the number of full library copies down.

I think a redundant copy of the full library should be sufficient while copies on the systems of the 'active' library may be better.

 

[dunlopj]

Whew....am I glad that my slightly OCD brain made me organize my music in a very favorable, well designed, well thought out and very organized manner.

Ask my wife as I spend way too much time adding new purchased music, tagging it just so and adding it to my 3 separate backups as well as my main PC.

I'm so proud of it...

 

[Proformance]

First of all, the external drives, ports, and links are not an issue, the sheer size of my file is.

The problem with the absolutes being expressed in this thread is that they do not coincide with the experiences other people are having with the specified conditions. It follows therefore, that your confidence might be misplaced.

Bob, you're not an IT guy, so I expect you wouldn't understand some of these things.

I understand problem solving quite well, and I think your methods are flawed.

Additionally, if you did some more simple searching, you would see that DJ software in general (and any application really) has problems with large amounts of data.

Every system ever created has limitations. The problem therefore, is easily re-framed as originating with your data not the system. This is not an "IT problem" it is a DJ entertainment problem - and that is my area of expertise.

It takes less than 5,000 discreet music or karaoke files to accommodate 90% of all requests a mobile DJ will encounter in his own market. In a 4 hour event with at least 3 hours of dancing, a DJ will turn away 6 people or less if programming entirely by request. We know however, that a mobile gig is rarely programmed entirely by request so, more often we turn away no one at all.

That reduces this problem to one simple question also posed to the OP: WHY?

WTF are you wasting your time with such large collections of unnecessary files? Professional DJs don't behave this way because, more is not better.

 

[DJ Bobcat]

That reduces your problem to one simple question also posed to the OP: WHY?

Normally, I wouldn’t even respond to posts that are so totally worthless.[emoji1] But if this idiot was standing in front of me asking this question, I’d SLAP HIM UPSIDE THE HEAD and say; “You STUPID IDIOT, go back and read the second paragraph of the OP’s first post.”

WTF are you wasting your time with such large collections of unnecessary files? Professional DJs don't behave this way because, more is not better.

And to THIS I would simply say; YOU may not BEHAVE this way, but there are as many ways DJ’s behave, as there are DJ’s. If I had to PICK one “PROFESSIONAL” DJ behavior to emulate, yours would be the LAST one I’d pick. No, that’s not true. If I had to emulate YOUR behavior, I’d simply get out of the DJ business.[emoji1]

 

[Proformance]

OK... One of the reasons for keeping my DJ libraries as small as possible is because keeping copies updated takes time. Smaller libraries will help a lot when copying a library onto a new PC or a new drive. But another thing I need to think about is reducing the number of copies of the full library. I currently have my main server library from which all copies are derived, and I have a backup server with an exact copy. And here’s the craziness... I have about 10 other copies. While a couple are offsite backups, most of the copies are on my DJ computers. Having smaller libraries will help a lot. I’m also considering putting shared copies of the server library on Small NAS’s, which will be shared by the main and backup computers in both my Large and Small Systems. THAT may not save any copies, though, as I’ll still have copies of the smaller library on each computer, but at least it will keep the number of full library copies down.

You are not speaking like a disc jockey. You sound more interested in running a server farm. I think if you truly want to do this right from a DJ perspective you need to get real about why and what it is you truly value - Do you want to be an effective DJ or the world's foremost data archivist?

If it's DJing that you care about then you need to realize that your music collection is a tool that needs regular sharpening.
It's not a "library" and you should not archive things that serve no purpose. We have iTunes, Amazon, and an entir emusic industry to do that archiving for us.

Here are some goals if you're serious about DJing professionally:

1.) Leave the full main library alone.
It's just a place to go looking for future odd ball requests so, you won't have to buy them again. One copy is enough. If you lose it, you will have lost nothing of serious value.

2.) The entirety of your working collection should be under 230GB of storage space
If it's not, then stop using uncompressed file formats and get rid of all the useless crap you never play. If you need NASA's mainframe to process your library then you are not in the DJ busness, you are archiving pop nostalgia for future alien anthropologists. Have 3 copies. One you work with, one that travels with you as a backup drive, and the third never goes anywhere, it's your fall back if you lose both of the others. You're not holding the nuclear launch codes so, by the time you get to 10 copies and off site storage you're abusing the copyrights.

3.) The entirety of a karaoke library should fit within 60GB of storage
This one requires rehab because KJs even more than DJs tend to be emotionally handcuffed to the size of their..... collection.
Again, the files need to be compressed and/or zipped and you need to eliminate the stuff no one wants to sing.

It takes so little power to run a music and karaoke show. It can literally be done flawlessly on oudated XP machines so, if someone is choking the most modern hardware and DJ software they are doing something wrong.

 

[ittigger]

The problem with the absolutes being expressed in this thread is that they do not coincide with the experiences other people are having with the specified conditions. It follows therefore, that your confidence might be misplaced.

Actually they do, your ignorance of such is your own self imposed problem.

I understand problem solving quite well, and I think your methods are flawed.

We know your methods are flawed. I've been successfully problem solving for several decades. You're trying to discuss a world that you do not understand. Stick to the shallow end of the pool Bob.

Every system ever created has limitations. The problem therefore, is easily re-framed as originating with your data not the system. This is not an "IT problem"

Every system created does have limitations - and those limitations are software and hardware driven. This makes it an IT problem. I'd love to see you attempt to explain to a records center that they don't actively use all their records, so just get rid of what you don't use all the time. You wouldn't make it to the end of Day 1.

it is a DJ entertainment problem - and that is my area of expertise.

Actually it's not your area of expertise. This would be a professional with ethics, of which you are neither. Come to think of it, an 'expert' is one who knows more and more about less and less until they know absolutely everything about absolutely nothing. You're right Bob, you are an expert.

It takes less than 5,000 discreet music or karaoke files to accommodate 90% of all requests a mobile DJ will encounter in his own market. That means in a 4 hour event with at least 3 hours of dancing programmed entirely from live requests the DJ will turn away 6 people or less. We know however, that a mobile gig is rarely programmed entirely from requests so, more often we turn away no one at all.

This means nothing.

That reduces your problem to one simple question also posed to the OP: WHY?

This really is not your concern. I know you like to tell others to research their answers but do not do your own research. Many people have large libraries. End of story.

WTF are you wasting your time with such large collections of unnecessary files? Professional DJs don't behave this way because, more is not better.

I don't think you have any say as to what a professional is or does. The size of someone's library does not make them any more or less a professional. Professionals have ethics and do things in accordance with laws and policies.

 

[DJ Bobcat]

You are not speaking like a disc jockey. You sound more interested in running a server farm. I think if you truly want to do this right from a DJ perspective you need to get real about why and what it is you truly value - Do you want to be an effective DJ or the world's foremost data archivist?

If it's DJing that you care about then you need to realize that your music collection is a tool that needs regular sharpening.
It's not a "library" and you should not archive things that serve no purpose. We have iTunes, Amazon, and an entir emusic industry to do that archiving for us.

Here are some goals if you're serious about DJing professionally:

1.) Leave the full main library alone.
It's just a place to go looking for future odd ball requests so, you won't have to buy them again. One copy is enough. If you lose it, you will have lost nothing of serious value.

2.) The entirety of your working collection should be under 230GB of storage space
If it's not, then stop using uncompressed file formats and get rid of all the useless crap you never play. If you need NASA's mainframe to process your library then you are not in the DJ busness, you are archiving pop nostalgia for future alien anthropologists. Have 3 copies. One you work with, one that travels with you as a backup drive, and the third never goes anywhere, it's your fall back if you lose both of the others. You're not holding the nuclear launch codes so, by the time you get to 10 copies and off site storage you're abusing the copyrights.

3.) The entirety of a karaoke library should fit within 60GB of storage
This one requires rehab because KJs even more than DJs tend to be emotionally handcuffed to the size of their..... collection.
Again, the files need to be compressed and/or zipped and you need to eliminate the stuff no one wants to sing. It takes so little power to run a music and karaoke show. It can literally be done flawlessly on oudated XP machines so, if someone is choking the most modern hardware and DJ software they are doing something wrong.

LOL... Well I gotta admit, you DO make me laugh. The degree of RIDICULOUSNESS in this post is OFF THE CHARTS!!! It would be nice if I could put an end to it, and though I realize it's not possible where you are concerned, I'm gonna give it a try.

First of all, to say that one cannot be "an effective DJ" AND "the world's foremost data archivist" at the same time is among the most ridiculous notions I've EVER seen posted. It's like saying Wayman Tisdale could not be an All Star basketball player AND a prominent professional jazz musician. People can be good at more than one thing... I may have a little pea brain, but I use ALL facets of it.

Now let's get to the other REALLY RIDICULOUS part of your post...

...Again, the files need to be compressed and/or zipped.

^^^^ This is about the most worthless advice you've given lately... Right up there with your LEGAL advice. Just to prove how worthless it is though, I did a quick sample to demonstrate. I took a music file...



... and made a copy of it. Then I turned on file compression...



... Look at the difference in size between the copy and the original...



... You'll notice there's NO DIFFERENCE!!!

You'll also see there's just a VERY small difference between the original and a ZIPPED version. So small that it makes no sense to do it. In fact, you have to unzip it to play it, or you get something like this...


... so what's the point???

First of all, MP3 files are ALREADY compressed. Compressing them further just means an additional loss of quality. Zipping files only removes UNUSED data, and there's not much unused data in audio and video files, AND they must be unzipped to be played or viewed, so you basically end up with a zipped copy that's about the same size as the original AND you have to have the unzipped original in order to play it, so you basically DOUBLED the amount of file space you're consuming for something you can't even use. THAT's CRAZY!!!

FINALLY...

...If it's DJing that you care about then you need to realize that your music collection is a tool that needs regular sharpening.It's not a "library".

The notion that my music library is NOT a library is ludicrous. Go google "music TOOL" or "DJ TOOL" and see if you get anything resembling a database of music. Then try "music LIBRARY".

This thread was created to talk about ways to get my MUSIC LIBRARY down to a more reasonable size. I said from the very beginning it is was filled with entire albums that would likely NEVER be played at a gig. I was NOT a DJ when I digitized my music collection. I'm certainly NOT the first guy who has done this, and others can relate. For any NOOB, I would recommend building a collection from scratch, rather than using your personal music library. That's what I shoulda done, but... that's NOT what I done! I'm fixin' it.

 

[Enchanted Sound]

For any NOOB, I would recommend building a collection from scratch

What I did with my music LIBRARY, is take the current CD's that I own, and use that as a starting point for my collection, and will PURCHASE my music as needed. As far as my karaoke collection, it was collection that I purchased over a few years, as zip files, but serato will not read them, so I had to unzip them on to a separate drive. Yes it's large, but it's all legal. With karaoke, yes it's only 4 hours, but the crowd is a lot more diverse, which means I will probably use 90% of my library over the period of a few months.