Scanner changes in 7.4

[andyg]

I've changed all scanning in the current 7.4 over to the new scanner
code I've been working on. I expect there will be bugs. I've started a
wiki page that attempts to describe how the new scanner in 7.4 works,
please read it and let me know if you need more info or have problems.

http://wiki.slimdevices.com/index.php/NewScanner

[Phil Meyer]

Post by andyg
I've changed all scanning in the current 7.4 over to the new scanner
code I've been working on. I expect there will be bugs.

Are you saying this is now merged to 7.4/trunk?

Post by andyg
I've started a wiki page that attempts to describe how the new scanner in 7.4 works,
please read it and let me know if you need more info or have problems.

Is MusicIP interraction still supported? I think I read somewhere that it didn't work in the noweb-sqlite branch?

I guess that MySQL support is still missing? Any adeas of a timeline for that? Until it is supported, or all plugins have been converted, I can't use this as my day-to-day server, and have to run SQLite version in parallel for beta testing.

Does anyone know of a decent (preferably freeware) GUI SQLite query analyser? I haven't found a decent one yet - running queries and capturing results with the console sqlite.exe command is such a chore.

Do you think that some existing bugs are now fixed? eg. if compilation status is changed, a scan for new/changed files will actually correctly apply the change to the DB?

Unsure from the page how auto rescan works. Is this a mode that can be enabled, so the library is automatically kept up to date? Does it automatically kick in the instant it sees a single file has changed? Other apps have a delay, such that it will start scanning a minute or so after it has detected no further file updates (to avoid file system thrashing, and less likely to get things wrong where songs on the same album are written back in a batch).

Rescannng in process (rather than spawning another process) - what modes do this, is it controllable? I think scanning in another process was better than in the server process, to avoid drop-outs in the music. Is in-process using another thread now?

[Andy Grundman]

Post by Phil Meyer

Post by andyg
I've changed all scanning in the current 7.4 over to the new scanner
code I've been working on. I expect there will be bugs.

Are you saying this is now merged to 7.4/trunk?

No, sorry I wasn't clear. This is only in SQLite and the nightlies.

Post by Phil Meyer

Post by andyg
I've started a wiki page that attempts to describe how the new scanner in 7.4 works,
please read it and let me know if you need more info or have
problems.

Is MusicIP interraction still supported? I think I read somewhere
that it didn't work in the noweb-sqlite branch?

MIP should still work as before, although I haven't tested it in the
past few weeks.

Post by Phil Meyer
I guess that MySQL support is still missing? Any adeas of a
timeline for that? Until it is supported, or all plugins have been
converted, I can't use this as my day-to-day server, and have to run
SQLite version in parallel for beta testing.

No MySQL support yet, sorry :(. Maybe I'll try to hack on this some
evening. But please don't avoid SQLite if your only concern is
performance. It performs very well in my experience. And yes my scans
are consistently faster than in 7.3.3. I meant to post some benchmarks
on the wiki page, will do that a bit later.

Post by Phil Meyer
Does anyone know of a decent (preferably freeware) GUI SQLite query
analyser? I haven't found a decent one yet - running queries and
capturing results with the console sqlite.exe command is such a chore.

If you Google for SQLite GUI you should find some. There is even one
that is a Firefox plugin.

Post by Phil Meyer
Do you think that some existing bugs are now fixed? eg. if
compilation status is changed, a scan for new/changed files will
actually correctly apply the change to the DB?

Not sure about this specific case but this is exactly the kind of bug
I want to fix. Some such as changing genres or years are already
handled. This is one of the areas where I really need people's help,
trying common use cases for changing tags.

Post by Phil Meyer
Unsure from the page how auto rescan works. Is this a mode that can
be enabled, so the library is automatically kept up to date? Does
it automatically kick in the instant it sees a single file has
changed? Other apps have a delay, such that it will start scanning
a minute or so after it has detected no further file updates (to
avoid file system thrashing, and less likely to get things wrong
where songs on the same album are written back in a batch).

Right now it's always enabled for Mac and Linux with local files. I
plan to make it a pref though. There is a built-in delay to handle
exactly the situation you describe. The rescan kicks off 15 seconds
after the last event is received.

Post by Phil Meyer
Rescannng in process (rather than spawning another process) - what
modes do this, is it controllable? I think scanning in another
process was better than in the server process, to avoid drop-outs in
the music. Is in-process using another thread now?

This needs testing to see if there is any impact. If it's a problem we
can easily have the auto-rescan kick off a new process. In-process is
supported for low-memory environments mostly.

[Phil Meyer]

Post by Andy Grundman
No, sorry I wasn't clear. This is only in SQLite and the nightlies.

Ah good, I can still update 7.4/trunk then! ;-)

Post by Andy Grundman

Post by Phil Meyer
Is MusicIP interraction still supported? I think I read somewhere
that it didn't work in the noweb-sqlite branch?

MIP should still work as before, although I haven't tested it in the
past few weeks.

Great - I'll try it out.

Post by Andy Grundman

Post by Phil Meyer
I guess that MySQL support is still missing? Any adeas of a
timeline for that? Until it is supported, or all plugins have been
converted, I can't use this as my day-to-day server, and have to run
SQLite version in parallel for beta testing.

But please don't avoid SQLite if your only concern is performance.

I have a secondary server set up using noweb-sqlite. Will try comparing performance when I get the same library content in both servers.

I will need to disable plugins in my main server for memory/speed performance comparisons.

It's not just performance - I rely on some third-party plugins that won't work with SQLite.

Post by Andy Grundman
It performs very well in my experience. And yes my scans
are consistently faster than in 7.3.3. I meant to post some benchmarks
on the wiki page, will do that a bit later.

I thought the compiled scanner were the main reason for faster scanning. Other people have posted that performance is worse.

Better/more important performance tests will be to run searches, browse album lists, etc. I'm not too bothered about scanning performance (or I wasn't when it was a separate process).

Post by Andy Grundman
If you Google for SQLite GUI you should find some. There is even one
that is a Firefox plugin.

I tried some; didn't work too well. I saw the Firefox plugin one, and saw it wasn't maintained any more. Seemed a bit bizzarre. I guess it's because firefox bookmarks used to be stored in a SQLite DB (but the format has changed?)

Post by Andy Grundman
Not sure about this specific case but this is exactly the kind of bug
I want to fix. Some such as changing genres or years are already
handled. This is one of the areas where I really need people's help,
trying common use cases for changing tags.

I've raised several bugs about this sort of issue in the past, so I can try them out to see if they are now fixed. Some may require schema changes.

Post by Andy Grundman
Right now it's always enabled for Mac and Linux with local files. I
plan to make it a pref though. There is a built-in delay to handle
exactly the situation you describe. The rescan kicks off 15 seconds
after the last event is received.

Great. Not turned on for Windows though? Is there a way to enable it?

Post by Andy Grundman
This needs testing to see if there is any impact. If it's a problem we
can easily have the auto-rescan kick off a new process. In-process is
supported for low-memory environments mostly.

Okay, will try it out. I'm fine with it being in-process for the full rescan (wipe) case, because I normally don't play local music until it's finished (typically full-rescan overnight). Playing music when scanning can mess up library, as playing a song invokes the scanner too. i.e. because it may not yet have merged various artists, etc.

[gharris999]

Post by Phil Meyer
I thought the compiled scanner were the main reason for faster scanning.
Other people have posted that performance is worse.

Just finished a new 7.4 wipe & scan:

Total Tracks: 44,333
Total Albums: 2,328
Total Artists: 992
Total Genres: 39
Total Playing Time: 3153:37:13

Total Time: 00:43:17 (Saturday, June 27, 2009 / 3:28 PM)

That is faster than my current 7.3.4 scans by a couple of minutes or
so.

[gharris999]

Andy: is the "Automatic detection and rescanning of new/deleted/changed
files" feature implemented yet?

[Andy Grundman]

Post by gharris999
Andy: is the "Automatic detection and rescanning of new/deleted/
changed
files" feature implemented yet?

Yep, on Mac and Linux for local files.

[dean blackketter]

Post by Andy Grundman

Post by gharris999
Andy: is the "Automatic detection and rescanning of new/deleted/
changed
files" feature implemented yet?

Yep, on Mac and Linux for local files.

NIce. Is there any facility for detecting changes at startup that may
have happened while SC was shut down? Or is a rescan required?

[Andy Grundman]

Post by dean blackketter

Post by Andy Grundman

Post by gharris999
Andy: is the "Automatic detection and rescanning of new/deleted/
changed
files" feature implemented yet?

Yep, on Mac and Linux for local files.

NIce. Is there any facility for detecting changes at startup that may
have happened while SC was shut down? Or is a rescan required?

Yeah, on startup a rescan check is done to see if anything has
changed. That should work now on all platforms.

[gharris999]

Andy: FYI: on a clear and rescan, I don't think the starting id column
values in the db tables are getting reset to 1. E.G: my genres table,
which has 39 records, starts at id==138; tracks, which has 47395
records: 1st id == 189507.

I don't what the largest value that a INTEGER PRIMARY KEY field can
hold. If it's 64 bits, I suppose that this isn't really a problem. I'd
have to rescan, what...389 trillion times before the field overflowed?

Anyway, this does seem to be different from how the mysql wipe and
rescan worked.

[Andy Grundman]

Post by gharris999
Andy: FYI: on a clear and rescan, I don't think the starting id column
values in the db tables are getting reset to 1. E.G: my genres table,
which has 39 records, starts at id==138; tracks, which has 47395
records: 1st id == 189507.
I don't what the largest value that a INTEGER PRIMARY KEY field can
hold. If it's 64 bits, I suppose that this isn't really a problem.
I'd
have to rescan, what...389 trillion times before the field overflowed?
Anyway, this does seem to be different from how the mysql wipe and
rescan worked.

http://www.sqlite.org/autoinc.html

It may be fixable, but probably not worth worrying about.

[gharris999]

Post by gharris999

Post by gharris999
Andy: FYI: on a clear and rescan, I don't think the starting id

column

Post by gharris999
values in the db tables are getting reset to 1. E.G: my genres

table,

Post by gharris999
which has 39 records, starts at id==138; tracks, which has 47395
records: 1st id == 189507.
I don't what the largest value that a INTEGER PRIMARY KEY field can
hold. If it's 64 bits, I suppose that this isn't really a problem.
I'd
have to rescan, what...389 trillion times before the field

overflowed?

Post by gharris999
Anyway, this does seem to be different from how the mysql wipe and
rescan worked.

http://www.sqlite.org/autoinc.html
It may be fixable, but probably not worth worrying about.

Hummm...signed. OK, make that 194 trillion rescans.

[Phil Meyer]

That means web page responses will get bigger and bigger ;-)

[Phil Meyer]

Post by Andy Grundman
Yeah, on startup a rescan check is done to see if anything has
changed. That should work now on all platforms.

Is that turned off if automatic rescan is disabled?

[Andy Grundman]

Post by Phil Meyer

Post by Andy Grundman
Yeah, on startup a rescan check is done to see if anything has
changed. That should work now on all platforms.

Is that turned off if automatic rescan is disabled?

It can't be disabled yet, but yeah it will be.

[radish]

There are some files missing when I do a full rescan, but I haven't
investigated which ones yet. I do see this when I restart the server
though:

[09-06-28 12:52:02.1368] Slim::Utils::Scanner::Local::__ANON__ (134)
Removing deleted files (18633)
[09-06-28 12:52:02.3639] Slim::Utils::Scanner::Local::__ANON__ (185)
Scanning new files (6)
[09-06-28 12:52:02.4345] Slim::Utils::Scanner::Local::__ANON__ (236)
Rescanning changed files (0)

I haven't deleted any files, certainly not 18000! My SC also seems to
be taking substantial CPU whilst idle (nothing playing and no web UI
open), and there's nothing in the log. Probably unrelated but I'm
killing my 7.4 for the moment until I figure out what's up.

FYI, this is on Squeezebox Server (v7.4, r27265, Sun Jun 28 04:00:08
PDT 2009)

[radish]

As a follow up to the missing files issue...

At the start of the scanner log I see this:


Code:
--------------------
[09-06-28 12:09:15.9801] Slim::Utils::Scanner::Local::__ANON__ (185) Scanning new files (20655)
--------------------


20655 is the correct number of files in my full library.

There are then a couple hundred lines like this:


Code:
--------------------
[09-06-28 12:24:47.2904] Slim::Formats::Movie::getTag (69) Warning: Use of uninitialized value in string eq at /usr/share/perl5/Slim/Formats/Movie.pm line 69.
--------------------


But the final count according to SC is just 18009 (so missing 2649
files). Actually, I just noticed it's going down (now 17748, now 17649).
So it's slowly pulling all the files out of the library - this is
probably related to the CPU usage I mentioned previously :)

This is on Ubuntu 8.10 with the library mounted over cifs.

[Andy Grundman]

Post by radish
But the final count according to SC is just 18009 (so missing 2649
files). Actually, I just noticed it's going down (now 17748, now 17649).
So it's slowly pulling all the files out of the library - this is
probably related to the CPU usage I mentioned previously :)
This is on Ubuntu 8.10 with the library mounted over cifs.

OK guess you found a bug, most likely the paths differ between what's
in the database and what the scanner found. Please enable
scan.scanner and scan.auto and see what it's doing.

[radish]

Post by Andy Grundman
OK guess you found a bug, most likely the paths differ between what's
in the database and what the scanner found. Please enable
scan.scanner and scan.auto and see what it's doing.

I enabled all the scanner logs (to debug level) and restarted SC. I see
the song count going down gradually and the server log looks like this:


Code:
--------------------

[09-06-28 22:59:10.7367] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/07. 83.flac
[09-06-28 22:59:11.0084] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/08. 3X5.flac
[09-06-28 22:59:11.1415] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/09. Love Song for No One.flac
[09-06-28 22:59:11.7524] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/10. Back to You.flac
[09-06-28 22:59:12.0093] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/11. Great Indoors.flac
[09-06-28 22:59:12.1311] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/12. Not Myself.flac
[09-06-28 22:59:12.7614] Slim::Utils::Scanner::Local::deleted (311) Handling deleted track /mnt/music/scan/flac-ripped/Artist Albums/John Mayer - Room for Squares/13. (Untitled).flac

--------------------

...basically repeating for the whole library.

I have confirmed that the paths listed are valid and point to files
which are still there and readable.

Let me know if there's other info I can provide!

[pippin]

Cool. iTunes integration on Mac started to work again :-)

[Andy Grundman]

Post by pippin
Cool. iTunes integration on Mac started to work again :-)

Cool, so far iTunes mode is completely untested. ;)

[pippin]

Post by Andy Grundman

Post by pippin
Cool. iTunes integration on Mac started to work again :-)

Cool, so far iTunes mode is completely untested. ;)

And it did NOT work one week ago.
Did you get my mail on the "Years" issue?

[pippin]

The bad thing is, recent 7.4s started to crash iPeng again.
Will file a few bugs tomorrow. Need some data on this one, as on the
"years" issue.

[gharris999]

Post by Andy Grundman

Post by gharris999
Andy: is the "Automatic detection and rescanning of new/deleted/
changed
files" feature implemented yet?

Yep, on Mac and Linux for local files.

Are there any limitations on this?

On my machine, my disk with the audio library is mounted via ntfs-3g to
/mnt/media. My music library is at /mnt/media/Music.

With SC 7.3.4, Stuart Hickinbottom's AutoRescan plugin works perfectly
with this arrangement.

On 7.4, so far, I can't get your changed files feature to detect
changes.

[Andy Grundman]

Post by gharris999

Post by Andy Grundman

Post by gharris999
Andy: is the "Automatic detection and rescanning of new/deleted/
changed
files" feature implemented yet?

Yep, on Mac and Linux for local files.

Are there any limitations on this?
On my machine, my disk with the audio library is mounted via ntfs-3g to
/mnt/media. My music library is at /mnt/media/Music.
With SC 7.3.4, Stuart Hickinbottom's AutoRescan plugin works perfectly
with this arrangement.
On 7.4, so far, I can't get your changed files feature to detect
changes.

His plugin also uses Inotify so yes, it should work.

[NigelMSB]

On Jun 27, 2009, at 12:27 PM, Phil Meyer <slim (AT) hergest (DOT)
demon.co.uk>

Post by Phil Meyer
Rescannng in process (rather than spawning another process) - what
modes do this, is it controllable? I think scanning in another
process was better than in the server process, to avoid drop-outs in
the music. Is in-process using another thread now?

This needs testing to see if there is any impact. If it's a problem we
can easily have the auto-rescan kick off a new process. In-process is
supported for low-memory environments mostly.

Can a lodge a vote for a separate scanner process (ideally default but
an option would be okay)? On Windows Vista or above, it's possible to
configure the scanner process to use Windows' low-priority I/O feature
(enhancement with basic patch at
http://bugs.slimdevices.com/show_bug.cgi?id=8525) to ensure that the
scanner doesn't affect streaming/UI. I assume the equivalent on Linux
would be ionice.

It'd be a shame to lose the ability to do this kind of thing.

Thanks!

[gharris999]

Post by NigelMSB
Can a lodge a vote for a separate scanner process (ideally default but
an option would be okay)? On Windows Vista or above, it's possible to
configure the scanner process to use Windows' low-priority I/O feature
(enhancement with basic patch at
http://bugs.slimdevices.com/show_bug.cgi?id=8525) to ensure that the
scanner doesn't affect streaming/UI. I assume the equivalent on Linux
would be ionice.
It'd be a shame to lose the ability to do this kind of thing.
Thanks!

The Linux versions already have this, under
Settings->Advanced->Performance. Since the current scanner process is
pegging my core2duo cpu at 50%...I assume that the new scanner code is
just ignoring this setting for now, while the bugs are being worked out.

[NigelMSB]

Post by gharris999
The Linux versions already have this, under
Settings->Advanced->Performance. Since the current scanner process is
pegging my core2duo cpu at 50%...I assume that the new scanner code is
just ignoring this setting for now, while the bugs are being worked out.

I'm not running Linux to check, but this is for disk I/O priority not
CPU priority (which I assume is what you mean).

[pippin]

Looks like I was too quick to proclaim that iTunes integration works, I
just judged from the fact that I saw no errors and didn't end up with an
empty database (as before). Yet it looks like iTunes import still breaks
off midterm.

[dean blackketter]

Do you have a log you can share in a bug, Pippin?

Post by pippin
Looks like I was too quick to proclaim that iTunes integration
works, I
just judged from the fact that I saw no errors and didn't end up with an
empty database (as before). Yet it looks like iTunes import still breaks
off midterm.
--
pippin
---
see iPeng, the Squeezebox iPhone remote, at penguinlovesmusic.com
------------------------------------------------------------------------
pippin's Profile: http://forums.slimdevices.com/member.php?
userid=13777
View this thread: http://forums.slimdevices.com/showthread.php?t=64981
_______________________________________________
beta mailing list
http://lists.slimdevices.com/mailman/listinfo/beta

[pippin]

Post by dean blackketter
Do you have a log you can share in a bug, Pippin?

http://bugs.slimdevices.com/show_bug.cgi?id=12581

Bug day :-)

[gharris999]

Post by NigelMSB
I'm not running Linux to check, but this is for disk I/O priority not
CPU priority (which I assume is what you mean).

Yes, you are right. I didn't really understand your post...but I do
now. Thanks.

[gharris999]

Post by andyg
I've changed all scanning in the current 7.4 over to the new scanner
code I've been working on. I expect there will be bugs. I've started a
wiki page that attempts to describe how the new scanner in 7.4 works,
please read it and let me know if you need more info or have problems.
http://wiki.slimdevices.com/index.php/NewScanner

I'm still a little confused by this. When I last checked that wiki
page, I thought it mentioned a compiled binary scanner. I don't see any
mention of that any more. Did I just imagine that? Are you moving away
from that concept?

If I'm running (on linux) from the noweb-sqlite svn branch, is the
correct scanner: server/scanner.pl ??

[gharris999]

OK, I see that the wiki still references "a single C-based module,
Audio::Scan." Again, does that C-based module get installed if one is
running and updating from the noweb-sqlite svn branch? All I'm finding
in my svn checkout is:
/usr/share/squeezecenter74_trunk/server/CPAN/Audio/Scan.pm

[Andy Grundman]

Post by gharris999
OK, I see that the wiki still references "a single C-based module,
Audio::Scan." Again, does that C-based module get installed if one is
running and updating from the noweb-sqlite svn branch? All I'm finding
/usr/share/squeezecenter74_trunk/server/CPAN/Audio/Scan.pm

Yes.

[gharris999]

Yes.

Ah. I see. Digging further, I find:

/usr/share/squeezecenter74_trunk/server/CPAN/arch/5.10/i386-linux-thread-multi/auto/Audio/Scan/Scan.so

..which has a file date of 6/18/09..so that must be it.

[Phil Meyer]

I notice this error in my log:

[23:47:36.9141] Slim::Utils::AutoRescan::init (75) Unable to use stat-based monitoring (Can't locate Slim/Utils/AutoRescan/Stat/Async.pm in @INC (@INC contains:
D:\Squeezebox Server\noweb-sqlite\Cache\InstalledPlugins P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread\auto P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10.0\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10.0\MSWin32-x86-multi-thread\auto P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread\auto P:\Music\SlimServer\SQLite\server\CPAN\arch\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\lib P:\Music\SlimServer\SQLite\server\CPAN P:/Music/SlimServer/SQLite/server P:/Music/SlimServer/SQLite/server P:/Programming/Perl 5.10/site/lib P:/Programming/Perl 5.10/lib .)

at (eval 1071) line 2.
BEGIN failed--compilation aborted at (eval 1071) line 2.
Compilation failed in require at P:/Music/SlimServer/SQLite/server/Slim/Utils/AutoRescan.pm line 74.

[Andy Grundman]

Post by Phil Meyer
D:\Squeezebox Server\noweb-sqlite\Cache\InstalledPlugins P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread\auto P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10.0\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10.0\MSWin32-x86-multi-thread\auto P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\CPAN\arch\5.10\MSWin32-x86-multi-thread\auto P:\Music\SlimServer\SQLite\server\CPAN\arch\MSWin32-x86-multi-thread P:\Music\SlimServer\SQLite\server\lib P:\Music\SlimServer\SQLite\server\CPAN P:/Music/SlimServer/SQLite/server P:/Music/SlimServer/SQLite/server P:/Programming/Perl 5.10/site/lib P:/Programming/Perl 5.10/lib

.)

Post by Phil Meyer
at (eval 1071) line 2.
BEGIN failed--compilation aborted at (eval 1071) line 2.
Compilation failed in require at P:/Music/SlimServer/SQLite/server/Slim/Utils/AutoRescan.pm line 74.

This is normal and can be safely ignored. That module is coming soon.

[radish]

Just a note that the problem I was seeing seems to have gone away with
the latest nightly. I'll post back (and file a bug) if I see a repeat.

[radish]

Post by radish
Just a note that the problem I was seeing seems to have gone away with
the latest nightly. I'll post back (and file a bug) if I see a repeat.

Spoke too soon, all my track counts are back to zero having been wiped
out. Filed a bug: http://bugs.slimdevices.com/show_bug.cgi?id=12655

[Phil Meyer]

Bug 9942 - Multiple ARTIST tags in APEv2 not recognised correctly
http://bugs.slimdevices.com/show_bug.cgi?id=9942

This bug is still present. The monkey audio files that I have with multiple artists open correctly in Foobar2000, but Squeezebox Server still interprets them as one single artist with a NULL character in the middle of the artist name.

Is this something that can be looked at whilst working on the scanner code?

[Greg Klanderman]

Post by andyg
I've changed all scanning in the current 7.4 over to the new scanner
code I've been working on. I expect there will be bugs. I've started a
wiki page that attempts to describe how the new scanner in 7.4 works,
please read it and let me know if you need more info or have problems.
http://wiki.slimdevices.com/index.php/NewScanner

This sounds like great stuff, Andy, I'm excited to check it out!

In automatic rescan mode, once it detects a song has changed, how long
does it take to update the database? Will it still be doing full
versions of all the slow post-scan steps, or will those be made
incremental and fast?

How about rescanning playlists? This is probably my biggest
frustration right now - I edit a playlist then to get the changes seen
by the server I need to manually rescan, and because of the post-scan
steps that takes a significant amount of time. It should at least be
smart enough to skip the post-scan steps if no tag changes are found
rescanning the playlists, which should usually be the case unless your
playlist changes refer to (new) files outside the server's normal
music directory. This could be a big win for very little effort.

On that web page you write:

| Scanning of file metadata and tags accounted for approx. 1/3 of the
| scan time in prior versions of SqueezeCenter. All the Perl-based file
| scanning modules have been replaced by a single C-based module,
| Audio::Scan. This reduces this metadata/tag reading time to a much
| smaller fraction of the scan process. The rest of the scan time is
| taken in the Perl database abstraction layer (DBIx::Class), and
| remains generally unchanged.

By what fraction has the tag scanning been sped up?

Attacking the component taking 1/3 of the time is certainly useful,
but usually one would first go after the component taking 2/3.. any
plans to speed up the database access? From what I understand DBIx is
really slow, is there a plan to use something else? Also, I would
think it'd be a big win to cache updates to the database and only
commit in bigger chunks while scanning.

I've got about 22,000 mp3s, and can extract the tags in less than 45
seconds so I really do not see why a full rescan should take any more
than about a minute:

[mp3] ***@lwm| find . -name '*.mp3' -print | wc -l
22162
[mp3] ***@lwm| time find . -name '*.mp3' -print0 | xargs -0 id3info | wc -c
11067476
find . -name '*.mp3' -print0 0.03s user 0.07s system 0% cpu 41.251 total
xargs -0 id3info 12.63s user 28.98s system 96% cpu 43.314 total
wc -c 0.12s user 0.96s system 2% cpu 43.311 total

And that's only about 10 megs of tag data (probably a lot less if you
consider how verbose the output of id3info is); there's no reason I
see not to store all the tag data in ram while scanning and commit to
the DB all at once at the end.

thanks,
Greg

[Andy Grundman]

Post by Greg Klanderman

Post by andyg
I've changed all scanning in the current 7.4 over to the new scanner
code I've been working on. I expect there will be bugs. I've started a
wiki page that attempts to describe how the new scanner in 7.4 works,
please read it and let me know if you need more info or have
problems.
http://wiki.slimdevices.com/index.php/NewScanner

This sounds like great stuff, Andy, I'm excited to check it out!
In automatic rescan mode, once it detects a song has changed, how long
does it take to update the database? Will it still be doing full
versions of all the slow post-scan steps, or will those be made
incremental and fast?

The rescan code fires 15 seconds after the last change. If using a
network share, the delay is a bit longer.

Post by Greg Klanderman
How about rescanning playlists? This is probably my biggest
frustration right now - I edit a playlist then to get the changes seen
by the server I need to manually rescan, and because of the post-scan
steps that takes a significant amount of time. It should at least be
smart enough to skip the post-scan steps if no tag changes are found
rescanning the playlists, which should usually be the case unless your
playlist changes refer to (new) files outside the server's normal
music directory. This could be a big win for very little effort.

Yeah playlists are also rescanned this way (although not well tested
yet).

Post by Greg Klanderman
| Scanning of file metadata and tags accounted for approx. 1/3 of the
| scan time in prior versions of SqueezeCenter. All the Perl-based file
| scanning modules have been replaced by a single C-based module,
| Audio::Scan. This reduces this metadata/tag reading time to a much
| smaller fraction of the scan process. The rest of the scan time is
| taken in the Perl database abstraction layer (DBIx::Class), and
| remains generally unchanged.
By what fraction has the tag scanning been sped up?
Attacking the component taking 1/3 of the time is certainly useful,
but usually one would first go after the component taking 2/3.. any
plans to speed up the database access? From what I understand DBIx is
really slow, is there a plan to use something else? Also, I would
think it'd be a big win to cache updates to the database and only
commit in bigger chunks while scanning.

Of course, but improving the performance of DBIx::Class is not a
trivial task. Definitely something we need to improve, no question.

Post by Greg Klanderman
I've got about 22,000 mp3s, and can extract the tags in less than 45
seconds so I really do not see why a full rescan should take any more
22162
11067476
find . -name '*.mp3' -print0 0.03s user 0.07s system 0% cpu 41.251 total
xargs -0 id3info 12.63s user 28.98s system 96% cpu 43.314 total
wc -c 0.12s user 0.96s system 2% cpu 43.311 total
And that's only about 10 megs of tag data (probably a lot less if you
consider how verbose the output of id3info is); there's no reason I
see not to store all the tag data in ram while scanning and commit to
the DB all at once at the end.

I don't really like comments like this; you can comparing apples and
oranges. You are completely ignoring the large amount of work SC does
once it gets your tag information. Raw tag scanning is fast, and
Audio::Scan is very fast by itself. But then you have to do something
with the tags, translate them into the right values, put them in the
database, etc. Sure, you could write all of that stuff in C but
matching the features in SC would be an incredible amount of work.
Netgear tried this and ended up with a really fast scanner with maybe
10% of the features. Porting the tag scanning code to C was a
manageable amount of work and made sense to live as a standalone
module. That's why I did it.

[Phil Leigh]

Andy - I do understand where you are coming from, but it should be
possible to read the tags from disk or MusicIP, manipulate the results
in memory and write the results to the database in less than 5 minutes
(on reasonable hardware). That would be my design goal for a 30k song
database!
What do you see as the major obstacle?
Phil
--
Phil Leigh

You want to see the signal path BEFORE it gets onto a CD/vinyl...it
ain't what you'd call minimal...
SB3 (wired) - TACT 2.2X (Linear PSU) + Good Vibrations S/W - MF
Triplethreat(Audiocom full mods) - Linn 5103 - Aktiv 5.1 system (6x
LK140's, ESPEK/TRIKAN/KATAN/SEIZMIK 10.5), Townsend Supertweeters, Blue
Jeans Digital,Kimber Speaker & Chord Interconnect cables
Outdoors: Boom
------------------------------------------------------------------------
Phil Leigh's Profile: http://forums.slimdevices.com/member.php?userid=85
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[Greg Klanderman]

Post by Andy Grundman
The rescan code fires 15 seconds after the last change. If using a
network share, the delay is a bit longer.

Sorry not to be clear, I wasn't asking how long before the rescan
fires, what I want to know is how fast you expect the rescan to take
to incorporate tag changes to a single track? What I'm trying to get
at is whether the time is proportional to the amount of tag changes,
or is it proportional to the size of the db? Are you just running the
current 'Look for new and changed music', in which case the post-scan
cleanup steps take a very long time, time proportional to the size of
the db. Or have you re-done the post scan logic to be incremental?

Post by Andy Grundman
Yeah playlists are also rescanned this way (although not well tested
yet).

Right, but again, can I incorporate a playlist change in seconds?
Currently, after doing a full wipe and rescan, and having made no
changes whatsoever, rescanning my playlists takes 13 seconds, but
merging various artists and database cleanups take like 3 more
minutes. If no track tags have changed, those post-scan steps should
be completely unnecessary, and even if a small number of track tags
have changed, it should be possible to incrementally incorporate those
changes quickly.

Post by Andy Grundman
Of course, but improving the performance of DBIx::Class is not a
trivial task. Definitely something we need to improve, no question.

I didn't expect it to be easy and I'm glad to hear it is something you
want to improve.

Post by Andy Grundman
I don't really like comments like this; you can comparing apples and
oranges. You are completely ignoring the large amount of work SC does
once it gets your tag information. Raw tag scanning is fast, and
Audio::Scan is very fast by itself. But then you have to do something
with the tags, translate them into the right values, put them in the
database, etc. Sure, you could write all of that stuff in C but
matching the features in SC would be an incredible amount of work.

By no means did I intend to trivialize what SC is doing, I'm actually
very happy with how the scanning works, have watched how much effort
it has taken to get it right over many years, and I think you in
particular have been doing great stuff on SC for a long while now so
I'm not trying to flame you or anyone there. But I do think it is
useful to take a step back to consider with an open mind how much time
scanning tags should reasonably take.

Compared to the speed and amount of RAM in modern computers, the
amount of tag data for 20k tracks is puny. I'm pretty familiar with
the scanner code and not aware of any processing it's doing that's not
essentially linear time w.r.t. the size of the tag data. I see no
reason that perl shouldn't be completely capable of doing that tag
post-processing in no time at all, and ignoring the database
operations, I don't have any reason to believe it is taking
appreciable time. Can you point me to any tag post-processing logic
that is fundamentally computationally intensive?

My tag data can be slurped out of the tracks using a C program in 45
seconds. Stripping out the extraneous output from 'id3info', that tag
data is just under 5 Mb, and to give an idea of the redundancy in that
data, compresses to less than 600 Kb.

A full rescan on 7.3 trunk takes 35 minutes. So if we assume the tag
reading in 7.3 is 1/3 the time, with the C tag reading that might go
down to 25 minutes, assuming the new tag reading takes twice the time
of my C program. That leaves about 23 minutes manipulating tag
data and stuffing it in the database - all with the CPU at 100%
utilization. That's an awful lot of computation on a 2.8 GHz xeon
which I just cannot see being justified for 5 Mb of data - that's
approaching (actually 77% of) a million CPU cycles per byte of data.

thanks,
Greg

[Andy Grundman]

Post by Greg Klanderman

Post by Andy Grundman
The rescan code fires 15 seconds after the last change. If using a
network share, the delay is a bit longer.

Sorry not to be clear, I wasn't asking how long before the rescan
fires, what I want to know is how fast you expect the rescan to take
to incorporate tag changes to a single track? What I'm trying to get
at is whether the time is proportional to the amount of tag changes,
or is it proportional to the size of the db? Are you just running the
current 'Look for new and changed music', in which case the post-scan
cleanup steps take a very long time, time proportional to the size of
the db. Or have you re-done the post scan logic to be incremental?

Oh, yeah, the time is definitely proportional to the amount of tag
changes. It does not run the same thing as a "look for changes" scan.

Post by Greg Klanderman

Post by Andy Grundman
Yeah playlists are also rescanned this way (although not well tested
yet).

Right, but again, can I incorporate a playlist change in seconds?
Currently, after doing a full wipe and rescan, and having made no
changes whatsoever, rescanning my playlists takes 13 seconds, but
merging various artists and database cleanups take like 3 more
minutes. If no track tags have changed, those post-scan steps should
be completely unnecessary, and even if a small number of track tags
have changed, it should be possible to incrementally incorporate those
changes quickly.

The goal is to support playlist changes in seconds, yes. No need to
merge artists as you say.

Post by Greg Klanderman
A full rescan on 7.3 trunk takes 35 minutes. So if we assume the tag
reading in 7.3 is 1/3 the time, with the C tag reading that might go
down to 25 minutes, assuming the new tag reading takes twice the time
of my C program. That leaves about 23 minutes manipulating tag
data and stuffing it in the database - all with the CPU at 100%
utilization. That's an awful lot of computation on a 2.8 GHz xeon
which I just cannot see being justified for 5 Mb of data - that's
approaching (actually 77% of) a million CPU cycles per byte of data.

There's not much else I can say other than to invite you to use
Devel::NYTProf to profile a scan and get an idea for the bits that are
slow. It's almost all DBIx::Class code. I plan to work with those
guys to try and improve performance, but not for this release.

[Greg Klanderman]

Post by Andy Grundman
Oh, yeah, the time is definitely proportional to the amount of tag
changes. It does not run the same thing as a "look for changes" scan.
The goal is to support playlist changes in seconds, yes. No need to
merge artists as you say.

This is absolutely great news Andy, thank you!! I think I caught
elsewhere that you're also squashing the remaining issues where a
"changes" scan gives different results than a "wipe and rescan", so
this will rock..

Post by Andy Grundman
There's not much else I can say other than to invite you to use
Devel::NYTProf to profile a scan and get an idea for the bits that are
slow. It's almost all DBIx::Class code. I plan to work with those
guys to try and improve performance, but not for this release.

That's a fine answer; I've been planning to go grovel around in there
for a few years now, so maybe after your stuff gets merged to 7.4
trunk I'll play around with it. Thanks for the pointer to the
profiler - perl's not really a primary language for me so that should
save me some time.

greg

[andyg]

The svn version of NYTProf adds a cool treemap feature. Here is the
scanner on about 1000 files using Perl 5.8. Note that DBIx::Class is
faster under Perl 5.10, so run that if you can.

[image: Loading Image...]

The "scan" block is all time spent actually scanning files. Artwork
resizing is "newFromJpegData" and "copyResampled". The huge block on
the left is MRO::Compat, needed to support some DBIx::Class features
under 5.8.

[Greg Klanderman]

Post by andyg
The svn version of NYTProf adds a cool treemap feature.

That's very cool.. is there any significance to where the rectangle
appears left/right or top/bottom? Or is the size (area) all that
matters? What do the colors indicate?

thanks,
greg

[Andy Grundman]

Post by Greg Klanderman

Post by andyg
The svn version of NYTProf adds a cool treemap feature.

That's very cool.. is there any significance to where the rectangle
appears left/right or top/bottom? Or is the size (area) all that
matters? What do the colors indicate?

No significance, what matters is their size relative to the whole.

I replaced the image with a better one, taken after adding
Class::C3::XS (which fixes most of the MRO::Compat slowness):

http://eng.slimdevices.com/~andy/treemap-scanner-5.8.png

Colors indicate packages, for example upper-left red =
DBIx::Class::Storage::*

I highly recommend installing Devel::NYTProf if you're at all
interested in this. It's a very powerful profiler.

[Phil Meyer]

Tried again today. Here's an error I get with MusicIP interraction:

[22:40:55.0527] Slim::Networking::SimpleAsyncHTTP::onError (230) Failed to connect to http://localhost:10002/api/filters (Connect timed out: Bad file descriptor)
[22:40:55.0536] Slim::Plugin::MusicMagic::Common::__ANON__ (76) Failed fetching filters from MusicIP

[Michael Herger]

Post by Phil Meyer
[22:40:55.0527] Slim::Networking::SimpleAsyncHTTP::onError (230) Failed
to connect to http://localhost:10002/api/filters (Connect timed out: Bad
file descriptor)

Can you do this request manually (wget or browser)?

[Phil Meyer]

Post by Michael Herger

Post by Phil Meyer
[22:40:55.0527] Slim::Networking::SimpleAsyncHTTP::onError (230) Failed
to connect to http://localhost:10002/api/filters (Connect timed out: Bad
file descriptor)

Can you do this request manually (wget or browser)?

Yes - works fine.

[andyg]

Wanted to give an update on 7.4 scanner performance after some changes I
made last night and this morning.

This is measuring the raw performance of the scanner, without any disk
read overhead, and is pretty encouraging.

External WD Passport USB2.0 with a mix of Various Artist tracks (8.9G,
1029 files), all reads from disk cache
MacBook Pro 2.53ghz
Avg of 5 runs

scanner.pl --wipe --rescan --progress

7.4 sqlite 2009-07-10, standalone, Perl 5.10.0 (avg: 85 tracks/sec)
MusicFolderScan: 13s / 12s / 12s / 13s / 13s

7.3.4 mysql 2009-07-10, standalone, Perl 5.8.8 (avg: 35 tracks/sec)
MusicFolderScan: 29s / 29s / 29s / 29s / 30s

[Greg Klanderman]

Post by andyg
7.4 sqlite 2009-07-10, standalone, Perl 5.10.0 (avg: 85 tracks/sec)
MusicFolderScan: 13s / 12s / 12s / 13s / 13s
7.3.4 mysql 2009-07-10, standalone, Perl 5.8.8 (avg: 35 tracks/sec)
MusicFolderScan: 29s / 29s / 29s / 29s / 30s

that's great Andy!

greg

[Phil Meyer]

Post by andyg
This is measuring the raw performance of the scanner, without any disk
read overhead, and is pretty encouraging.

Encouraging, yes.

How much of that is due to the scanner being an exe, and how much is due to Sqlite though. I'm thinking that it's mainly the compiled scanner processing that is the difference, not the DB engine?

I'm not really all that bothered with scanner performance, as I would hope to not need to do full scans all that often, if new/changed files scans were to work better. And if I do full rescans, I tend to do them when I'm out or sleeping anyway.

I'm interested in seeing some performance stats between SQL queries, i.e. for browsing and searching the music library.

[Andy Grundman]

Post by Phil Meyer

Post by andyg
This is measuring the raw performance of the scanner, without any disk
read overhead, and is pretty encouraging.

Encouraging, yes.
How much of that is due to the scanner being an exe, and how much is
due to Sqlite though. I'm thinking that it's mainly the compiled
scanner processing that is the difference, not the DB engine?

Err, it was on my Mac. Also, the term "compiled" on Windows is
misleading, it's still Perl. The DB engine is not a factor here,
SQLite is plenty fast. The problem is DBIx::Class and the changes I
put in work around some of those issues.

[Phil Meyer]

Post by Andy Grundman
The problem is DBIx::Class and the changes I
put in work around some of those issues.

Even more encouraging.

Thanks!

[Phil Meyer]

I tried the noweb-sqlite latest code stream again tonight.

Running perl source.

Kicking the scanner off from the web UI still doesn't show any scanner progress in the web UI. Is it meant to? I saw a checkin comment about not showing progress if started with --progress, but I haven't used that switch.

Also, the scanner doesn't seem to be passed the --logfile switch, so it defaults to C:\Documents and Settings\All Users\Application Data\Squeezebox\Logs\scanner.log. I guess it is picking up the --prefsdir, because I think it's writing to the correct cachedir (specified in the prefs).

[Phil Meyer]

It seems to have trouble with diacritics in file names.

eg. I have a playlist that references the path to a file

#EXTINF:-1,Neunundfünfzig 49
M:\Music\Phil's Music\Electronic\Wolfram Spyra & Chris Lang\Achtundsechzig 24\01 - Neunundfünfzig 49.mp3

The scanner contains - twice!:

[20:36:58.8625] Slim::Formats::Playlists::Base::playlistEntryIsValid (120) Warning: file:///M:/Music/Phil%27s%20Music/Electronic/Wolfram%20Spyra%20&%20Chris%20Lang/Achtundsechzig%2024/01%20-%20Neunundf found in playlist:
file:///M:/Music/Slimserver/Playlists/Bad%20Tags.m3u doesn't exist on disk - skipping!
[20:36:58.9443] Slim::Formats::Playlists::Base::playlistEntryIsValid (120) Warning: file:///M:/Music/Phil%27s%20Music/Electronic/Wolfram%20Spyra%20&%20Chris%20Lang/Achtundsechzig%2024/01%20-%20Neunundf found in playlist:
file:///M:/Music/Slimserver/Playlists/Bad%20Tags.m3u doesn't exist on disk - skipping!


After that, I also have some errors reported:

[20:40:00.6532] Slim::Formats::loadTagFormatForType (100) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 100.
[20:40:00.6537] Slim::Formats::loadTagFormatForType (103) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103.
[20:40:00.6541] Slim::Formats::loadTagFormatForType (103) Error: Couldn't load module: : [syntax error at (eval 971) line 2, at EOF
]
[20:40:00.6547] Slim::Formats::loadTagFormatForType (103) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103)
frame 1: Slim::Formats::loadTagFormatForType (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 213)
frame 2: Slim::Music::Artwork::_readCoverArtTags (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 148)
frame 3: Slim::Music::Artwork::readCoverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 4: (eval) (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 5: Slim::Schema::Track::coverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 434)
frame 6: Slim::Schema::Track::coverArtExists (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 96)
frame 7: Slim::Music::Artwork::findArtwork (P:/Music/SlimServer/SQLite/server/Slim/Music/Import.pm line 364)
frame 8: Slim::Music::Import::runScanPostProcessing (P:/Music/SlimServer/SQLite/server/scanner.pl line 304)
frame 9: (eval) (P:/Music/SlimServer/SQLite/server/scanner.pl line 304)
frame 10: main::main (P:/Music/SlimServer/SQLite/server/scanner.pl line 525)

[20:40:00.7836] Slim::Formats::loadTagFormatForType (100) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 100.
[20:40:00.7841] Slim::Formats::loadTagFormatForType (103) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103.
[20:40:00.7845] Slim::Formats::loadTagFormatForType (103) Error: Couldn't load module: : [syntax error at (eval 972) line 2, at EOF
]
[20:40:00.7850] Slim::Formats::loadTagFormatForType (103) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103)
frame 1: Slim::Formats::loadTagFormatForType (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 213)
frame 2: Slim::Music::Artwork::_readCoverArtTags (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 148)
frame 3: Slim::Music::Artwork::readCoverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 4: (eval) (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 5: Slim::Schema::Track::coverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 434)
frame 6: Slim::Schema::Track::coverArtExists (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 96)
frame 7: Slim::Music::Artwork::findArtwork (P:/Music/SlimServer/SQLite/server/Slim/Music/Import.pm line 364)
frame 8: Slim::Music::Import::runScanPostProcessing (P:/Music/SlimServer/SQLite/server/scanner.pl line 304)
frame 9: (eval) (P:/Music/SlimServer/SQLite/server/scanner.pl line 304)
frame 10: main::main (P:/Music/SlimServer/SQLite/server/scanner.pl line 525)


Phil

[Phil Meyer]

The scanner process finished soon after the error:

[20:40:22.3232] Slim::Music::Import::endImporter (620) Completed findArtwork Scan in 146 seconds.
[20:40:22.4664] Slim::Music::Import::runScanPostProcessing (407) Starting Database optimization.
[20:40:30.3570] Slim::Music::Import::endImporter (620) Completed dbOptimize Scan in 7 seconds.
[20:40:31.8883] Slim::bootstrap::theEND (430) Got to the END
[20:40:31.8888] Slim::bootstrap::sigint (394) Got sigint

I then looked in the web UI and I didn't have any new music in the library (was still reporting the old library content).

So I stopped SbS and restarted it, and it then showed the new library content.

[Mnyb]

Whats this ?

"
Remove deleted files (130 of 130) Running 00:00:00
"

The count of so called deleted files is increasing for every new record
I add ?

Btw i have not deleted any files on this disc for months ?

[Mnyb]

A good way to crash the "new and changed scan" is to fiddle with pause
and play on an active player

[JJZolx]

Post by andyg
Wanted to give an update on 7.4 scanner performance after some changes I
made last night and this morning.
This is measuring the raw performance of the scanner, without any disk
read overhead, and is pretty encouraging.

Wouldn't these same code optimizations benefit the 7.4 Trunk running
MySQL?

It's definitely made a huge difference. I have a SQLite branch server
and a 7.4 Trunk server running 24x7, and both have clear/rescans
scheduled to run each night (at different times). Scan times, as
reported on the info page below.

The library consists of all FLAC files, 24217 tracks in 2029 albums,
811 artists.

7.4 MySQL: 35:17
7.4 SQLite: 16:18

Very nice.

[egd]

Total Tracks: 96,668
Total Albums: 6,333
Total Artists: 2,170
Total Genres: 46

The server has finished scanning your music collection.
Total Time: 01:38:38 (Tuesday, July 14, 2009 / 8:03 PM)

From memory that's shaved about 20mins off my scan time.

--
egd

Internet forums: conclusive proof depth of gene pool is indeed variable,
monkeys can be taught to cut code, and world peace is utterly
unrealistic...

Integrating MusicIP with SqueezeCenter...'*here's how*'
(http://wiki.slimdevices.com/index.php/Integrating_MusicIP_with_SqueezeCenter).
------------------------------------------------------------------------
egd's Profile: http://forums.slimdevices.com/member.php?userid=3425
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[egd]

The scan also omitted a hell of a lot of artwork.
--
egd

Internet forums: conclusive proof depth of gene pool is indeed variable,
monkeys can be taught to cut code, and world peace is utterly
unrealistic...

Integrating MusicIP with SqueezeCenter...'*here's how*'
(http://wiki.slimdevices.com/index.php/Integrating_MusicIP_with_SqueezeCenter).
------------------------------------------------------------------------
egd's Profile: http://forums.slimdevices.com/member.php?userid=3425
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[mariowing]

I don't know if this is the correct tread, please advice me where to
post if this is inapropriate.

I have quite a few m4a-files (ripped cd's in iTunes) and for those
files the scanner does not pick up the genres. Is this a known bug or
limitation of the new scanner? At the moment I am running 7.4 - r27438
on Windows Home Server.
--
mariowing

SqueezeCenter on Windows Home Server. Squeezebox Duet in living rom,
Squeezebox Boom in bathroom and home office. SqueezePlay at work,
connected through Hamachi VPN. iTunes integration through Erlands Custom
Scan.
------------------------------------------------------------------------
mariowing's Profile: http://forums.slimdevices.com/member.php?userid=22779
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[Andy Grundman]

Can you file a bug and attach one of these files? Please enter
'Audio::Scan' in the tag field so I'm sure to see it.

Post by mariowing
I don't know if this is the correct tread, please advice me where to
post if this is inapropriate.
I have quite a few m4a-files (ripped cd's in iTunes) and for those
files the scanner does not pick up the genres. Is this a known bug or
limitation of the new scanner? At the moment I am running 7.4 - r27438
on Windows Home Server.

[mariowing]

Post by Andy Grundman
Can you file a bug and attach one of these files? Please enter
'Audio::Scan' in the tag field so I'm sure to see it.

I am sorry, but I didn't find the tag field. But the bug is here:
http://bugs.slimdevices.com/show_bug.cgi?id=12875
--
mariowing

SqueezeCenter on Windows Home Server. Squeezebox Duet in living rom,
Squeezebox Boom in bathroom and home office. SqueezePlay at work,
connected through Hamachi VPN. iTunes integration through Erlands Custom
Scan.
------------------------------------------------------------------------
mariowing's Profile: http://forums.slimdevices.com/member.php?userid=22779
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[dwilliams01]

Are the scanner changes "complete" so that we should just file bugs or
is it still undergoing overhaul?

I'm seeing, with the 27567 nightly build for Windows XP, a problem with
some albums ending up not together. Each track is listed as a seperate
album.

I also have some albums that are showing up under various artists when
it appears that both the album artist and track artist are the same
(Queen).

This was from a clear/full scan after a full wipe of the install.

What more information would be useful to help track them down?

[andy-SBQ2+]

I wouldn't say the changes are complete, but you can file bugs about
that sort of thing.

I think the best way for us to reproduce the problem is for you to
attach 2 files to a bug, which, when scanned, produce the wrong
results (i.e. they aren't grouped properly).

Please add "SQLite" to the Bugzilla keywords field when you file a
bug, and cc me (andy at slimdevices.com).

Post by dwilliams01
Are the scanner changes "complete" so that we should just file bugs or
is it still undergoing overhaul?
I'm seeing, with the 27567 nightly build for Windows XP, a problem with
some albums ending up not together. Each track is listed as a
seperate
album.
I also have some albums that are showing up under various artists when
it appears that both the album artist and track artist are the same
(Queen).
This was from a clear/full scan after a full wipe of the install.
What more information would be useful to help track them down?

[mbw23]

ok so after all my problems with getting a release version to work, I
took inspiration from Bret and decided to head for 7.4 country. took me
a bit to get it working but when i did, i eventually had to kill -9
slimserver.pl as it was causing other processes to suffer.

Code:
--------------------

Memory: 2048M phys mem, 36M free mem, 4103M total swap, 357M free swap
PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND
4859 root 1 60 0 3583M 1197M sleep 1:26 0.12% slimserver.pl

--------------------


That's SQ'Lite' ?? :)

Either that's a memory leak or the system requirements will have to be
updated to say 8GB RAM or the like. Is this memory esplosion normal for
7.4?

This is on Solaris 10 x86, perl 5.10, SC v7.4, r27604

[mbw23]

heh and now it won't start to scan.. I just keep getting this in the
output:

The following modules failed to load: DBD::SQLite DBI

To download and compile them, please run:
/usr/local/squeezecenter/Bin/build-perl-modules.pl DBD::SQLite DBI

Exiting..


Both of those modules are up to date in the system perl. I've even
copied them to the SC CPAN and no luck. And even though it says
"Exiting.." it keeps running.

[egd]

However, I can also tell you that SQLite performs like an absolute dog
doing simple things like browsing albums - I can literally sit and watch
the albumart get rendered album by album. Performance with a library of
this size in SQLite is not anywhere near acceptable at present.I'm hoping the fact there's been no response to this isn't an indication

that performance isn't going to be looked at and that we're just going
to be steamrolled with a slower interface like it or not?
--
egd

Internet forums: conclusive proof depth of gene pool is indeed variable,
monkeys can be taught to cut code, and world peace is utterly
unrealistic...

Integrating MusicIP with SqueezeCenter...'*here's how*'
(http://wiki.slimdevices.com/index.php/Integrating_MusicIP_with_SqueezeCenter).
------------------------------------------------------------------------
egd's Profile: http://forums.slimdevices.com/member.php?userid=3425
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[Andy Grundman]

Post by egd

However, I can also tell you that SQLite performs like an absolute dog
doing simple things like browsing albums - I can literally sit and watch
the albumart get rendered album by album. Performance with a
library of
this size in SQLite is not anywhere near acceptable at present.I'm
hoping the fact there's been no response to this isn't an indication

that performance isn't going to be looked at and that we're just going
to be steamrolled with a slower interface like it or not?

"watch the album art get rendered album by album" <- This has
absolutely nothing to do with SQLite. Artwork resizing is a dog and
there isn't much we can do about it. Don't disable pre-caching and if
it's still slow you can use the lower quality resizing option.

I am planning on improving the way artwork resizing works for 7.4 so
it won't bog the server down. But it won't be any faster.

[egd]

"watch the album art get rendered album by album" <- This has absolutely
nothing to do with SQLite. Artwork resizing is a dog and
there isn't much we can do about it. Don't disable pre-caching and if
it's still slow you can use the lower quality resizing option.
I am planning on improving the way artwork resizing works for 7.4 so
it won't bog the server down. But it won't be any faster.

Running 7.3.x with MySQL doesn't present the same issue at all - screen
renders in a fraction of the time on the same hardware...
--
egd

Internet forums: conclusive proof depth of gene pool is indeed variable,
monkeys can be taught to cut code, and world peace is utterly
unrealistic...

Integrating MusicIP with SqueezeCenter...'*here's how*'
(http://wiki.slimdevices.com/index.php/Integrating_MusicIP_with_SqueezeCenter).
------------------------------------------------------------------------
egd's Profile: http://forums.slimdevices.com/member.php?userid=3425
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[egd]

"watch the album art get rendered album by album" <- This has absolutely
nothing to do with SQLite. Artwork resizing is a dog and
there isn't much we can do about it. Don't disable pre-caching and if
it's still slow you can use the lower quality resizing option.
I am planning on improving the way artwork resizing works for 7.4 so
it won't bog the server down. But it won't be any faster.

Seriously Andy, whilst the performance may have nothing at all to do
with SQLite it's a very, very far cry from what I'ver experienced with
7.x thus far...if it's not the DB then it's something else, but
something has definitely changed. 7.4 is terribly slow browsing my
library etc. yet the same hardware running 7.3 is snappy.
--
egd

'*Transporter*'
(http://www.logitechsqueezebox.com/products/transporter.html) > '*SCA2*'
(http://www.atc.gb.net/sca2.htm) > '*SCM100SL AT*'
(http://www.atc.gb.net/scm100slat.htm). Integrating MusicIP with
SqueezeCenter...'*here's how*'
(http://wiki.slimdevices.com/index.php/Integrating_MusicIP_with_SqueezeCenter).

Every story has 3 sides: Yours, mine and the truth.
------------------------------------------------------------------------
egd's Profile: http://forums.slimdevices.com/member.php?userid=3425
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[Phil Meyer]

Still getting this issue:

[20:22:53.7300] Slim::Networking::SimpleAsyncHTTP::onError (230) Failed to connect to http://localhost:10002/api/filters (Connect timed out: Bad file descriptor)
[20:22:53.7314] Slim::Plugin::MusicMagic::Common::__ANON__ (76) Failed fetching filters from MusicIP

And then when I play tracks, it can't scrobble:

[20:24:14.7504] Slim::Networking::SimpleAsyncHTTP::onError (230) Failed to connect to http://post.audioscrobbler.com/?hs=true&p=1.2&c=ss7&v=sc7.4&u=hergest&t=12
47858647&a=46807db1dc150e31ea82b513ff5af70b (Error reading headers: Server closed connection without sending any data back at P:\Music\SlimServer\SQLite\server\
CPAN/Net/HTTP/Methods.pm line 306.
...propagated at P:\Music\SlimServer\SQLite\server\CPAN/Net/HTTP/NB.pm line 32.
)
[20:24:14.7513] Slim::Plugin::AudioScrobbler::Plugin::_handshakeError (361) Error handshaking with Last.fm: Error reading headers: Server closed connection with
out sending any data back at P:\Music\SlimServer\SQLite\server\CPAN/Net/HTTP/Methods.pm line 306.
...propagated at P:\Music\SlimServer\SQLite\server\CPAN/Net/HTTP/NB.pm line 32.
[20:24:14.7520] Slim::Plugin::AudioScrobbler::Plugin::_handshakeError (383) retrying in 1 minute(s)

[Andy Grundman]

Are you really running MusicIP?

Last.fm's server could be down, does it ever work?

Post by Phil Meyer
[20:22:53.7300] Slim::Networking::SimpleAsyncHTTP::onError (230)
Failed to connect to http://localhost:10002/api/filters (Connect
timed out: Bad file descriptor)
[20:22:53.7314] Slim::Plugin::MusicMagic::Common::__ANON__ (76)
Failed fetching filters from MusicIP
[20:24:14.7504] Slim::Networking::SimpleAsyncHTTP::onError (230)
Failed to connect to http://post.audioscrobbler.com/?hs=true&p=1.2&c=ss7&v=sc7.4&u=hergest&t=12
Server closed connection without sending any data back at P:\Music
\SlimServer\SQLite\server\
CPAN/Net/HTTP/Methods.pm line 306.
...propagated at P:\Music\SlimServer\SQLite\server\CPAN/Net/
HTTP/NB.pm line 32.
)
[20:24:14.7513]
Slim::Plugin::AudioScrobbler::Plugin::_handshakeError (361) Error
handshaking with Last.fm: Error reading headers: Server closed
connection with
out sending any data back at P:\Music\SlimServer\SQLite\server\CPAN/
Net/HTTP/Methods.pm line 306.
...propagated at P:\Music\SlimServer\SQLite\server\CPAN/Net/
HTTP/NB.pm line 32.
[20:24:14.7520]
Slim::Plugin::AudioScrobbler::Plugin::_handshakeError (383)
retrying in 1 minute(s)
_______________________________________________
beta mailing list
http://lists.slimdevices.com/mailman/listinfo/beta

[Phil Meyer]

Post by Andy Grundman
Are you really running MusicIP?

Yes. I can browse that URL in Firefox and it returns the list of filters.

Post by Andy Grundman
Last.fm's server could be down, does it ever work?

Yes, in 7.4/trunk seems to work. I have an RSS newsfeed of my recently played tracks from last.fm, and I can see tracks that I've played from 7.4/trunk, but none from the noweb-sqlite branch.

[Siduhe]

Post by Phil Meyer
[22:40:55.0527] Slim::Networking::SimpleAsyncHTTP::onError (230) Failed
to connect to http://localhost:10002/api/filters (Connect timed out: Bad
file descriptor)
[22:40:55.0536] Slim::Plugin::MusicMagic::Common::__ANON__ (76) Failed
fetching filters from MusicIP

I've seen this error message several times, but I've always been able
to resolve it by refreshing the cache in MIP (I run headless) and
stopping/starting SC when the filters then get picked up correctly.
--
Siduhe

Who am I on 'LAST.FM' (http://www.last.fm/user/siduhe)?
"-Siduhe Loved Tracks radio got the thumbs up. Feedback included: yeah,
it's good... got the odd dodgy track tho...-" (c) 'ModelCitizen'
(http://www.last.fm/user/Modelcitizen)
Oh yeah, I'm giving 'Twitter' (http://twitter.com/Siduhe) a try - and
surprising myself how many interesting people are out there...
------------------------------------------------------------------------
Siduhe's Profile: http://forums.slimdevices.com/member.php?userid=723
View this thread: http://forums.slimdevices.com/showthread.php?t=64981

[Phil Meyer]

I think the most important next steps to complete are:

1. When starting a scan, indicate that a scan is running!
I've clicked "Rescan" several times now, and wondered why it's not reporting any scan progress, not blocking the Rescan button from being pressed again, etc. Then remembered that the scan is running in the background silently. Scanner logging is not being reported for me either.

Maybe this is my environment setup - I run from perl source, with prefs and cache folders located in non-standard locations. When SbS kicks of a scan, it launches perl.exe to run the scanner, so I don't know where it is attempting to write its log output from that process (I'm guessing it's not passing the slimserver.pl startup params on to the scanner.pl process).

2. Don't start a rescan automatically when starting up and detecting the library is empty.
I've turned off auto-detection at the moment, but even with auto-detection disabled, it still detects when the library is empty at startup (or incomplete?) and kicks of a full scan.
i.e. I'd like to configure a library path and check settings, before deciding the best time to do a full scan.

[Phil Meyer]

I found a scanner log tucked away in the default location. Scanner apparently finished 1/2 hour ago, but there was no notification.

There appear to have been some errors reported, but scanner still completed, rather than crashing out.
My library still has 0 albums with 0 songs by 0 artists.


[00:18:29.0341] Slim::Formats::loadTagFormatForType (100) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 100.
[00:18:29.0347] Slim::Formats::loadTagFormatForType (103) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103.
[00:18:29.0354] Slim::Formats::loadTagFormatForType (103) Error: Couldn't load module: : [syntax error at (eval 972) line 2, at EOF
]
[00:18:29.0361] Slim::Formats::loadTagFormatForType (103) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103)
frame 1: Slim::Formats::loadTagFormatForType (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 213)
frame 2: Slim::Music::Artwork::_readCoverArtTags (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 148)
frame 3: Slim::Music::Artwork::readCoverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 4: (eval) (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 5: Slim::Schema::Track::coverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 434)
frame 6: Slim::Schema::Track::coverArtExists (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 96)
frame 7: Slim::Music::Artwork::findArtwork (P:/Music/SlimServer/SQLite/server/Slim/Music/Import.pm line 343)
frame 8: Slim::Music::Import::runScanPostProcessing (P:/Music/SlimServer/SQLite/server/scanner.pl line 294)
frame 9: (eval) (P:/Music/SlimServer/SQLite/server/scanner.pl line 294)
frame 10: main::main (P:/Music/SlimServer/SQLite/server/scanner.pl line 515)

[00:18:29.1528] Slim::Formats::loadTagFormatForType (100) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 100.
[00:18:29.1534] Slim::Formats::loadTagFormatForType (103) Warning: Use of uninitialized value within %Slim::Formats::tagClasses in concatenation (.) or string at P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103.
[00:18:29.1538] Slim::Formats::loadTagFormatForType (103) Error: Couldn't load module: : [syntax error at (eval 973) line 2, at EOF
]
[00:18:29.1546] Slim::Formats::loadTagFormatForType (103) Backtrace:

frame 0: Slim::Utils::Log::logBacktrace (P:/Music/SlimServer/SQLite/server/Slim/Formats.pm line 103)
frame 1: Slim::Formats::loadTagFormatForType (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 213)
frame 2: Slim::Music::Artwork::_readCoverArtTags (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 148)
frame 3: Slim::Music::Artwork::readCoverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 4: (eval) (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 388)
frame 5: Slim::Schema::Track::coverArt (P:/Music/SlimServer/SQLite/server/Slim/Schema/Track.pm line 434)
frame 6: Slim::Schema::Track::coverArtExists (P:/Music/SlimServer/SQLite/server/Slim/Music/Artwork.pm line 96)
frame 7: Slim::Music::Artwork::findArtwork (P:/Music/SlimServer/SQLite/server/Slim/Music/Import.pm line 343)
frame 8: Slim::Music::Import::runScanPostProcessing (P:/Music/SlimServer/SQLite/server/scanner.pl line 294)
frame 9: (eval) (P:/Music/SlimServer/SQLite/server/scanner.pl line 294)
frame 10: main::main (P:/Music/SlimServer/SQLite/server/scanner.pl line 515)

[00:18:52.9210] Slim::Music::Import::endImporter (595) Completed findArtwork Scan in 571 seconds.
[00:18:53.0653] Slim::Music::Import::runScanPostProcessing (386) Starting Database optimization.
[00:19:09.6718] Slim::Music::Import::endImporter (595) Completed dbOptimize Scan in 16 seconds.
[00:19:13.0781] Slim::bootstrap::theEND (430) Got to the END
[00:19:13.0785] Slim::bootstrap::sigint (394) Got sigint

[dwilliams01]

Windows XP SP3 MCE - version r27567

Just a quick note since I think that you are still working on the
scanner. I show an error in the log at line 91 (malformed query). Here
is an excerpt from the scanner log file:

[09-07-26 14:10:26.6067] Slim::Schema::Storage::throw_exception (87)
Error: Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::st execute
failed: database disk image is malformed [for Statement "INSERT INTO
playlist_track ( playlist, position, track) VALUES ( ?, ?, ? )"] at
/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Storage.pm line
91
[09-07-26 14:10:26.6081] Slim::Schema::Storage::throw_exception (87)
Backtrace:

frame 0: Slim::Utils::Log::logBacktrace
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Storage.pm line
87)
frame 1: Slim::Schema::Storage::throw_exception
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>DBIx/Class/Storage/DBI.pm
line 429)
frame 2: DBIx::Class::Storage::DBI::dbh_do
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>DBIx/Class/Storage/DBI.pm
line 989)
frame 3: DBIx::Class::Storage::DBI::_execute
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>DBIx/Class/Storage/DBI.pm
line 1011)
frame 4: DBIx::Class::Storage::DBI::insert
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>DBIx/Class/Row.pm line 328)
frame 5: DBIx::Class::Row::insert
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>DBIx/Class/ResultSet.pm line
2058)
frame 6: DBIx::Class::ResultSet::create
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Playlist.pm line
111)
frame 7: Slim::Schema::Playlist::_addTracksToPlaylist
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Playlist.pm line
39)
frame 8: Slim::Schema::Playlist::__ANON__
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Playlist.pm line
47)
frame 9: (eval)
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Playlist.pm line
43)
frame 10: Slim::Schema::Playlist::setTracks
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 594)
frame 11: Slim::Utils::Scanner::Local::scanPlaylistFileHandle
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 435)
frame 12: Slim::Utils::Scanner::Local::__ANON__
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 444)
frame 13: Slim::Utils::Scanner::Local::new
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 218)
frame 14: Slim::Utils::Scanner::Local::__ANON__
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 233)
frame 15: Slim::Utils::Scanner::Local::__ANON__
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local/Async.pm
line 69)
frame 16: Slim::Utils::Scanner::Local::Async::__ANON__
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local/Async.pm
line 137)
frame 17: Slim::Utils::Scanner::Local::Async::find
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 58)
frame 18: Slim::Utils::Scanner::Local::find
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Utils/Scanner/Local.pm
line 309)
frame 19: Slim::Utils::Scanner::Local::rescan
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Music/PlaylistFolderScan.pm
line 88)
frame 20: Slim::Music::PlaylistFolderScan::startScan
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Music/Import.pm line
492)
frame 21: Slim::Music::Import::runImporter
(/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Music/Import.pm line
328)
frame 22: Slim::Music::Import::runScan (scanner.pl line 293)
frame 23: (eval) (scanner.pl line 287)
frame 24: main::main (scanner.pl line 526)

[09-07-26 14:10:26.6129] Slim::Schema::Playlist::setTracks (51) Error:
Failed to add tracks to playlist: [Carp::Clan::__ANON__():
Carp::Clan::__ANON__(): DBI Exception: DBD::SQLite::st execute failed:
database disk image is malformed [for Statement "INSERT INTO
playlist_track ( playlist, position, track) VALUES ( ?, ?, ? )"] at
/<C:\PROGRA~1\SQUEEZ~1\server\scanner.exe>Slim/Schema/Storage.pm line
91
]
[09-07-26 14:10:26.6599] Slim::Utils::Scanner::Local::__ANON__ (241)
Rescanning changed files (0)
[09-07-26 14:10:26.6609] Slim::Music::Import::runScanPostProcessing
(352) Starting merge of various artists albums
[09-07-26 14:10:37.6846] Slim::Music::Import::endImporter (624)
Completed mergeVariousAlbums Scan in 11 seconds.
[09-07-26 14:10:37.8959] Slim::Music::Import::runScanPostProcessing
(360) Starting artwork scan

-------------------------------

It hasn't finished yet, but it seems to be finding everything OK so
far. It is looking like it will be a little over 2.5 hours for about
9200 tracks.

[dwilliams01]

OK, spoke a bit too soon.

A few odd things:

1) Scanning reported that it was complete. Shut down SbS and
restarted. SbS reported that it was rescanning but it wasn't. No
scanner.exe was running or was started. Hitting the abort option caused
it to report the completion time that it had reported previously.

2) There was nothing after the artwork import. There used to be some
cleanup/consolidation/optimization tasks that would run and be reported
previously.

3) Many/most albums are not reported properly in the album list. They
are listed multiple times. Looking into the details some tracks seem to
be listed under the album and also again by themselves - leading to 2
album entries. These are single artist albums and do not have any known
non-ascii characters in anything.

4) Even though the MusicIP import reported almost the entire library
being pulled in (mixable only mode) no albums or tracks seem to be
mixable (no "M" anywhere).

5) I does seem like most of the artwork scanned correctly. Previous
versions had many, many covers missing.

Should some/all of these be added as bugs now? Is there any additional
logging that you would like to see first?

Thanks

[dwilliams01]

After the above mentioned clear and rescan I tried just doing an update
scan to see if that would help clear things up. It didn't. See
attached scanner log file.


+-------------------------------------------------------------------+
|Filename: scanner.log |
|Download: http://forums.slimdevices.com/attachment.php?attachmentid=7881|
+-------------------------------------------------------------------+