IPB

Welcome Guest ( Log In | Register )

Python Grabber scripts
2E7AH
post May 28 2009, 15:27
Post #1





Group: Validating
Posts: 2424
Joined: 21-May 08
Member No.: 53675



Some users reported errors using the scripts I posted here
The errors are result of incorrectly saving the copied scripts, probably indentation errors, so I'm posting those scripts as ready made files here:

[attachment=5122:Discogs.zip]
Go to the top of the page
+Quote Post
 
Start new topic
Replies
tore
post Mar 20 2010, 02:03
Post #2





Group: Members
Posts: 41
Joined: 6-February 10
Member No.: 77932



Here's a modification of what I described above.

This python script which is a modification of 2E7AH's scripts fetches Genre and Style from discogs. When it meets a track from an album it has not processed before, it writes those tags to a text file. If it meets a track from an album it has processed a track from before, it does not check discogs - rather, it just fetches the tags from those text files. In essence, it fetches discogs genre and style while minimizing the amount of requests. This will allow you to tag thousands more files per 24 hours. For temporary storage of information, it uses text files.

My text files are in the folder C:\Python26 .. Edit the script and change them for whatever values you want to use!

CODE
import urllib, urllib2, gzip, cStringIO
import xml.etree.ElementTree
from xml.dom import minidom
from encodings import utf_8
from grabber import LyricProviderBase

class Discogs_GetGenre(LyricProviderBase):
def GetName(self):
return 'Discogs GenStyles'

def GetVersion(self):
return '0.1'

def GetURL(self):
return 'http://www.discogs.com'

def Query(self, handles, status, abort):
result = []
api_key = '783001745d'

for handle in handles:
status.Advance()

if abort.Aborting():
return result

artist = handle.Format("[%artist%]")
album = handle.Format("[%album%]")
album_file = open("C:\Python26\!album.txt", "r")
test = album_file.read()
album_file.close()

try:
if album == test:
text_file = open("C:\Python26\!tag3.txt", "r")
writeout = text_file.read()
text_file.close()
result.append(writeout)
else:
URL_s = 'http://www.discogs.com/search?type=all&q=' + artist.lower().replace(' ','+') + '+' + album.lower().replace(' ','+') + '&f=xml&api_key=' + api_key
request = urllib2.Request(URL_s)
request.add_header('Accept-Encoding', 'gzip')
response = urllib2.urlopen(request)
data = response.read()
unzipped_data = gzip.GzipFile(fileobj = cStringIO.StringIO(data)).read()
res = minidom.parseString(unzipped_data)
uri_1 = res.getElementsByTagName("uri")[0]
rel_id = uri_1.childNodes[0].data.encode('utf-8').rpartition('/')[2]

URL_r = 'http://www.discogs.com/release/' + rel_id + '?f=xml&api_key=' + api_key
request = urllib2.Request(URL_r)
request.add_header('Accept-Encoding', 'gzip')
response = urllib2.urlopen(request)
data = response.read()
unzipped_data = gzip.GzipFile(fileobj = cStringIO.StringIO(data)).read()
xml.etree.ElementTree.fromstring(unzipped_data)

def getGenres(tree):
genres = []
release = tree.find('release')
genreList = release.find('genres')
if genreList:
for i in genreList:
genres.append(i.text)
return genres

def getStyles(tree):
styles = []
release = tree.find('release')
styleList = release.find('styles')
if styleList:
for i in styleList:
styles.append(i.text)
return styles

lyric=getGenres(xml.etree.ElementTree.fromstring(unzipped_data))
lyric.append('//')
lyric.append(getStyles(xml.etree.ElementTree.fromstring(unzipped_data)))
lyric=str(lyric).strip('[').strip(']').replace(',', ';').replace('\'','').replace('[','')
result.append(lyric)
album_file = open("C:\Python26\!album.txt", "w")
album_file.write(album)
album_file.close()
text_file = open("C:\Python26\!tag3.txt", "w")
text_file.write(lyric)
text_file.close()
except Exception, e:
traceback.print_exc(file=sys.stdout)
result.append('')
album_file = open("C:\Python26\!album.txt", "w")
album_file.write(album)
album_file.close()
text_file = open("C:\Python26\!tag3.txt", "w")
text_file.write('')
text_file.close()
continue

return result

if __name__ == "__main__":
LyricProviderInstance = Discogs_GetGenre()


The output to the tag will be "%genre%; //; %style%". I don't know how to get rid of the semicolons just yet, but you can clean those up as you split the tag into genre and style tags.

And yes, it's written quick and dirty. I haven't yet beautified it and the script probably won't look the same tomorrow. tongue.gif

edit :

I've tested it on some 5000 files and it works perfectly smile.gif

This post has been edited by tore: Mar 20 2010, 02:14
Go to the top of the page
+Quote Post

Posts in this topic
- 2E7AH   Python Grabber scripts   May 28 2009, 15:27
- - auldyin   QUOTE (2E7AH @ May 28 2009, 15:27) Some u...   May 28 2009, 18:19
- - 2E7AH   Search format: artist-title. LRC (time-stamped) l...   Jun 22 2009, 23:24
- - 2E7AH   Mistype error corrected and some specific parsing,...   Jun 24 2009, 12:46
- - 2E7AH   Mistype error corrected and some specific parsing,...   Jun 24 2009, 15:15
- - 2E7AH   corrected code page for accented chars etc.   Jun 24 2009, 19:56
- - 2E7AH   RE: Python Grabber scripts   Jun 24 2009, 21:33
- - elenhil   Same error as before: CODEfoo_grabber_python...   Jun 26 2009, 12:40
- - 2E7AH   That can only happen if you don't have artist ...   Jun 26 2009, 15:17
|- - elenhil   QUOTE (2E7AH @ Jun 26 2009, 18:17) That c...   Jun 26 2009, 21:45
- - 2E7AH   Updated AMG review: fasterif fetching multiple rel...   Jun 26 2009, 16:57
|- - tberman333   QUOTE (2E7AH @ Jun 26 2009, 10:57) Update...   Sep 22 2009, 02:46
- - tere   maybe there is a way to make a plugin also www.lol...   Jun 29 2009, 06:24
- - 2E7AH   QUOTE (tere @ Jun 29 2009, 07:24) maybe t...   Jun 29 2009, 23:53
- - 2E7AH   Search format: artist-title. We are the leading ly...   Jun 30 2009, 16:36
- - q-stankovic   Hello 2E7AH! I tried to download some of y...   Sep 22 2009, 17:31
- - 2E7AH   Some strange interest in scripts today I'll up...   Sep 23 2009, 20:05
- - q-stankovic   QUOTE (2E7AH @ Sep 23 2009, 21:05) Sorry ...   Sep 25 2009, 17:18
- - 2E7AH   hm... I don't know what you mean with database...   Sep 25 2009, 17:37
- - 2E7AH   Also maybe isn't so good idea making lyrics gr...   Sep 25 2009, 18:56
- - q-stankovic   QUOTE (2E7AH @ Sep 25 2009, 18:37) I...   Sep 26 2009, 15:07
- - 2E7AH   Here are updated AMG scripts: - AMG_Genre.py -...   Oct 11 2009, 07:04
- - 2E7AH   Updated AMG review script: - changed %artist% to...   Oct 16 2009, 14:40
- - Benji99   Hey 2E7AH, Any chance you could make a script for...   Oct 29 2009, 02:32
- - 2E7AH   Aren't you from Canada? It looks easy to find...   Oct 29 2009, 02:58
|- - Benji99   QUOTE (2E7AH @ Oct 28 2009, 21:58) Aren...   Oct 29 2009, 03:13
- - 2E7AH   I'll try this one, it should be OK (although d...   Oct 29 2009, 03:19
|- - Benji99   QUOTE (2E7AH @ Oct 28 2009, 22:19) I...   Oct 29 2009, 03:36
- - 2E7AH   Benji99, here is the script: I've tested it ...   Oct 31 2009, 06:12
|- - Benji99   QUOTE (2E7AH @ Oct 31 2009, 00:12) Benji9...   Nov 11 2009, 07:29
|- - 2E7AH   QUOTE (Benji99 @ Nov 11 2009, 08:29) 1st ...   Nov 11 2009, 08:05
|- - Benji99   QUOTE (2E7AH @ Nov 11 2009, 02:05) Is tha...   Nov 21 2009, 04:31
- - 2E7AH   Once again AMG scripts Now GENRE, STYLE, MOOD an...   Nov 10 2009, 09:31
- - 2E7AH   Download this AMG release script: Problem with pr...   Nov 10 2009, 11:08
|- - tberman333   Thank you! This is great! QUOTE (2E7AH ...   Nov 10 2009, 15:17
- - 2E7AH   Enjoy I didn't forgot about composer/performe...   Nov 10 2009, 16:22
|- - tberman333   QUOTE (2E7AH @ Nov 10 2009, 10:22) Enjoy ...   Nov 10 2009, 18:10
|- - Benji99   QUOTE (2E7AH @ Nov 10 2009, 11:22) Enjoy ...   Mar 27 2010, 23:15
- - T.P Wang   @2E7AH: I think replace the u'\uff5e...   Nov 21 2009, 03:09
- - garbanzo.bean   2E7AH, i'm trying to use your AMG script, but ...   Jan 8 2010, 15:54
|- - tberman333   QUOTE (garbanzo.bean @ Jan 8 2010, 09:54)...   Jan 8 2010, 16:46
- - tore   I have a quick question regarding the python disco...   Mar 15 2010, 12:29
|- - 2E7AH   QUOTE (tore @ Mar 15 2010, 12:29) Would i...   Mar 15 2010, 13:15
- - tore   By the way, I did find this little piece of inform...   Mar 15 2010, 15:05
- - tore   I've now tried playing around with your last.f...   Mar 15 2010, 20:11
|- - 2E7AH   look at the other last.fm script, i.e. CODEtoptag...   Mar 15 2010, 20:28
|- - tore   QUOTE (2E7AH @ Mar 15 2010, 20:28) look a...   Mar 15 2010, 20:38
|- - 2E7AH   I just pasted that part from that other last.fm sc...   Mar 15 2010, 20:48
|- - tore   QUOTE (2E7AH @ Mar 15 2010, 20:48) I just...   Mar 15 2010, 21:38
- - 2E7AH   hey, why ride motorcycle when I have my bike le...   Mar 15 2010, 21:51
|- - tore   QUOTE (2E7AH @ Mar 15 2010, 21:51) hey, w...   Mar 15 2010, 22:34
|- - 2E7AH   QUOTE (tore @ Mar 15 2010, 22:34) I made ...   Mar 15 2010, 23:00
|- - tore   QUOTE (2E7AH @ Mar 15 2010, 23:00) QUOTE ...   Mar 15 2010, 23:14
|- - tore   For an example of what information could be pulled...   Mar 16 2010, 00:26
- - tore   To anyone having a problem with the discogs 5000 r...   Mar 19 2010, 15:05
|- - 2E7AH   I didn't bothered with 5000 limit on Discogs, ...   Mar 19 2010, 18:52
- - tore   Here's a modification of what I described abov...   Mar 20 2010, 02:03
- - xtr   Is it possible to use these scripts with a foo_run...   Mar 21 2010, 23:05
- - 2E7AH   If you have python, I guess you know something abo...   Mar 21 2010, 23:21
- - 2E7AH   Are you really counting on that Script is there,...   Mar 29 2010, 00:10
- - Benji99   I was thinking of writing a small program (evaluat...   Mar 31 2010, 21:13
- - 2E7AH   I'm just working on the script, and it's a...   Apr 1 2010, 03:15
|- - Benji99   QUOTE (2E7AH @ Mar 31 2010, 22:15) I...   Apr 4 2010, 22:15
- - Hitchhiker427   Does anyone still have 2E7AH's scripts for Lyr...   Aug 6 2010, 00:55
- - grimes   Found a Lyrics-site with API: ChartLyrics Here is...   Aug 7 2010, 21:03
|- - Hitchhiker427   QUOTE (grimes @ Aug 7 2010, 15:03) Found ...   Aug 9 2010, 20:14
- - Beta4Me   For all that are interested...I am coding in PHP a...   Oct 6 2010, 13:06
|- - tberman333   QUOTE (Beta4Me @ Oct 6 2010, 08:06) For a...   Oct 23 2010, 23:05
|- - Beta4Me   QUOTE (tberman333 @ Oct 24 2010, 08:05) Q...   Oct 24 2010, 06:38
- - lo-fi   yep. amg scripts not working here either   Oct 24 2010, 00:41
|- - Decalicatan_Decalicatan   QUOTE (lo-fi @ Oct 24 2010, 01:41) y...   Oct 24 2010, 09:41
- - Wooops   Since most of the script links seem to be broken -...   Oct 25 2010, 05:54
- - Beta4Me   How did the original python grabber scripts work? ...   Oct 26 2010, 02:47
- - mrinferno   AMG redesigned their site and search methods mid-O...   Oct 26 2010, 02:53
|- - Beta4Me   QUOTE (mrinferno @ Oct 26 2010, 12:53) AM...   Oct 26 2010, 03:59
|- - tberman333   QUOTE (Beta4Me @ Oct 25 2010, 22:59) QUOT...   Oct 26 2010, 21:16
|- - Beta4Me   QUOTE (tberman333 @ Oct 27 2010, 07:16) Q...   Oct 26 2010, 23:25
|- - tberman333   QUOTE (Beta4Me @ Oct 26 2010, 18:25) Than...   Oct 27 2010, 01:37
- - Decalicatan_Decalicatan   I have uploaded one of the 2E7AH amg scripts (from...   Oct 26 2010, 11:10
|- - Beta4Me   QUOTE (Decalicatan_Decalicatan @ Oct 26 2010,...   Oct 26 2010, 11:15
|- - Schuft   QUOTE (Decalicatan_Decalicatan @ Oct 26 2010,...   Feb 13 2011, 09:41
|- - klonuo   QUOTE (Schuft @ Feb 13 2011, 10:41) Is th...   Feb 13 2011, 16:53
- - Schuft   Ok - you're right - phyton installation was mi...   Feb 16 2011, 18:37
- - MDMA   Would it be possible to use this component to fetc...   Feb 8 2012, 16:21
- - Fractal_Mortality   Does anyone know where i could find updated script...   Feb 3 2013, 20:30


Reply to this topicStart new topic
1 User(s) are reading this topic (1 Guests and 0 Anonymous Users)
0 Members:

 



RSS Lo-Fi Version Time is now: 20th April 2014 - 22:26