PDA

View Full Version : [Python] |2eap's Basic Neopets Archive



|2eap
02-08-2017, 04:23 AM
I'm posting this as a reference for myself and for others as my drive and other cloud DB's are cluttered.

The code below is a collection of definitions I've collected and tested. Some are adopted, others I wrote myself. Majority I at least made adjustments to, to make them smooth and function properly.
I'm only posting basic functions for tasks around the account. Some other functions I've developed I can't part with for obvious reasons.


Majority of functions below loop until everything has been cleared/done at the given task assuming pin has been disabled


UPDATED: 2/26/2017
Contribute?

Some shed items are glitched returning errors when included and removal. Need to be able to clear out shed still when this happens.
Anything else or suggestions.

Change Log

FOUND BUGS - Fixed version: SAVED
Added TCGS, neodeck, closet, shed
Added date functions for various things.


#################################### Item Dumping Functions ######################################

# Credits Ghosts & Darkbyte for adopted code.
# Assuming Pin has been disabled.

#################################### Item Dumping Functions ######################################





#Fixed | glitched items caused it to error, too lengthy and time consuming to remove items 1 by 1 and manuever around glitched items.
#[Only registered and activated users can see links]
def DumpShed(acc):
print " Dumping Shed Items"
while 1==1:
html = acc.get("[Only registered and activated users can see links]")
sheditemscheck = html.count("<input type='text' name='")
if sheditemscheck == 0:
print " Status: done removing shed items"
break
else:
time.sleep(2)
#do this and break if last page.
#strip html
pos1 = html.find("<tr class='contentModuleHeaderAlt'>")
pos2 = html.find("</table>")
itemshtml = html[pos1:pos2]
postdata = {}
postdata['page'] = '1'
#loop through items
startpos = 0
while itemshtml.find("<input type='text' name='", startpos) != -1:
#find input name
pos3 = itemshtml.find("<input type='text' name='", startpos) + 25
pos4 = itemshtml.find("'", pos3)
#find item qty
pos5 = itemshtml.find("<td align='center'><strong>", startpos) + len("<td align='center'><strong>")
pos6 = itemshtml.find("<td align='center'><strong>", pos5) + len("<td align='center'><strong>")
pos7 = itemshtml.find('</strong>', pos6)
postdata[itemshtml[pos3:pos4]] = itemshtml[pos6:pos7]
time.sleep(2)
htmlcheck = acc.post('[Only registered and activated users can see links]', postdata, '[Only registered and activated users can see links]')
if htmlcheck.find("move the selected Neopoint items"):
print " Status: glitched items in shed. done removing shed items"
break

#Fixed | Cycles through each, clean code, goes directly to necessary pages, no dilly dallying. added wait times to make more human since lots of moving parts and requests.
def DumpTCGS(acc, username):
print " Dumping TCG's"
while 1==1:
html = acc.get("[Only registered and activated users can see links]")
pos1 = html.find('<strong>Total Cards:&nbsp;&nbsp;') + len('<strong>Total Cards:&nbsp;&nbsp;')
pos2 = html.find('</strong>',pos1)
totalCards = html[pos1:pos2]
if totalCards == '0':
print " Status: done removing TCG's"
break
else:
time.sleep(2)
#remove TCGs
#grab which editions have TCG's so we aren't opening every page. lets be efficient.
position1 = html.find("<tr align='center'>")
position2 = html.find("</table>", position1)
editionshtml = html[position1:position2]
EDITIONS = []
startpos = 0

while editionshtml.find("<a href=", startpos) != -1:
pos3 = editionshtml.find('<a href="', startpos) + 9
pos6 = editionshtml.find('"', pos3)
editionsurl = editionshtml[pos3:pos6]
pos4 = editionshtml.find("(", pos3) + 1
pos5 = editionshtml.find(")", pos4)
ednum = editionshtml[pos4:pos5]
if int(ednum) > 0:
EDITIONS.append(editionsurl)
startpos = pos5
for x in EDITIONS:

stpos = 0
html2 = acc.get('[Only registered and activated users can see links]' + x, '[Only registered and activated users can see links]')
time.sleep(1)
#strip html to make it easier to find data.
stripos1 = html2.find('<strong>Pages:</strong>', stpos)
stripos2 = html2.find('</table>', stripos1)
pageshtml = html2[stripos1:stripos2]
pages = []
while pageshtml.find("<font color='red'>", stpos) != -1:
posit1 = pageshtml.find("<font color='red'>", stpos) + 18
posit2 = pageshtml.find("</font>", posit1)
pages.append(pageshtml[posit1:posit2])
stpos = posit2
for page in pages:

offset = str((int(page)*9) - 9)
html3 = acc.get('[Only registered and activated users can see links]' + x + '&offset=' + offset)
time.sleep(2)
starpos = 0
car** = []
while html3.find('<img onclick="viewcard', starpos) != -1:

posit1 = html3.find('<img onclick="viewcard', starpos) + 23
posit2 = html3.find(')', posit1)
car**.append(html3[posit1:posit2])
starpos = posit2
for card in car**:

#Finally open the cards window and remove from TCG Album
edition = x.split('edition=')[1]
URL = '[Only registered and activated users can see links]' + str(username) + '&edition=' + str(edition) + '&card=' + str(card)
acc.get(URL)
time.sleep(1)

postdata = {}
postdata['action'] = 'rc'
postdata['owner'] = str(username)
postdata['card'] = str(card)
postdata['edition'] = str(edition)
acc.post(URL, postdata, URL)
print " Status: finished removing items from edition - " + edition

#Fixed | Wasn't properly removing closet items. Added count. Removed break to double check. Removed un-needed elif
def DumpCloset(acc):
print " Dumping items in Closet"
while 1==1:
html = acc.get("[Only registered and activated users can see links]")
itemcount = html.count("<input type='text'")
if int(itemcount) == 0:
print " Status: Done dumping Closet"
break
else:
time.sleep(2)
#do this and break at the end.
#trim html first
pos1 = html.find('<b>Items:</b>') + 13
pos2 = html.find('<b>Items:</b>', pos1)
newhtml = html[pos1:pos2]
postdata = {}
postdata["page"] = '1'
#Now loop through items
startpos = 0
while newhtml.find("<input type='text'", startpos) != -1:
#find item qty
position5 = newhtml.find("<td align='center'><b>", startpos) + 22
position6 = newhtml.find("<td align='center'><b>", position5) + 22
position7 = newhtml.find("</b>", position6)
itemQTY = newhtml[position6:position7]
position1 = newhtml.find("<input type='text'", position7) + 25
position2 = newhtml.find("'", position1)
itemID = newhtml[position1:position2]
startpos = position2
postdata[itemID] = itemQTY
acc.post('[Only registered and activated users can see links]', postdata, '[Only registered and activated users can see links]')
print " Status: Removed page of closet items."

#Fixed | added shop exist check, more details, got rid of unneeded elif by adding item counter instead. Result, much shorter code.
def DumpShop(acc):
print " Dumping Shop"
while 1==1: #Loop until broken
html = acc.get("[Only registered and activated users can see links]")
shopexistcheck = html.count('(size')
itemcounts = html.count("<input type='hidden' name='obj_id_")
if shopexistcheck == 0:
print " Status: User has no shop"
break
elif itemcounts == 0:
print " Status: done removing items"
break
else:
time.sleep(2)
#do this and break at the end
postdata = {}
postdata["type"] = "update_prices"
postdata["order_by"] = "price"
postdata["view"] = ""
postdata["lim"] = "30"
postdata["obj_name"] = ""
start_position = 0
while html.find("height=80 width=80>", start_position) != -1:
pos1 = html.find("<input type='hidden' name='obj_id_", start_position) + 28
pos2 = html.find("'", pos1)
pos3 = html.find("value='", pos2) + 7
pos4 = html.find("'", pos3)
postdata[html[pos1:pos2]] = html[pos3:pos4]
pos5 = html.find("name='oldcost_", pos4) + 6
pos6 = html.find("'", pos5)
pos7 = html.find("value='", pos6) + 7
pos8 = html.find("'", pos7)
postdata[html[pos5:pos6]] = html[pos7:pos8]
pos9 = html.find("name='", pos8) + 6
pos10 = html.find("'", pos9)
postdata[html[pos9:pos10]] = "0"
pos11 = html.find("back_to_inv[", pos10)
pos12 = html.find("]", pos11) + 1
pos13 = html.find("<td width=50 bgcolor='#ffffcc' align=center><b>", start_position) + 47
pos14 = html.find("</b>", pos13)
count = html[pos13:pos14]
postdata[html[pos11:pos12]] = count

start_position = pos4
acc.post("[Only registered and activated users can see links]", postdata, "[Only registered and activated users can see links]")


#Fixed
def DumpGallery(acc):
print " Dumping Gallery"
while 1==1: #Loop through to double check we got it all
html = acc.get("[Only registered and activated users can see links]")
itemcount = html.count("remove_arr[")
if itemcount == 0:
print " Status: Done Removing Gallery Items"
break
else:
time.sleep(2)
postdata = {}
postdata['dowhat'] = 'remove'
postdata['hidden_user_cat'] = ''
postdata['view'] = 'all'

pos4 = 0
while html.find("remove_arr[", pos4) != -1:
pos1 = html.find("remove_arr[", pos4)
pos2 = html.find("]", pos1) + 1
pos3 = html.find('Qty</option><option value="', pos2) + 27
pos4 = html.find('"', pos3)
postdata[html[pos1:pos2]] = html[pos3:pos4]

acc.post("[Only registered and activated users can see links]", postdata)
print " Status: Removing Gallery Items"

#Fixed
def DumpInventory(acc):
print " Dumping Inventory"
while 1==1: # loop until broken
html = acc.get('[Only registered and activated users can see links]')
items = html.count("id_arr")
NCitems = html.count("cash_radio_arr") / 3
strip_position_1 = html.find('<b>Shed</b>')
strip_position_2 = html.find('Check All', strip_position_1)
html = html[strip_position_1:strip_position_2]
if items == 0 and NCitems == 0:
print " Status: inventory empty"
break
elif NCitems > 0:
time.sleep(1)
# find NC items first then finish going through.
NC_item_array = []
startpos = 0
while html.find('cash_radio_arr', startpos) > 1:
pos1 = html.find('cash_radio_arr[', startpos) + 15
pos2 = html.find(']', pos1)
pos3 = html.find("'closet'", pos2)
startpos = pos3
NC_item_array.append(html[pos1:pos2])

postdata = {}
postdata["buyitem"] = "0"
counter = 1
for items_id in NC_item_array:
postdata["cash_radio_arr[" + str(items_id) + "]"] = "deposit"
counter +=1
if counter == 30:
break
acc.post('[Only registered and activated users can see links]', postdata)
print " Status: moved neocash items to SDB"
else:
time.sleep(2)
item_array = []
# find NP items
startpos = 0
while html.find('id_arr', startpos) > 1:
pos1 = html.find('value="', startpos) + 7
pos2 = html.find('"', pos1)
the_data = html[pos1:pos2]
startpos = pos2
if the_data.isdigit():
if len(item_array) < 70:
item_array.append(the_data)

postdata = {}
postdata["buyitem"] = "0"
counter = 1

for item_id in item_array:
postdata["id_arr[" + str(counter) + "]"] = item_id
postdata["radio_arr[" + str(counter) + "]"] = "deposit"
counter += 1
if counter == 70:
break

acc.post('[Only registered and activated users can see links]', postdata)
print " Status: moved neopoint items to SDB"

#Fixed
def DumpShopTill(acc):
print " Dumping Shop Till"
while 1==1: #loop to ensure we got it all
html = acc.get("[Only registered and activated users can see links]")
pos1 = html.find("You currently have <b>") + 22
pos2 = html.find(" ", pos1)
shopexistcheck = html.count('You currently have <b>')
neopoints = html[pos1:pos2].replace(",", "")
if int(shopexistcheck) == 0:
print " Status: No shop"
break
elif neopoints == "0":
print " Status: No neopoints in shop till"
break
else:
time.sleep(1)
if int(neopoints) > 50000000:
neopoints = '50000000'

postdata = {}
postdata['type'] = 'withdraw'
postdata['amount'] = neopoints

acc.post("[Only registered and activated users can see links]", postdata)

#Fixed
def DumpPureToBank(acc):
print " Dumping pure to bank"
while 1==1: #loop forver until broken to double check we got it all.
html = acc.get("[Only registered and activated users can see links]")
bankexistcheck = html.count('Current Balance:</td>')
pos1 = html.find('"/inventory.phtml">') + 19
pos2 = html.find("<", pos1)
pure = ""
for char in list(html[pos1:pos2]):
if char.isdigit():
pure += char
pure = float(pure) # pure on hand

if bankexistcheck == 0:
time.sleep(3)
print " Satus: creating bank and depositing pure"
postdata = {}
postdata['type'] = 'new_account'
postdata['name'] = acc.user
postdata['add1'] = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(int(random.random() * 20)))
postdata['add2'] = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(int(random.random() * 20)))
postdata['add3'] = ''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(int(random.random() * 20)))
postdata['employment'] = 'Chia Custodian'
postdata['salary'] = '10,000 NP and below'
if pure > 2500:
postdata['account_type'] = '2'
elif pure > 5000:
postdata['account_type'] = '3'
elif pure > 1000000:
postdata['account_type'] = '11'
elif pure > 2000000:
postdata['account_type'] = '12'
elif pure > 5000000:
postdata['account_type'] = '13'
elif pure > 7500000:
postdata['account_type'] = '14'
elif pure > 10000000:
postdata['account_type'] = '15'
else:
postdata['account_type'] = '1'
pure = int(pure / 500000 * 500000)
postdata['initial_deposit'] = str(pure)
html = acc.post("[Only registered and activated users can see links]", postdata)
f = open('creating bank page.html', 'w')
f.write(html)
elif pure == 0:
print " Status: Finished depositing pure"
break

else:
time.sleep(2)
print " Status: depositing pure on hand to bank."
postdata = {}
postdata['type'] = 'deposit'
pure = int(pure / 500000 * 500000)
postdata['amount'] = str(pure)
html = acc.post("[Only registered and activated users can see links]", postdata)
f = open('deposition pure on hand to bank.html', 'w')
f.write(html)

#Fixed
def DumpStocks(acc):
print " Dumping Stocks"
while 1==1:
html = acc.get("[Only registered and activated users can see links]")
pos1 = html.find("<table align=center cellpadding=3 cellspacing=0 border=1>")
pos2 = html.find('<input type="hidden" name="type" value="sell">')
stocktable = html[pos1:pos2]

if html.find("<table align=center cellpadding=3 cellspacing=0 border=1>") == -1 or html.find('<td align="center"><b>0</b></td>') != -1:
print " Status: Done removing stocks"
break
else:
time.sleep(2)
postdata = {}
pos1 = html.find("_ref_ck=") + 8
pos2 = html.find("'", pos1)
postdata['_ref_ck'] = html[pos1:pos2]
postdata['type'] = 'sell'

iCount = 0
totalValue = 0

pos2 = 0
while stocktable.find("sell[", pos2) != -1 and iCount < 400 and totalValue < 75000000:
iCount += 1
pos1 = stocktable.find("sell[", pos2)
pos2 = stocktable.find('"', pos1)
pos3 = stocktable.rfind('<tr>', 0, pos2)
pos4 = stocktable.find('<td align="center">', pos3) + 19
pos5 = stocktable.find("<", pos4)
postdata[stocktable[pos1:pos2]] = stocktable[pos4:pos5].replace(",", "")
pos6 = stocktable.find('<td align="center">', pos5) + 19
pos7 = stocktable.find("<", pos6)
price = int(stocktable[pos4:pos5].replace(",", ""))
qty = int(stocktable[pos6:pos7].replace(",", ""))
totalValue += price * qty

acc.post("[Only registered and activated users can see links]", postdata)

#Fixed
def DumpWeapons(acc):
print " Removing attached BD weapons"
while 1==1: #Loop until broken to make sure we got them all
#Cases of same item attached to two different pet show up as one item.
html = acc.get("[Only registered and activated users can see links]")

items = []


if html.find('"name":"') == -1:
print " Status: Done Removing BD Weapons"
break
else:
position2 = 0
while html.find('"name":"', position2) != -1:
position1 = html.find('"name":"', position2) + 8
position2 = html.find('"', position1)
items.append(html[position1:position2])

for item in items:
position3 = html.rfind('"oii":"', 0, position1) + 7
position4 = html.find('"', position3)

position5 = html.find(item, position2)
position6 = html.rfind('<div class="petName">', 0, position5) + 21
position7 = html.find("<", position6)

postdata = {}
postdata['gucid'] = ''
postdata['oii'] = html[position3:position4]
postdata['petName'] = html[position6:position7]

acc.post("[Only registered and activated users can see links]", postdata)

#Fixed
def DumpNeodeck(acc, username):
print " Removing neodeck cards"
while 1==1: #Check over ourself until sure cards are removed.

html = acc.get("[Only registered and activated users can see links]" + username + "&show=cards")
neodecknum = html.count("href='process_neodeck.phtml")

if int(neodecknum) == 0:
print " Status: neodeck empty"
break
else:
time.sleep(1)
cards = []
position2 = 0
while html.find("border='0'><br><b>", position2) != -1:
position1 = html.find("border='0'><br><b>", position2) + 18
position2 = html.find("<", position1)
cards.append(html[position1:position2])

for card in cards:
pos1 = html.find(card)
pos2 = html.find("href='process_neodeck.phtml", pos1) + 6
pos3 = html.find("'", pos2)

acc.get("[Only registered and activated users can see links]" + html[pos2:pos3])







General Non-Neopets Functions


#### Date Functions ####
def waitToTomorrow(): # returns time to wait until next day. Cen edit the time delta to further your wait, or wait until a certain time of next day.
tomorrow = datetime.datetime.replace(datetime.datetime.now() + datetime.timedelta(days=1),
hour=0, minute=0, second=0)
delta = tomorrow - datetime.datetime.now()
return delta.seconds

def last_day_of_month(any_day): #returns same format as put in
next_month = any_day.replace(day=28) + datetime.timedelta(days=4) # this will never fail
return next_month - datetime.timedelta(days=next_month.day)

def formatFIRSTDAY(): #grabs first day, and converts to Gregorian.
date_format = "%Y-%m-%dT%H:%M:%S"
FIRSTDAY = datetime.datetime.replace(datetime.datetime.now(),
day=1, hour=0, minute=0, second=0)
return FIRSTDAY.strftime(date_format)

def sendEmail(receivingemail, BODY)
#Can retry attempts with a loop. make sure return false is no longer in the try exception and outside of for loop.
#ex. for i in range(#numberoftrys):
try:
FROM = #Your email string
TO = receivingemail
SUBJECT = #Your subject string

body = string.join((
"From: %s" % FROM,
"To: %s" % TO,
"Subject: %s" % SUBJECT,
"",
BODY), "\r\n")

server = smtplib.SMTP(#your smtp domain string, #Your STMP port)
server.ehlo()
server.login(#youremail string, #youremail password string)
server.sendmail(FROM, TO, body)
return True
except:
print "error sending email"
return False



JellyNeo Item Info Grabber




import urllib
import urllib2



def grabfooditems2():

startpage = "0"
outputlist = []
while 1==1:#Loop forever (well until we call break)

url = "[Only registered and activated users can see links][]=70&cat[]=81&cat[]=3&cat[]=49&cat[]=27&cat[]=22&cat[]=25&cat[]=101&cat[]=41&cat[]=1&cat[]=48&cat[]=20&cat[]=42&cat[]=43&cat[]=46&cat[]=47&cat[]=40&cat[]=44&cat[]=80&cat[]=117&cat[]=65&cat[]=68&cat[]=17&cat[]=8&cat[]=11&cat[]=7&cat[]=45&min_price=1&max_price=5000&limit=100&status=1&start=" + startpage
response = urllib2.urlopen(url)
html = response.read()
slichtml1 = html.find("results for your search.</p>") #Slim down the html a bit to remove all the other shit we dont need
slicehtml2 = html.find("itemdb-sidebar-search",slichtml1) #Slim down the html a bit to remove all the other shit we dont need
newhtml = html[slichtml1:slicehtml2]
#print newhtml
currentposition = 0





#Html chunk....
###<li>
###<a href="/item/4463/"><img src="[Only registered and activated users can see links]" alt="1/3 Twirly Fruit Omelette" title="1/3 Twirly Fruit Omelette" class="item-result-image"></a><br><a href="/item/4463/">1/3 Twirly Fruit Omelette</a>
###<br><span class="text-small"><a href="/item/4463/price-history/" class="price-history-link" title="November 11, 2016">170 NP</a></span>
##</li>
##<li>
print "extracting shit from page number " + startpage
if not newhtml.find('<a href="/item/',currentposition) >1:
print "Found page end"
return outputlist
while newhtml.find('<a href="/item/', currentposition)>1:
# Get URL for Item Page
currentloopitems = []
pos1 = newhtml.find('<a href="/item/',currentposition) + 15
pos2 = newhtml.find('/', pos1)
itemid = (newhtml[pos1:pos2])
itemid = itemid


pos3 = newhtml.find(' alt="', pos2)+6
pos4 = newhtml.find('" ', pos3)
itemname = newhtml[pos3:pos4]

currentposition = newhtml.find("</span>",pos2) #Jump over the extra link it extracted before


currentloopitem = {"ItemName" : itemname ,"Itemid" : itemid} #Make a dictionary object
outputlist.append(currentloopitem)
startpage = str(int(startpage) + 100)
currentposition = 0



Itemlist = grabfooditems2()
for item in Itemlist:
print item["ItemName"] + "\t" + item["Itemid"]




More coming as I create/perfect them.

Zachafer
02-08-2017, 11:38 PM
@|2eap ([Only registered and activated users can see links]) nice post
For Jellyneo Item DB, use the printer-friendly view to see all results on one page.
[Only registered and activated users can see links][...]

|2eap
02-09-2017, 12:00 AM
@|2eap ([Only registered and activated users can see links]) nice post
For Jellyneo Item DB, use the printer-friendly view to see all results on one page.
[Only registered and activated users can see links][...]


I was looking for the url, does it still exist or did they get rid of the printer friendly version?

Zachafer
02-09-2017, 12:03 AM
I was looking for the url, does it still exist or did they get rid of the printer friendly version?

It's still there. You can just add "print/" in the URL but the button is here:


[Only registered and activated users can see links]

|2eap
02-09-2017, 12:11 AM
It's still there. You can just add "print/" in the URL but the button is here:


[Only registered and activated users can see links]

Well I'll be damned.
If it were a snake it woulda bit me!

Curious
02-27-2017, 12:07 AM
Cool. This gave me an idea.

I was able to harvest the data by classes, download each image, group items by object, and convert prices to integers using some of your code here.

Made it SUPER easy to add a method that converts the integers to a USD price.

Thanks, big help!

|2eap
02-27-2017, 02:03 AM
Cool. This gave me an idea.

I was able to harvest the data by classes, download each image, group items by object, and convert prices to integers using some of your code here.

Made it SUPER easy to add a method that converts the integers to a USD price.

Thanks, big help!

Solid solid, for more accurate prices I recommend using Mopheads price guide.

But perfect to grab each items info. Hopefully made it easier to harvest Jellyneos DB to create your own :)