[Various] HTTP Wrappers

[Miguel]

Various HTTP Wrappers, if you have any to add to the list, post them here.

VB6
Gluraks ([Only registered and activated users can see links])
RipperWrapper ([Only registered and activated users can see links])
NiteWrapper ([Only registered and activated users can see links])
cxWrapper ([Only registered and activated users can see links])
HaloWrapper ([Only registered and activated users can see links])

VB.NET
cxWrapper ([Only registered and activated users can see links])

C#
cxWrapper ([Only registered and activated users can see links])
mBot HTTP Wrapper ([Only registered and activated users can see links])
SD[Only registered and activated users can see links] ([Only registered and activated users can see links])

Java
Zer0's Wrapper ([Only registered and activated users can see links])

[Reemer]

Java HTTP Wrapper
Written by Zer0,


import java.io.*;
import java.net.*;
import java.util.*;
import java.util.zip.*;
import java.awt.image.BufferedImage;
import javax.imageio.ImageIO;

public class Connection implements Serializable {
static final long serialVersionUID = 1L;

String domain, referer;
Map<String,String> cookies;

static String rpUseragent = "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.14) Gecko/20080509 Firefox/2.0.0.14";
static String rpAcceptText = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
static String rpAcceptPng = "image/png,image/*;q=0.8,*/*;q=0.5";
static String rpAcceptLanguage = "en-us,en;q=0.5";
static String rpAcceptEncoding = "gzip, deflate";
static String rpAcceptCharset = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";
static String rpKeepAlive = "300";
static String rpConnection = "keep-alive";
static String rpContentType = "application/x-[Only registered and activated users can see links]";

public Connection( String domain, Map<String,String> cookies, String referer ) {
this.domain = domain;
this.cookies = cookies;
this.referer = referer;
}

public Connection( String domain, Map<String,String> cookies ) {
this( domain, cookies, null );
}

public Connection( String domain, String referer ) {
this( domain, new HashMap<String,String>(), referer );
}

public Connection( String domain ) {
this( domain, new HashMap<String,String>(), null );
}

public String get( String url ) {
if( url.charAt( 0 ) == '/' )
url = domain + url;

try {
[Only registered and activated users can see links] conn = ([Only registered and activated users can see links])( new URL( url.replaceAll( " ", "%20" ) ).openConnection() );
setRequestProperties( conn );
conn.setRequestMethod( "GET" );
referer = url;
return read( conn );
} catch( IOException e1 ) {
e1.printStackTrace();
return null;
}
}

public String post( String url, String[][] data ) {
if( url.charAt( 0 ) == '/' )
url = domain + url;

try {
[Only registered and activated users can see links] conn = ([Only registered and activated users can see links])( new URL( url.replaceAll( " ", "%20" ) ).openConnection() );
setRequestProperties( conn );
conn.setRequestMethod( "POST" );
conn.setDoOutput( true );

StringBuilder sb = new StringBuilder();

for( int i = 0; i < data[0].length; i++ )
sb.append( URLEncoder.encode( data[0][i], "UTF-8" ) ).append( '=' ).append( URLEncoder.encode( data[1][i], "UTF-8" ) ).append( '&' );

conn.setRequestProperty( "Content-Type", rpContentType );
conn.setRequestProperty( "Content-Length", Integer.toString( sb.length()-1 ) );

PrintWriter out = new PrintWriter( new BufferedWriter( new OutputStreamWriter( conn.getOutputStream() ) ) );
out.write( sb.substring( 0, sb.length()-1 ) );
out.close();

referer = url;
return read( conn );
} catch( IOException e1 ) {
e1.printStackTrace();
return null;
}
}

public BufferedImage getImage( String url ) {
try {
[Only registered and activated users can see links] conn = ([Only registered and activated users can see links])( new URL( ( url.charAt( 0 ) == '/' ? domain+url : url ).replaceAll( " ", "%20" ) ).openConnection() );
setRequestProperties( conn );
conn.setRequestMethod( "GET" );
conn.setRequestProperty( "Accept", rpAcceptPng );
return ImageIO.read( conn.getInputStream() );
} catch( IOException e1 ) {
e1.printStackTrace();
return null;
}
}

public boolean hasCookie( String key ) {
return cookies.containsKey( key );
}

public String getCookieString() {
StringBuilder sb = new StringBuilder();

for( String s : cookies.keySet() )
sb.append( s ).append( '=' ).append( cookies.get( s ) ).append( ';' );

return sb.toString();
}

private void setRequestProperties( [Only registered and activated users can see links] conn ) {
conn.setInstanceFollowRedirects( false );
conn.setRequestProperty( "User-Agent", rpUseragent );
conn.setRequestProperty( "Accept", rpAcceptText );
conn.setRequestProperty( "Accept-Language", rpAcceptLanguage );
conn.setRequestProperty( "Accept-Encoding", rpAcceptEncoding );
conn.setRequestProperty( "Accept-Charset", rpAcceptCharset );
conn.setRequestProperty( "Keep-Alive", rpKeepAlive );
conn.setRequestProperty( "Connection", rpConnection );

if( referer != null && referer.length() != 0 )
conn.setRequestProperty( "Referer", referer );

if( cookies != null && cookies.size() != 0 )
conn.setRequestProperty( "Cookie", getCookieString() );
}

private String read( [Only registered and activated users can see links] conn ) throws IOException {
BufferedReader in = null;

if( conn.getContentEncoding() == null )
in = new BufferedReader( new InputStreamReader( conn.getInputStream() ) );
else
if( conn.getContentEncoding().equalsIgnoreCase( "gzip" ) )
in = new BufferedReader( new InputStreamReader( new GZIPInputStream( conn.getInputStream() ) ) );
else if( conn.getContentEncoding().equalsIgnoreCase( "deflate" ) )
in = new BufferedReader( new InputStreamReader( new InflaterInputStream( conn.getInputStream(), new Inflater( true ) ) ) );

StringBuilder sb = new StringBuilder();
String s;

while( ( s = in.readLine() ) != null )
sb.append( s ).append( '\n' );

putCookies( conn.getHeaderFields().get( "Set-Cookie" ) );
return sb.toString();
}

private void putCookies( List<String> cookieList ) {
if( cookieList == null )
return;

int index;

for( String cookie : cookieList )
cookies.put( cookie.substring( 0, index = cookie.indexOf( '=' ) ), cookie.substring( index+1, cookie.indexOf( ';', index ) ) );
}
}

[j03]

./m Y U NO POST PYTHON WRAPPER?

[Miguel]

Because I need to salvage it off one of three TB hard drives and that would be a major pain in the ass

[j03]

Trust me, it'll be worth it in the end. :P

[Zachafer]

NiteWrapper for VB6.

One of my favorite wrappers (has a built-in fast OCR)

[Zachafer]

cxWrapper by cx323 (VB6, C#, VBNET)


Private Declare Function Req Lib "cxWrapper.dll" (ByVal Method As String, ByVal URL As String, ByVal Referrer As String) As String
Private Declare Function DownloadImage Lib "cxWrapper.dll" (ByVal Method As String, ByVal URL As String, ByVal Referrer As String, ByVal Path As String) As Boolean
Private Declare Sub ClearCookies Lib "cxWrapper.dll" ()
Private Declare Sub ClearStoredCookies Lib "cxWrapper.dll" ()
Private Declare Function StoreCookies Lib "cxWrapper.dll" (ByVal key As String, ByVal erase_current As Boolean) As Boolean
Private Declare Function LoadCookies Lib "cxWrapper.dll" (ByVal key As String) As Boolean
Private Declare Function SetUA Lib "cxWrapper.dll" (ByVal UserAgent As String) As Boolean
Private Declare Function IpicDownload Lib "cxWrapper.dll" (ByVal Method As String, ByVal URL As String, ByVal Referrer As String) As IPicture

[Zachafer]

VB6 HaloWrapper by Raui

[Miguel]

Wondering if I can get average load times of google.com for each of these wrappers?

[Zachafer]

Wondering if I can get average load times of google.com for each of these wrappers?

Tonight? Or tomorrow.

[Miguel]

Anytime in the next like week, haha

[j03]

Wow totally forgot about cxWrapper!

[Zachafer]

Can we get this thread stickied please?

[taeryn]

Do we have a breakdown of the features, pros/cons and so on of the different wrappers?
Thank you for the post as well :)

[james087]

mmm what about the php wrapper inside of Zachafer 's Dailies Script zWrapper?


class zWrapper
{

private $cookies, $LastPage, $useProxy;

public $proxyHost, $proxyPort;

public function Request($method, $url, $referer = '', $showHeaders = false)
{
$showHeaders = (bool)$showHeaders;
$file = '';
$host = '';
$data = '';
if (strcasecmp($method, 'GETPOST') == 0) {
if (($x = strpos($url, '~~')) !== false) {
$data = substr($url, $x + 2);
$url = substr($url, 0, $x);
}
} else
if (strcasecmp($method, 'POST') == 0) {
if (($x = strpos($url, '?')) !== false) {
$data = substr($url, $x + 1);
$url = substr($url, 0, $x);
}
}
if (strcasecmp(substr($url, 0, 7), '[Only registered and activated users can see links]') == 0) {
$host = substr($url, 7);
}
if (($x = strpos($host, '/')) !== false) {
$file = substr($host, $x);
$host = substr($host, 0, $x);
} else {
$file = '/';
}
if (isset($referer)) {
$referer = 'Referer: ' . $referer;
} else
if (isset($this->LastPage)) {
$referer = 'Referer: ' . $this->LastPage;
}
$this->LastPage = $url;
$fp = null;
if ($this->useProxy)
$fp = fsockopen($this->proxyHost, $this->proxyPort) or die($errstr . ' (' . $errno .
')');
else
$fp = fsockopen($host, 80) or die($errstr . ' (' . $errno . ')');
if ($fp) {
fwrite($fp, $this->headers($method, $file, $host, $referer, $data));
$r = '';
$s = microtime(true);
while (!feof($fp) && (microtime(true) - $s < 10000))
$r .= fgets($fp, 256);
fclose($fp);
$temp = explode("\r\n\r\n", $r, 2);
$this->process_headers($temp[0]);
if (strpos($temp[0], 'Content-Encoding: gzip') !== false) {
$temp[1] = $this->el_zip_gzDecode($temp[1]);
return $showHeaders ? implode("\r\n\r\n", $temp) : empty($temp[1]) ? implode("\r\n\r\n",
$temp) : $temp[1];
} else {
return $showHeaders ? $r : empty($temp[1]) ? implode("\r\n\r\n", $temp) : $temp[1];
}
}
}

public function clearCookies()
{
$this->cookies = array();
}

public function getCookies()
{
return $this->cookies;
}

public function setProxy($host, $port)
{
$this->proxyHost = $host;
$this->proxyPort = $port;
$this->useProxy = true;
}

public function offProxy()
{
$this->useProxy = false;
}

public function setCookies($cookies)
{
$this->cookies = $cookies;
}

public function __construct()
{
date_default_timezone_set('America/Los_Angeles');
$this->cookies = array();
if (!function_exists('stripos')) {
function stripos($haystack, $needle)
{
return strpos($haystack, stristr($haystack, $needle));
}
}
}

private function el_zip_gzDecode($data)
{
$gzMagic = sprintf("%X%X", ord(substr($data, 1, 1)), ord(substr($data, 0, 1)));
if ($gzMagic != "8B1F") {
[Only registered and activated users can see links]
return $unpacked;
}
$flags = ord(substr($data, 3, 1));
$headerlen = 10;
if ($flags & 4) {
$extralen = unpack('v', substr($data, 10, 2));
$extralen = $extralen[1];
$headerlen += 2 + $extralen;
}
if ($flags & 2) // CRC at end of header

$headerlen += 2;
return gzinflate(substr($data, $headerlen, strlen($data) - $headerlen - 8));
}

private function cookieString()
{
if (count($this->cookies) > 0) {
$string = 'Cookie:';
foreach ($this->cookies as $k => $v)
$string .= " $k=$v;";
return $string . "\r\n";
}
}

private function process_headers($headers)
{
$size = count($lines = explode("\n", $headers));
$x = -1;
while (++$x < $size) {
if (preg_match('/^Set-Cookie: ([^;]+);/', $lines[$x], $s)) {
list($key, $val) = explode("=", $s[1], 2);
$this->cookies[$key] = $val;
}
}
}

private function headers($method, $file, $host, $referer = "", $data = "")
{
$isPost = stripos($method, 'POST') !== false;
[Only registered and activated users can see links]
$file) . " [Only registered and activated users can see links]";
$h .= 'Host: ' . $host . "\r\n";
$h .= "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\n";
$h .= "Accept-Language: en-us,en;q=0.5\r\n";
$h .= "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7\r\n";
$h .= "Accept-Encoding: gzip,deflate\r\n";
if (!empty($referer))
$h .= $referer . "\r\n";
$h .= $this->cookieString();
$h .= "User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13\r\n";
if ($isPost) {
$h .= "Content-Type: application/x-[Only registered and activated users can see links]";
$h .= 'Content-Length: ' . strlen($data) . "\r\n";
}
$h .= "Keep-Alive: 115\r\n";
$h .= "Connection: keep-alive\r\n\r\n";
#$h .= "Connection: close\r\n\r\n";
if ($isPost)
$h .= $data;
return $h;
}
}



Lol check out what I made
[Only registered and activated users can see links]
all my shyt's going to be in php now :P
well unless you guys strongly prefer either vb.net or (preferably)C# so that you can more easily step through my code.

[Zachafer]

I strongly prefer php :)

[james087]

I'm not sure what you're doing there with PHP as a desktop application, but looks nice! Though in my experience running it as a program instead of a web script was somewhat slow, what's the speed like there?
All I made was a hello world app :P the rest of those files are Zachafer's. This is my first experience running php as a desktop app so not sure on benchmarks. Even more unsure now that I've learned I can compile php.

What programs have you run that seemed somewhat slow that were in PHP?

[james087]

Just some stuff I've written, might be better in a newer PHP release though Oh cool I didn't think anyone else here would have any experience with php as a desktop application. Did you use GTK+ / glade to make the gui? Seems GTK is much more popular with python, but I have zero experience with python.. figured I'd learn GTK+ /glade first and then maybe switch to python since there is so much more active development going on with it compared to php.. if php isn't good enough for what I need it to do.

Are you concerned about speed for things like auto buying?

[james087]

PHP is designed for web development Yeah, the vast majority of my background is in web development php, which is why the idea of using PHP as a desktop application appeals to me. My first language was C# / asp.net (also for web), but moved to PHP just because it was the first job I could find.

I'm probably underestimating the importance of speed for these small little applications. I'm sure I'll have tons of fun trying to optimize when that time comes :P

I'll switch to python if it becomes unrealistic to use php. Joe likes python right?

./m Y U NO POST PYTHON WRAPPER?

[j03]

Yeah, the vast majority of my background is in web development php, which is why the idea of using PHP as a desktop application appeals to me. My first language was C# / asp.net (also for web), but moved to PHP just because it was the first job I could find.

I'm probably underestimating the importance of speed for these small little applications. I'm sure I'll have tons of fun trying to optimize when that time comes :P

I'll switch to python if it becomes unrealistic to use php. Joe likes python right?

I like C# a lot right now.

[james087]

I like C# a lot right now.

Yeah? All right I guess I'll go with C# then. Always wanted to get back into it. Really kicked my ass the first time, time for round 2 :p
No sense in learning php/gtk+ since it may or may not work when I can simply learn something I know will work..
wish it was multi-platform out of the gate though =.=

[james087]

What about C# with Mono?
[Only registered and activated users can see links]

I haven't researched it, just a quick google.

[james087]

oh :(
Well there's always craptastic Java..
What do you like coding in Miguel ?

[james087]

Thanks, will do.

[ikakk]

james087

Just posted an account class for Python if you choose to use it. It includes AMF requests :)

[Only registered and activated users can see links](include-AMF-requests)

[james087]

I saw! Very nice man, thank you.

[Zachafer]

Re-uploaded VB6 wrappers:
Glurak's
RipperWrapper

[Zachafer]

Various HTTP Wrappers, if you have any to add to the list, post them here.

VB6
Gluraks ([Only registered and activated users can see links])
RipperWrapper ([Only registered and activated users can see links])
NiteWrapper ([Only registered and activated users can see links])
cxWrapper ([Only registered and activated users can see links])
HaloWrapper ([Only registered and activated users can see links])

VB.NET
cxWrapper ([Only registered and activated users can see links])

C#
cxWrapper ([Only registered and activated users can see links])
mBot HTTP Wrapper ([Only registered and activated users can see links])
SD[Only registered and activated users can see links] ([Only registered and activated users can see links])

Java
Zer0's Wrapper ([Only registered and activated users can see links])

Cleaned up C# mBot wrapper - I renamed it to kWrapper

[j03]

Cleaned up C# mBot wrapper - I renamed it to kWrapper

Did you add any features to the class or notable changes?

Edit: Just noticed the class uses [Only registered and activated users can see links] instead of [Only registered and activated users can see links] bad move but I guess what you did might have been for the practice.

[Zachafer]

Gluraks - [Only registered and activated users can see links]
Ripper - [Only registered and activated users can see links]

Joe please update links

[j03]

Gluraks - [Only registered and activated users can see links]
Ripper - [Only registered and activated users can see links]

Joe please update links

Updated. Can you explain what is updated between the two?

[Zachafer]

Updated. Can you explain what is updated between the two?

Nothing - the links in OP were broken (attachment not found...)