Topic: Hidden url finder on remote server with cURL slow timout (Page 1 of 1) $Pages that link to <a href="https://ozoneasylum.com/backlink?for=27781" title="Pages that link to Topic: Hidden url finder on remote server with cURL slow timout (Page 1 of 1)" rel="nofollow" >Topic: Hidden url finder on remote server with cURL slow timout <span class="small">(Page 1 of 1)</span>\$

Sam Bipolar (III) Inmate From: Belgium Insane since: Oct 2002	posted 04-13-2006 16:02 Hi all, I've written a php script to find a hidden url on a remote server. the hidden url is in the range: http://www.dummy.com/000001.htm to http://www.dummy.com/999999.htm I thought the best way was to write a php script which validates every link... This is what I've got so far code: <html> <?php function url_exists($strURL) { $resURL = curl_init(); curl_setopt($resURL, CURLOPT_URL, $strURL); curl_setopt($resURL, CURLOPT_BINARYTRANSFER, 1); curl_setopt($resURL, CURLOPT_HEADERFUNCTION, 'curlHeaderCallback'); curl_setopt($resURL, CURLOPT_FAILONERROR, 1); curl_exec ($resURL); $intReturnCode = curl_getinfo($resURL, CURLINFO_HTTP_CODE); curl_close ($resURL); if ($intReturnCode != 200 && $intReturnCode != 302 && $intReturnCode != 304) { return false; }Else{ return true ; } } for($i = 0; $i < 1000000; $i++) { $j=str_pad($i, 6, "0", STR_PAD_LEFT); $url="http://www.dummy.com/".$j.".htm"; If(url_exists($url)) { Echo"$url"; exit; }Else{ Echo""; } } ?> </html> Guess what, it's slow and doesn't finish because of script timeouts. 1 - Is this the way to go? 2 - Can I make this script faster? 3 - Will it generate too much traffic on my webserver and will I get into trouble with my provider? Thanks in advance, Sam
Tyberius Prime Maniac (V) Mad Scientist with Finglongers From: Germany Insane since: Sep 2001	posted 04-13-2006 17:17 This might work, but hitting 10^6 pages ain't gonna be ever fast. Now, I don't even want to know why there's this hidden url around... nayhow, I'd advise on doing that only against your own server - you're bound to generate a lot of traffic. All right, I'd see if the server suppered HTTP HEAD instead of HTTP GET (check with a known url, read the rfc if yo must), I'd probably dump curl in this and do the talking directly on a tcp socket on port 80, but that' just an off chance. Otherwise, you probably should be running this in serveral threads (or server php page calls), just to lower the latency. See... the actuall sending of data is neglible in what you are doing - but it simply takes half a second to establish a new tcp connection, and ask the server wether there's a page there. But you could probably do 50 of them at the same time. Oh, and I'd so put a known good url in there just to check that your code would actually print on the found url . so long, ->Tyberius Prime

Topic: Hidden url finder on remote server with cURL slow timout (Page 1 of 1) $Pages that link to <a href="https://ozoneasylum.com/backlink?for=27781" title="Pages that link to Topic: Hidden url finder on remote server with cURL slow timout (Page 1 of 1)" rel="nofollow" >Topic: Hidden url finder on remote server with cURL slow timout <span class="small">(Page 1 of 1)</span>\$

Sam
Bipolar (III) Inmate

From: Belgium
Insane since: Oct 2002

posted 04-13-2006 16:02

Hi all,

I've written a php script to find a hidden url on a remote server.
the hidden url is in the range:

http://www.dummy.com/000001.htm to http://www.dummy.com/999999.htm

I thought the best way was to write a php script which validates every link...

This is what I've got so far

code:

<html>
<?php
   function url_exists($strURL) { 
    $resURL = curl_init(); 
    curl_setopt($resURL, CURLOPT_URL, $strURL); 
    curl_setopt($resURL, CURLOPT_BINARYTRANSFER, 1); 
    curl_setopt($resURL, CURLOPT_HEADERFUNCTION, 'curlHeaderCallback'); 
    curl_setopt($resURL, CURLOPT_FAILONERROR, 1); 

    curl_exec ($resURL); 

    $intReturnCode = curl_getinfo($resURL, CURLINFO_HTTP_CODE); 
    curl_close ($resURL); 

    if ($intReturnCode != 200 && $intReturnCode != 302 && $intReturnCode != 304) { 
       return false; 
    }Else{ 
        return true ; 
    } 
   } 

for($i = 0; $i < 1000000; $i++)
{
   $j=str_pad($i, 6, "0", STR_PAD_LEFT);
   $url="http://www.dummy.com/".$j.".htm";
   If(url_exists($url)) {
      Echo"$url";
      exit;
   }Else{ 
    Echo""; 
   } 
}
?>
</html>

Guess what, it's slow and doesn't finish because of script timeouts.
1 - Is this the way to go?
2 - Can I make this script faster?
3 - Will it generate too much traffic on my webserver and will I get into trouble with my provider?

Thanks in advance,
Sam

Tyberius Prime
Maniac (V) Mad Scientist with Finglongers

From: Germany
Insane since: Sep 2001

posted 04-13-2006 17:17

This might work, but hitting 10^6 pages ain't gonna be ever fast.

Now, I don't even want to know why there's this hidden url around... nayhow, I'd advise on doing that only against your own server - you're bound to generate a lot of traffic.

All right, I'd see if the server suppered HTTP HEAD instead of HTTP GET (check with a known url, read the rfc if yo must),
I'd probably dump curl in this and do the talking directly on a tcp socket on port 80, but that' just an off chance.

Otherwise, you probably should be running this in serveral threads (or server php page calls), just to lower the latency.
See... the actuall sending of data is neglible in what you are doing - but it simply takes half a second to establish a new tcp connection, and ask the server wether there's a page there. But you could probably do 50 of them at the same time.

Oh, and I'd so put a known good url in there just to check that your code would actually print on the found url .

so long,

->Tyberius Prime

Post Reply

Your User Name:
Your Password:
Login Options:	Remember Me On This Computer

Your Text: Insert Slimies » Insert UBB Code » Close Last Tag \| All Tags UBB Help	Loading...
Options:	Show Signature Enable Slimies Enable Linkwords

Maximum Security
OZONE
DHTML/Javascript
Server-Side Scripting - Oh my!
CSS - DOM - XHTML - XML - XSL - XSLT
Stupid Basic HTML
Visual Therapy
Photoshop
Photoshop Pong, Anyone?
*WARNING* BIG SIG APPROACHING
Photography
3D Modelling & Rendering
Multimedia/Animation
Print Graphics
Holding Pens
Philosophy and other Silliness
Outpatient Counseling
Site reviews!
Mad Scientists' Laboratory
Getting to know the Grail

Maximum Security

OZONE

DHTML/Javascript

Server-Side Scripting - Oh my!

CSS - DOM - XHTML - XML - XSL - XSLT

Stupid Basic HTML

Visual Therapy

Photoshop

Photoshop Pong, Anyone?

***WARNING*** BIG SIG APPROACHING

Photography

3D Modelling & Rendering