Preserved Topic: https file retrieval from the command line

Ok, this is off topic, but I know a couple of you guys probably have some thoughts here...

I would like to pull files via https on a scheduled basis. Another constraint is that I don't necessarily own the pulling machines.

So a few approaches suggest themselves, help me see the true path here please.

1) everyone already has iexplore, which can retieve the file and negotiate the SSL. If there's a command-line invocation that lets me use it, I'm home free. But I haven't found that anywhere.

2) everyone already has iexplore, which is (still?) built on wininet.dll as I recall. Can I write a console app in VC that calls wininet's API to do the grab? this seems appealing to me because I could tolerate distributing a small executable. The heavy lifting is still done by dll's lying about the system.

3) I could wrestle with the SSL myself, but that ain't gonna happen

4) I could use another SSL library (if I knew about one that I could distribute).

I poked around on download.com and found some really small proggies that seem to do what I want. But I don't want to use shareware if I can write this myself.

I want the interface to work like fetch on FreeBSD or wget on Linux:
wfetch https://secure.example.com/seekrit/stuff.html mydownload.html
would grab the contents of that URL and stuff it in that file.

C'mon guys, help a brother out.

Heh - you didn't ask ME...

Go to sourceforge and search for keyword "curl" -
Curl is a utility that does just that... you can call it from Perl also.
It also lets you do FTP cmds from command-lines.

Keep in mind, you're gonna have to do the SSL negotiation yourself -
depending on the server you're going to, they may not be able to
handle the cypher you build your utility with - it's kind of involved.
poke around in the curl source - it's the most advanced tool I've
found that does this.




[This message has been edited by kevincar (edited 06-09-2001).]

Here's direct link to cUrl's web site: http://curl.haxx.se/

Actually kevincar, I specifically was targeting the question at you because you had indicated familiarity with Win32. And I figured mr.maX would weigh in as well.

Thanks for the tip, I'll go check it out.

So there's no way that you guys know of to force iexplore to do the stunt I want?

AFAIK, iexplore.exe doesn't accept any parameters. So, you can either play with wininet.dll (as you already mentioned), or you can integrate IE ActiveX component in your application (it doesn't have to be visible) and you can fetch the source code of the page and save it locally...

Well, from VB, you can use the IExplorer COM control -- really kinda spiffy
I have to admit... you can insert a browser into a VB program. Unless you are
actually DISPLAYING the content of the HTTPS page you are getting, I would
not mess with it.

You haven't given a compelling reason yet WHY you need to use IExplorer just to
"get" the HTTPS page(s), other than "everyone has it" -

BTW-
SSL is freely available HERE. It's a good library, free, and clear, and has RSA/ Diffie-Hellman, and all the readily used cyphers - I think it even does TLS.

If you want to truly "roll-your-own" industrial strength SSL utility, I would NOT use
IExplorer - Here's what I did with Java (pretty much what you need, I think):

1. I d/l'ed and compiled SSL (link already given)
2. I write a JNDI "wrapper" around SSL - you can find one readily coded
HERE.
3. Your "utility" does the following:
a. open an SSL socket using SSL socket open...
b. does a post/ or get on the remote site to the URL / URI
c. Opens the socket input stream, and reads until the bytestream is
exhausted.

Here's the "guts" of the Java-version -- taken directly from an international
payment gateway project :-)

/* SSLEAYY ALGORITHM */

   javaSSLSocket s = new javaSSLSocket("ssl.wherever.net",443);

   s.setSoLinger(true, 30);



   DataOutputStream dos = new DataOutputStream(s.getOutputStream());

   dos.writeBytes("POST /URI/receiver.dll?param1=thisval\r\n");   // SSLeay



   dos.writeBytes("Accept: text/*\r\n");

   dos.writeBytes("User-Agent: SSL TestApp\r\n");

   dos.writeBytes("Content-Type: x-Visa-II/x-auth\r\n");

   dos.writeBytes("Content-Length: " + Integer.toString(contLen+2) + "\r\n");

   dos.writeBytes("\r\n");     // TERMINATE HEADERS



    for(i=0; i < contLen+2; i++) {

        int temp = myBytes[i] & 0xff;

       dos.write(myBytes[i] & 0xff);

       }

notice the code makes a POST, sends the HTTP headers, then sends the
"file" post-data - this illutrates pretty much what your utility will have to do.

Notice the little "for-loop" at the end? - THAT's a little magic -
you HAVE to ship bytes - not java chars down the wire... Java forces all
chars to be 16-bit NLS, sign-extended... boy does THAT make a mess at the other
end... BTW: I think VB does the same thing - beware.

Whew- this is turning into a large message - let's pause and reflect;

1. I don't think you want to use Iexplorer, but I may be wrong - I just did what you want to do differently.

2. There's a bunch of stuff out there that should give you the tools to do the job...
Curl, SSLeay, JavaSSL, and maybe some others.

3. You should be able to contain your project as a simple socket application
using the SSLeay (ie, OpenSSL) DLL, and an executable - forget about
IExplorer.

That's pretty much the answer I want. Thanks, KC.

Whoops!

I forgot how to post the reading of the HTTPS response!
If anyone needs that, drop me an email. and I'll post it here.

I've checked out cURL, and it looks like the perfect fit for my needs. I especially appreciate the dual licenses.