|
Post by Pjot on Jun 30, 2018 20:37:52 GMT 1
Folks, We all know how easy BaCon can fetch a website using plain HTTP: OPEN website$ & ":80" FOR NETWORK AS mynet SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet REPEAT RECEIVE dat$ FROM mynet total$ = total$ & dat$ UNTIL ISFALSE(WAIT(mynet, 500)) CLOSE NETWORK mynet PRINT total$
But how about HTTPS? For fun, I fiddled around with OpenSSL to see how it works. There's a lot more code to it. I am using my system's OpenSSL header files and library. Here it is: ' SSL library PRAGMA INCLUDE <openssl/ssl.h> PRAGMA LDFLAGS -lssl -lcrypto
' Size of buffer where to store data CONST BufferSize = 32768
' Using RAM disk as a string OPTION MEMSTREAM TRUE
' BaCon must not choke on SSL functions OPTION PARSE FALSE
' Request to send to remote webserver (CONST is a macro def) CONST req$ = "GET / HTTP/1.1\r\nHost: " & TOKEN$(website$, 1, ":") & "\r\n\r\n"
' Some SSL related variables DECLARE ctx TYPE SSL_CTX* DECLARE ssl TYPE SSL*
' Which website we need to fetch IF AMOUNT(ARGUMENT$) = 1 THEN website$ = "www.google.com:443" ELSE website$ = TOKEN$(ARGUMENT$, 2) ENDIF
' Initialize SSL SSL_library_init()
' Create SSL context object ctx = SSL_CTX_new(SSLv23_method()) ssl = SSL_new(ctx)
' Make SSL aware of SNI SSL_set_tlsext_host_name(ssl, TOKEN$(website$, 1, ":"))
' Connect to website creating a socket OPEN website$ FOR NETWORK AS mynet
' Perform the SSL handshake using the socket SSL_set_fd(ssl, mynet) IF SSL_connect(ssl) <= 0 THEN EPRINT "SSL connect error" END 1 ENDIF
' Setup buffer for the data coming back mem = MEMORY(BufferSize) OPEN mem FOR MEMORY AS buf$
' Send the GET request to the remote server SSL_write(ssl, req$, LEN(req$)) REPEAT ' Fetch the response into the buffer IF SSL_read(ssl, buf$, BufferSize) < 0 THEN CONTINUE total$ = total$ & buf$ memset((void*)mem, 0, BufferSize) UNTIL ISFALSE(WAIT(mynet, 500))
' Bring down SSL SSL_shutdown(ssl)
' Close handle and free memory CLOSE MEMORY buf$ CLOSE NETWORK mynet FREE mem
' Show result PRINT total$
Hope it works for you, Regards Peter EDIT: set buffer to 0 at each fetch. EDIT 2: enable SNI in SSL negotiation with 'SSL_set_tlsext_host_name()' EDIT 3: getting rid of BIO stuff to simplify code
|
|
|
Post by vovchik on Jun 30, 2018 21:16:52 GMT 1
Dear Peter,
Thanks. Yes, a bit more code, but it works perfectly.
With kind regards, vovchik
|
|
|
Post by Pjot on Jul 1, 2018 19:00:40 GMT 1
Thanks for verifying vovchik! It turns out that, often, many websites are hosted on one and the same server, and the server must provide multiple certificates for those domains hosted. A web client therefore requires the Server Name Indication (SNI) extension to TLS. Without the SNI extension, the OpenSSL layer confusingly generates the error 'SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure' even though SSLv3 is disabled altogether. I have updated the above program by adding a call to 'SSL_set_tlsext_host_name', and also added some minor tweaks, so it should work even better now Regards Peter Update: more improvements to the code by deleting unnecessary stuff.
|
|
|
Post by Pjot on Jul 2, 2018 21:09:51 GMT 1
All, As a reference, now HTTPS using the GnuTLS library. The below code is a simplified version of their demonstration program. Contrary to OpenSSL, it seems that GnuTLS has less mercy with X509 certificate errors. Webservers using self-signed certificates are not accepted, and I had to comment the 'gnutls_session_set_verify_cert()' function, otherwise I could not see the ASCII dump of the website. This seems to be the only way of allowing self-signed certificates. Regards Peter ' GnuTLS library PRAGMA INCLUDE <gnutls/gnutls.h> <gnutls/x509.h> PRAGMA LDFLAGS -lgnutls
' Size of buffer where to store data CONST BufferSize = 32768
' Using RAM disk as a string OPTION MEMSTREAM TRUE
' BaCon must not choke on SSL functions OPTION PARSE FALSE
' Request to send to remote webserver (CONST is a macro def) CONST req$ = "GET / HTTP/1.1\r\nHost: " & TOKEN$(website$, 1, ":") & "\r\n\r\n"
DECLARE session TYPE gnutls_session_t DECLARE xcred TYPE gnutls_certificate_credentials_t
IF gnutls_check_version("3.4.6") = NULL THEN EPRINT "GnuTLS 3.4.6 or later is required for this example!" END 1 ENDIF
' Which website we need to fetch IF AMOUNT(ARGUMENT$) = 1 THEN website$ = "www.google.com:443" ELSE website$ = TOKEN$(ARGUMENT$, 2) ENDIF
' For backwards compatibility with gnutls < 3.3.0 gnutls_global_init()
' X509 stuff gnutls_certificate_allocate_credentials(&xcred)
' Sets the system trusted CAs for Internet PKI gnutls_certificate_set_x509_system_trust(xcred)
' Initialize TLS session gnutls_init(&session, GNUTLS_CLIENT) gnutls_server_name_set(session, GNUTLS_NAME_DNS, TOKEN$(website$, 1, ":"), LEN(TOKEN$(website$, 1, ":")))
' It is recommended to use the default priorities gnutls_set_default_priority(session)
' Put the x509 credentials to the current session gnutls_credentials_set(session, GNUTLS_CRD_CERTIFICATE, xcred)
' Uncomment if certificate errors are not allowed (eg block self signed) '====================================================================== 'gnutls_session_set_verify_cert(session, TOKEN$(website$, 1, ":"), 0) '======================================================================
' Cpnnect to website creating a socket OPEN website$ FOR NETWORK AS mynet
gnutls_transport_set_int(session, mynet) gnutls_handshake_set_timeout(session, GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT)
' Perform the TLS handshake gnutls_handshake(session) PRINT gnutls_session_get_desc(session) FORMAT "=========== Session info: %s ===========\n"
' Setup buffer for the data coming back mem = MEMORY(BufferSize) OPEN mem FOR MEMORY AS buf$
' Send data gnutls_record_send(session, req$, LEN(req$)) REPEAT ' Fetch the response into the buffer, error quits IF gnutls_record_recv(session, buf$, BufferSize) <= 0 THEN END total$ = total$ & buf$ memset((void*)mem, 0, BufferSize) UNTIL ISFALSE(WAIT(mynet, 500))
' Deinit TLS gnutls_bye(session, GNUTLS_SHUT_RDWR) gnutls_deinit(session) gnutls_certificate_free_credentials(xcred) gnutls_global_deinit()
' Close handle and free memory CLOSE MEMORY buf$ CLOSE NETWORK mynet FREE mem
' Show result PRINT total$
|
|
|
Post by vovchik on Jul 2, 2018 23:15:07 GMT 1
Dear Peter,
I am getting compiler errors, and I think it might be a version problem. For one, GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT is not recognized, although I have the headers installed. For another, a number of functions are not found in the lib. I'll try to find out that the problem is...
With kind regards, vovchik
|
|
|
Post by Pjot on Jul 3, 2018 2:23:30 GMT 1
Hi vovchik,
How do the errors look like? If the program is called 'https.bac', can you show the contents of 'https.bac.log'?
BR Peter
|
|
|
Post by vovchik on Jul 3, 2018 8:55:49 GMT 1
Dear Peter,
There first log gives me this:
gnuhttps.bac.c: In function 'main': gnuhttps.bac.c:178:39: error: 'GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT' undeclared (first use in this function) gnutls_handshake_set_timeout(session, GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT); ^ gnuhttps.bac.c:178:39: note: each undeclared identifier is reported only once for each function it appears in make: *** [gnuhttps.bac.o] Error 1
As a kluge, I created the missing data:
DECLARE GNUTLS_DEFAULT_HANDSHAKE_TIMEOUT=-1 TYPE unsigned int
Then I get this, which indicates, I think, missing functions in my version of the the lib:
gnuhttps.bac.o: In function `main': gnuhttps.bac.c:(.text.startup.main+0x30b): undefined reference to `gnutls_certificate_set_x509_system_trust' gnuhttps.bac.c:(.text.startup.main+0x5d7): undefined reference to `gnutls_transport_set_int' gnuhttps.bac.c:(.text.startup.main+0x5ea): undefined reference to `gnutls_handshake_set_timeout' gnuhttps.bac.c:(.text.startup.main+0x602): undefined reference to `gnutls_session_get_desc' collect2: error: ld returned 1 exit status make: *** [gnuhttps] Error 1
With kind regards, vovchik
PS. If I upgrade the lib to 28 (I have 26 and 27 installed), it will remove all sorts of useful stuff in my setup.
|
|
|
Post by bigbass on Jul 3, 2018 13:51:32 GMT 1
Hello Peter I just tested the last code demo you posted and here is a terminal capture of the result compiled and ran with libssl-dev libgcrypt20-dev libgnutls28-dev Hope that gives you some feedback you could use result.tar.gz (16.06 KB) Joe
|
|
|
Post by Pjot on Jul 3, 2018 17:33:54 GMT 1
Hi bigbass, Thanks, I can see that the program works on your system - cool! @ vovchik: indeed, your system is bound to an older version. The function 'gnutls_check_version' should warn for older versions during runtime, however, you run into problems during compile time already. But OpenSSL works Thanks again! BR Peter
|
|
|
Post by Pjot on Jul 3, 2018 20:43:34 GMT 1
Gentlemen, As OpenSSL and GnuTLS are large and have more restrictive licenses than BaCon, I tried to find a small TLS implementation which we can use without restriction. Fortunately, there is a pretty nice implementation in one source file called TLSE which comes with a Public Domain license. It depends on the libtomcrypt which is Public Domain also (and available in Synaptic). So after downloading the TLSE zip file, the below program will include its source code (similar as with the canvas plugins), and then also successfully can perform HTTPS connections. I will try to create an INCLUDE file for BaCon to facilitate simple HTTPS connections. Regards Peter ' Undefine these BaCon functions to avoid conflict with libtomcrypt USEH #undef ROR #undef ROL ENDUSEH
' Include source files PRAGMA INCLUDE tlse-master/tlse.h PRAGMA INCLUDE tlse-master/tlse.c PRAGMA LDFLAGS -ltomcrypt PRAGMA OPTIONS -DLTM_DESC
' Size of buffer where to store data CONST BufferSize = 32768
' Using RAM disk as a string OPTION MEMSTREAM TRUE
' BaCon must not choke on SSL functions OPTION PARSE FALSE
' Request to send to remote webserver (CONST is a macro def) CONST req$ = "GET / HTTP/1.1\r\nHost: " & TOKEN$(website$, 1, ":") & "\r\nConnection: close\r\n\r\n"
DECLARE clientssl TYPE SSL*
IF AMOUNT(ARGUMENT$) = 1 THEN website$ = "www.google.com:443" ELSE website$ = TOKEN$(ARGUMENT$, 2) ENDIF
' Create SSL context object clientssl = SSL_CTX_new(SSLv3_client_method())
' Set sni tls_sni_set(clientssl, TOKEN$(website$, 1, ":"))
' Connect to website creating a socket OPEN website$ FOR NETWORK AS mynet
' Perform the SSL handshake using the socket SSL_set_fd(clientssl, mynet) IF SSL_connect(clientssl) <> 1 THEN EPRINT "Handshake error!" END 1 ENDIF
' Setup buffer for the data coming back mem = MEMORY(BufferSize) OPEN mem FOR MEMORY AS buf$
' Send the GET request to the remote server SSL_write(clientssl, req$, LEN(req$)) WHILE WAIT(mynet, 500) ' Fetch the response into the buffer IF SSL_read(clientssl, buf$, BufferSize) > 0 THEN total$ = total$ & buf$ POKE mem, 0 SIZE BufferSize ELSE BREAK ENDIF WEND
' Bring down TLS SSL_shutdown(clientssl) SSL_CTX_free(clientssl)
' Close handle and free memory CLOSE MEMORY buf$ CLOSE NETWORK mynet FREE mem
' Show result PRINT total$
|
|
|
Post by vovchik on Jul 3, 2018 21:07:29 GMT 1
Dear Peter, I just tested using your instructions and it works like a charm. Thanks! With kind regards, vovchik
|
|
|
Post by rikky on Jul 4, 2018 7:52:26 GMT 1
The OPENSSL version works like a sharm on de RPi All the other versions, I cannot get to work. I have installed nettle and gmp and unbound-anchor and libtomcrypt-develop and probably something, I forgot to take notition of. After a lot of triel and error. eventually got to : Makefile.bacon:6: recipe for target 'https.bac.o' failed Compiler error:
Description: file '/home/pi/Playroom/https/https.bac' line 59: POKE mem, 0 SIZE BufferSize Cause: unknown type name 'ltc_ecc_set_type' Rik. Attachments:https.bac.log (9.07 KB)
|
|
|
Post by Pjot on Jul 4, 2018 10:30:26 GMT 1
Hi Rikky,
Thank for trying! It works with BaCon 3.8 beta but you're not using this version. So you have to replace the following line:
POKE mem, 0 SIZE BufferSize
...with this code:
memset((void*)mem, 0, BufferSize)
After this change, compilation should work.
Please let me know the outcome.
BR Peter
|
|
|
Post by rikky on Jul 4, 2018 11:20:59 GMT 1
|
|
|
Post by vovchik on Jul 4, 2018 12:14:29 GMT 1
Dear Rik,
It works perfectly on my RPI3 (and on Mint), after doing:
sudo apt-get install libtomcrypt-dev
and unpacking tlse-master. Then create the file Peter posted above the tlse-master dir. Then compile.
With kind regards, vovchik
|
|