Fetching HTTPS

[Pjot]

All,

As the internet connections more and more tend to use SSL/TLS, I thought of a way to add the TLS encapsulation to BaCon.

After looking for very tiny TLS implementations (like this one in pure BASH script), it turns out that there is not really a 'small' implementation for C, which is all-round and yet small enough to embed in BaCon. Also, the encryption and certificate stuff is pretty complicated, and a lot of work to maintain.

Instead, I have added a BASIC specification similar as to the REGEX library functionality. In the latest BaCon version, it is possible to enable TLS in the network code (client only) and use a TCP connection as usual. This will retrieve the website in a plain text fashion while the decryption is performed by an external library.

By default, BaCon will use the well-known OpenSSL library, which is a famous and well-established implementation. But it is possible to define another library as well, assuming that it brings an OpenSSL compatible API. So next to OpenSSL, the new TLS feature was tested with GnuTLS and WolfSSL as well. Next to that, LibreSSL and BoringSSL should work also (to be tested).

Obviously, the header files should be available on your platform (Ubuntu/Mint should install the 'libssl-dev' package for OpenSSL).

Note that BaCon will not verify the validity of the certificate. The intention is to enable access to internet services which use SSL/TLS encapsulation (like HTTP, SMTP, POP3, IMAP, etc). If you need elaborate certificate verification, please refer to the SSL wrapper which uses TLSE.

Example code below. As always, feel free to mention any feedback in this forum.

BR
Peter

' Enable SSL/TLS in the below code, using OpenSSL by default
OPTION TLS TRUE

' Uncomment one of the following lines for your favorite TLS library

' This is the default
'PRAGMA TLS openssl INCLUDE <openssl/ssl.h> LDFLAGS -lssl -lcrypto

' Example with GnuTLS
'PRAGMA TLS gnutls INCLUDE <gnutls/openssl.h> LDFLAGS -lgnutls-openssl

' Example with WolfSSL
'PRAGMA TLS wolfssl INCLUDE <wolfssl/openssl/ssl.h> LDFLAGS -lwolfssl

website$ = "www.basic-converter.org"

OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet

WHILE WAIT(mynet, 500)
   RECEIVE dat$ FROM mynet
   total$ = total$ & dat$
WEND

CLOSE NETWORK mynet

PRINT total$


[vovchik]

Dear Peter,

Worked out of the box for me. I still have to test with the various implementations. Thanks.

With kind regards,
vovchik

[rikky]

Jah, it works.

I do get a '301 Moved Permanently' at some sites,
but I suppose that that means that it works.

 Rik.

[Pjot]

Thanks everybody,

While testing, I found that there still is an issue with Server Name Indication. I will work on this upcoming days.

Next to this issue, another issue popped up. A simple program like the below did not work, but it is already fixed in the latest beta.

See how few lines of code now are needed to implement SMTP using TLS on port 465:

OPTION TLS TRUE

CONST CRNL$ = CR$ & NL$

FUNCTION Interaction$(connection, command$)

   LOCAL data$, response$

   SEND command$ & CRNL$ TO connection
   WHILE WAIT(connection, 2000)
       RECEIVE data$ FROM connection
       response$ = response$ & data$
       IF REGEX(response$, CRNL$ & "$") THEN BREAK
   WEND

   RETURN response$

END FUNCTION

server$ = "smtp.gmail.com"

OPEN server$ & ":465" FOR NETWORK AS net
RECEIVE info$ FROM net
PRINT "CONNECT: ", info$

PRINT "HELO: ", Interaction$(net, "HELO basic-converter.org")
PRINT "AUTH: ", Interaction$(net, "AUTH LOGIN")
PRINT "Account: ", Interaction$(net, B64ENC$("user"))
PRINT "Password: ", Interaction$(net, B64ENC$("passwd"))
PRINT "Mail from: ", Interaction$(net, "mail from:<[e-mail]>")
PRINT "Rcpt to: ", Interaction$(net, "rcpt to:<[e-mail]>")
PRINT "DATA: ", Interaction$(net, "DATA")
PRINT Interaction$(net, "From: Peter <[e-mail]>" & CRNL$ & "To: BaCon <[e-mail]>" & CRNL$ & "Subject: test" & \
   CRNL$ & CRNL$ & "Hello world 2" & CRNL$ & "." & CRNL$)
PRINT "QUIT: ", Interaction$(net, "QUIT")
CLOSE NETWORK net
PRINT "Connection closed."


Regards
Peter

[Pjot]

All,

The Server Name Indication extension for TLS has been added as well. Now BaCon also can reach websites which actually share the same IP address (virtual hosts).

This finalizes the TLS implementation.

As a remark, note that a plain HTTP GET may retrieve some seemingly random numbers in between the fetched data. So a complete HTML payload may occur in BaCon with random ASCII numbers in between, and a terminating '0' in ASCII after the </html> tag.

But this is perfectly normal. Some webservers send their data in separate chunks. Each chunk is prefixed with an indication on the amount of bytes, and this is a number in ASCII, surrounded by a Carriage Return / Newline pair. Please see this RFC.

Fortunately, it can easily be removed by EXTRACT$. Example:

OPTION TLS TRUE

website$ = "www.google.com"

OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet
WHILE WAIT(mynet, 1000)
   RECEIVE dat$ FROM mynet
   total$ = total$ & dat$
   IF REGEX(dat$, "\r\n\r\n$") THEN BREAK           : ' Quit receiving data when end indicator was reached
WEND

CLOSE NETWORK mynet

PRINT EXTRACT$(total$, "\r\n[0-9a-fA-F]+\r\n", TRUE) : ' Remove chunk indicators from HTML data


Regards
Peter

[rikky]

Either I did something wrong the last time, or this new improvement also solved my 
'301 Moved Permanently' messages.

Whatever the reason,
All sites that I try get downloaded easily now.

 Thanks,
 Rik.

[Pjot]

Thanks rik,

More sample code: we can look up our MAC address using an online API (uses HTTPS) at api.macvendors.com.

OPTION TLS TRUE

website$ = "api.macvendors.com"

mac$ = "b0:52:16:d0:3c:fb"

OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET /" & mac$ & " HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet

RECEIVE info$ FROM mynet

CLOSE NETWORK mynet

PRINT TOKEN$(info$, 2, "\r\n\r\n")


BR
Peter

[Pjot]

All,

I have updated the documentation explaining all new TLS features.

BR
Peter

[bigbass]

Hello Peter

I used your code demo in this thread
posted on
Jan 13, 2019 at 5:14am
basic-converter.proboards.com/post/11809/thread

I am using bacon 4.3.1

I was having problems with my browser code
using webkit and TLS and only with the fossil site
I get a
Unacceptable TLS certificate
and thought to track down why
by testing with bacon code only non webkit

I just changed the website url below


OPTION TLS TRUE

website$ = "www.chiselapp.com"

OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet
WHILE WAIT(mynet, 1000)
   RECEIVE dat$ FROM mynet
   total$ = total$ & dat$
   IF REGEX(dat$, "\r\n\r\n$") THEN BREAK           : ' Quit receiving data when end indicator was reached
WEND

CLOSE NETWORK mynet

PRINT EXTRACT$(total$, "\r\n[0-9a-fA-F]+\r\n", TRUE) : ' Remove chunk indicators from HTML data





my terminal output is

HTTP/1.1 301 Moved Permanently
Date: Fri, 05 Feb 2021 02:35:59 GMT
Server: Apache/2.4.46 (Unix) OpenSSL/1.0.2u
Location: chiselapp.com/
Content-Length: 230
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://chiselapp.com/">here</a>.</p>
</body></html


is that the expected output?

if yes, then I still need to dig more into the problem with webkit

any suggestions would be useful
Joe

P.S
my end goal is just to get
chiselapp.com/user/bacon/repository/bacon/timeline/
to load again within webkit (it used to work I don't know what had changed)

everything else works except the fossil site

[alexfish]

Hi Joe

can try extract the moved to address

OPTION TLS TRUE
'<p>The document has moved <a href="https://chiselapp.com/">here</a>.</p>
website$ = "chiselapp.com"

OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet
WHILE WAIT(mynet, 1000)
   RECEIVE dat$ FROM mynet
   total$ = total$ & dat$
   IF REGEX(dat$, "\r\n\r\n$") THEN BREAK           : ' Quit receiving data when end indicator was reached
WEND

CLOSE NETWORK mynet

PRINT EXTRACT$(total$, "\r\n[0-9a-fA-F]+\r\n", TRUE) : ' Remove chunk indicators from HTML data

BR
Alex

[bigbass]

Thanks Alex

if I remove the prefix www.
it will connect so that

leaves me with the TLS problem
I will remove the fossil site from
my browser list in webkit until
get that to work ( I suspect it is a security issue
with web kit that I have to sort out still it used to work...)

thanks Alex for the feedback
Joe

I added a safety for copy and paste website addresses for the lazy


website$ = REPLACE$(website$,"www." ,"")
website$ = REPLACE$(website$,"https://" ,"")

OPTION TLS TRUE
'<p>The document has moved <a href="https://chiselapp.com/">here</a>.</p>
website$ = "https://chiselapp.com"
'website$ = "www.chiselapp.com"

website$ = REPLACE$(website$,"www." ,"")
website$ = REPLACE$(website$,"https://" ,"")

then .....
OPEN website$ & ":443" FOR NETWORK AS mynet

[alexfish]

Hi Joe & All

Here have done a quicky follow redirection of Site Moved ,
May need more work as in the loop to find the address

also added bit if only want the header

BR
Alex

OPTION TLS TRUE
'<p>The document has moved <a href="https://chiselapp.com/">here</a>.</p>

FUNCTION Header$(STRING website$) TYPE STRING
LOCAL dat$, total$ TYPE STRING
OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "HEAD / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet
WHILE WAIT(mynet, 1000)
RECEIVE dat$ FROM mynet
total$ = total$ & dat$
WEND

CLOSE NETWORK mynet


RETURN total$

END FUNCTION

' ==============================================

FUNCTION Get$(STRING website$) TYPE STRING
LOCAL dat$, total$ TYPE STRING
OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\n\r\n" TO mynet
WHILE WAIT(mynet, 1000)
RECEIVE dat$ FROM mynet
total$ = total$ & dat$
WEND
CLOSE NETWORK mynet

RETURN total$

END FUNCTION

LOCAL Result$ TYPE STRING
LOCAL site$ =  "www.chiselapp.com"
LOCAL dimension TYPE int
LOCAL fnd$, addr$ TYPE STRING

Result$ = Get$(site$)

IF INSTR(Result$, "301 Moved" ) THEN

IF ( INSTR(Result$,"<p") AND INSTR(Result$,"href") )  THEN
SPLIT Result$  BY NL$ TO array$ SIZE dimension

FOR i = 1 TO dimension-1
fnd$ =  array$[i]
IF (  INSTR(fnd$,"<p") AND INSTR(fnd$,"href") ) THEN

'PRINT fnd$
'<p>The document has moved <a href="https://chiselapp.com/">here</a>.</p>
fnd$ = REPLACE$(fnd$,CHR$(34),"'")
site$ = MID$(fnd$,INSTR(fnd$,"//")+2)
site$= MID$(site$,1 ,INSTR(site$,"'")-1)
site$= REPLACE$(site$,"/","")
PRINT  "Found new"
PRINT site$
Result$ = Get$(site$)
PRINT "Results"
PRINT Result$
END IF
NEXT

ELSE
PRINT " network error"


END IF

ELSE
PRINT "SUCCESS"
PRINT Result$
END IF

[alexfish]

Hi All

Why Look at a Header.

Well , in the above there is a Redirect, but if one looks at the header of chisel app

this is the Result of looking up the header


result
HTTP/1.1 301 Moved Permanently
Date: Fri, 05 Feb 2021 20:06:32 GMT
Server: Apache/2.4.46 (Unix) OpenSSL/1.0.2u
Location: https://chiselapp.com/
Content-Type: text/html; charset=iso-8859-1

Location: chiselapp.com/

Sometimes it is Worth looking up and checking

BR
Alex

[Pjot]

Feb 5, 2021 3:51:27 GMT 1 bigbass said:

my terminal output is

HTTP/1.1 301 Moved Permanently
Date: Fri, 05 Feb 2021 02:35:59 GMT
Server: Apache/2.4.46 (Unix) OpenSSL/1.0.2u
Location: chiselapp.com/
Content-Length: 230
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://chiselapp.com/">here</a>.</p>
</body></html

is that the expected output?

Well, the 301 means that the URL is not correct. So, instead of connecting to "www.chiselapp.com" you're supposed to connect to simply "chiselapp.com" (without the www-prefix).

The amount of returned data is quite large, and I had to change my program a bit to make it work. You may run into other issues with other websites. Programming a web client is not easy

The below code works for me:

OPTION TLS TRUE

website$ = "chiselapp.com"

OPEN website$ & ":443" FOR NETWORK AS mynet

SEND "GET / HTTP/1.1\r\nHost: " & website$ & "\r\nUser-Agent: Mozilla/5.0\r\nConnection: keep-alive\r\n\r\n" TO mynet
WHILE WAIT(mynet, 1000)
   RECEIVE data$ FROM mynet
   total$ = total$ & data$
   IF RIGHT$(data$, 4) = "\r\n\r\n" THEN BREAK
WEND

CLOSE NETWORK mynet

PRINT EXTRACT$(total$, "\r\n[0-9a-fA-F]+\r\n", TRUE)


HTH,
Peter

[bigbass]

Hello Peter And Alex


thanks for your demo code Alex
Thanks Peter for your improved working demo for chiselapp.com

this is very useful for keeping the the dependencies light
(compared to webkit where nothing is easy to port )


I will be able to spend some time tomorrow looking at the webkit TLS problem
which is not a bacon problem


Joe