Simple Web Search Engine

[alexfish]

Hi All

A simple search engine based on CURL & DuckDuckGo

I have not work out some of the codes in the url links. yet most are readable
will update as on going if can work out some of the codes..


Have Fun

Alex


' Simple Terminal Search Engine > requires Curl
LOCAL x
LOCAL Search$= "curl https://duckduckgo.com/html/?q="


INPUT "Search DuckDuckGo ......> " ,Inp$
' Must hav a result ------------------------------
IF LEN(Inp$) > 0 THEN

ELSE
Inp$ = "News"

END IF

PRINT "Searching for:: " & Inp$





SPLIT Inp$ BY " " TO Bits$ SIZE x


FOR n = 0 TO x-1

IF LEN(Bits$[n])>0 THEN
Bits$[n]= CHOP$ (Bits$[n])

IF n < x-1 THEN
Search$=Search$ & Bits$[n] & "%"
ELSE
Search$=Search$ & Bits$[n]
ENDIF

ENDIF

NEXT


HTML$ = EXEC$(Search$)
IF NOT(INSTR(HTML$,"<a")) THEN
PRINT " Can not Resolve Host : DuckDuckGo"
PRINT " Check Internet Connection"
END
END IF
'Clean Up    and save to file

SPLIT  HTML$ BY "\n" TO Res$ SIZE x

FOR n = 0 TO x-1

Res$[n]= CHOP$ (Res$[n])
IF LEN(Res$[n])>0 THEN
tem$ = LEFT$(Res$[n],1)
IF tem$ == "<" THEN
Results$ = CHOP$(Results$ & Res$[n]) & "\n"
END IF
END IF

NEXT




SAVE Results$ TO CURDIR$ & "/index.html"
IF FILEEXISTS(CURDIR$&"/index.html") THEN

PRINT "Results saved in --> " & CURDIR$&"/index.html"
Result$ = ""

OPEN CURDIR$&"/index.html" FOR READING AS myfile
WHILE NOT(ENDFILE(myfile)) DO
READLN txt$ FROM myfile
txt$ = CHOP$(txt$)
IF NOT(ENDFILE(myfile)) THEN
IF LEFT$(txt$,2) == "<a"  THEN

txt$ = REPLACE$(txt$,"'","'")
txt$ = REPLACE$(txt$,"</b>","")
txt$ = REPLACE$(txt$,"<b>","")
txt$ = REPLACE$(txt$,"</a>","")
txt$ = REPLACE$(txt$,"%3A",":")
txt$ = REPLACE$(txt$,"%2F","/")
txt$ = REPLACE$(txt$,"%2D","-")
IF INSTR(txt$,"nofollow") THEN
HTMLTAG$ = LEFT$(txt$,INSTR(txt$,">"))
END IF

IF (INSTR(txt$,"nofollow" )) THEN
COLOR FG TO YELLOW
ELSE
COLOR RESET
END IF

PRINT MID$(txt$,INSTR(txt$,">")+1)
IF INSTR(txt$,"result__url") THEN
COLOR FG TO RED
HTMLTAG$ =MID$(HTMLTAG$,INSTR(HTMLTAG$,"http"))
HTMLTAG$= LEFT$(HTMLTAG$,LEN(HTMLTAG$)-2)
PRINT HTMLTAG$
COLOR FG TO BLUE
PRINT
PRINT "**************************************************"
PRINT
COLOR RESET

END IF
Result$ = Result$ & txt$



END IF
ENDIF
WEND
CLOSE FILE myfile

END IF

[alexfish]

Hi all

further enhancements & filters

this on can link to a default Browser if xdg-open works

on start 'enter:: y ' to enable the xdg-open

after a search

option to continue y/n


If call to browser
enter 0 to pass or 1 TO

If wondering why INPUT STRING , I be getting a Seg fault , yet testing of the doc code
INPUT "What is your age? ", age
PRINT "You probably were born in ", YEAR(NOW) - age

Works . so need to debug that one

BR
Alex

Found a BUG looks like a callback to search again = something odd with the links count ?
have updated code to reset the count to 0 ,but still the same
After a call to the browser m can exit for now

' Simple Terminal Search engine > requires Curl
' this also requires default command "xdg-open"

PROTO system
LOCAL links TYPE int

LOCAL x=0
LOCAL browser=0
LOCAL tags=0
LOCAL Search$= "curl https://duckduckgo.com/html/?q="
LOCAL ALLHTMLTAGS$
INPUT "See result in default browser y/n> " ,Inp$
Inp$=UCASE$(Inp$)
IF INSTR(Inp$,"Y")==1 THEN
browser=1
END IF

LABEL start
tags=0
ALLHTMLTAGS$=""
INPUT "Search DuckDuckGo ......> " ,Inp$
CLEAR
IF Inp$ == "quit" THEN

INPUT " Quit y/n  > " ,test$
test$= CHOP$(test$)

IF (NOT(LEN(test$))) THEN
END
ELSE

test$ = UCASE$(test$)
IF INSTR(test$,"Y") THEN

END
ENDIF
END IF
END IF
' Must hav a result ------------------------------
IF LEN(Inp$) > 0 THEN

ELSE
Inp$ = "News"

END IF

PRINT "Searching for:: " & Inp$

SPLIT Inp$ BY " " TO Bits$ SIZE x


FOR n = 0 TO x-1

IF LEN(Bits$[n])>0 THEN
Bits$[n]= CHOP$ (Bits$[n])

IF n < x-1 THEN
Search$=Search$ & Bits$[n] & "%"
ELSE
Search$=Search$ & Bits$[n]
ENDIF

ENDIF

NEXT


HTML$ = EXEC$(Search$)
IF NOT(INSTR(HTML$,"<a")) THEN
PRINT " Can not Resolve Host : DuckDuckGo"
PRINT " Check Internet Connection"
END
END IF
'Clean Up    and save to file

SPLIT  HTML$ BY "\n" TO Res$ SIZE x

FOR n = 0 TO x-1

Res$[n]= CHOP$ (Res$[n])
IF LEN(Res$[n])>0 THEN
tem$ = LEFT$(Res$[n],1)
IF tem$ == "<" THEN
Results$ = CHOP$(Results$ & Res$[n]) & "\n"
END IF
END IF

NEXT




SAVE Results$ TO CURDIR$ & "/index.html"
IF FILEEXISTS(CURDIR$&"/index.html") THEN

PRINT "Results saved in --> " & CURDIR$&"/index.html"
Result$ = ""

OPEN CURDIR$&"/index.html" FOR READING AS myfile
WHILE NOT(ENDFILE(myfile)) DO
READLN txt$ FROM myfile
txt$ = CHOP$(txt$)
IF NOT(ENDFILE(myfile)) THEN
IF LEFT$(txt$,2) == "<a"  THEN
'http://baconbashtexas.com/%3Fproduct%3D45
'baconbashtexas.com/?product=45

txt$ = REPLACE$(txt$,"'","'")
txt$ = REPLACE$(txt$,"</b>","")
txt$ = REPLACE$(txt$,"<b>","")
txt$ = REPLACE$(txt$,"</a>","")
txt$ = REPLACE$(txt$,"%3A",":")
txt$ = REPLACE$(txt$,"%2F","/")
txt$ = REPLACE$(txt$,"%2D","-")
txt$ = REPLACE$(txt$,"%3D","=")
txt$ = REPLACE$(txt$,"%3F","?")
txt$ = REPLACE$(txt$,"%23","#")
IF INSTR(txt$,"nofollow") THEN
HTMLTAG$ = LEFT$(txt$,INSTR(txt$,">"))
END IF

IF (INSTR(txt$,"nofollow" )) THEN
COLOR FG TO YELLOW
INCR tags
IF ODD(tags)  THEN
PRINT STR$(tags) & " ..."
ELSE
COLOR FG TO RED
PRINT
PRINT STR$(tags) & " _"
END IF
ELSE
COLOR RESET
END IF

PRINT MID$(txt$,INSTR(txt$,">")+1)
IF INSTR(txt$,"result__url")  THEN
COLOR FG TO RED
HTMLTAG$ =MID$(HTMLTAG$,INSTR(HTMLTAG$,"http"))
HTMLTAG$= LEFT$(HTMLTAG$,LEN(HTMLTAG$)-2)
PRINT HTMLTAG$
ALLHTMLTAGS$=ALLHTMLTAGS$ &  HTMLTAG$ & "\n"
COLOR FG TO BLUE
PRINT
PRINT "**************************************************"
PRINT
COLOR RESET

END IF
Result$ = Result$ & txt$

END IF
ENDIF
WEND
CLOSE FILE myfile
END IF
tags = (tags -1)/2
IF browser THEN
PRINT " Select 1 TO " ,tags , "  or  " , 0 , "  continue"
INPUT "Select Link number Open Browser  > " ,links$
links=VAL(links$)
PRINT " Value ", links
IF links== 0 THEN

GOTO contin
END IF
IF ODD(links) THEN
links = links +1
END IF
PRINT links
SPLIT ALLHTMLTAGS$ BY "\n" TO HTP$ SIZE t
PRINT " total > " , t-1
IF  (links/2<=t-1) THEN
PRINT " SELECTED > " ,links/2
PRINT HTP$[(links/2)-1]
   xdg$ = "xdg-open " &  HTP$[(links/2)-1] & " &"
system(xdg$)
ELSE

GOTO contin
'PRINT HTP$[links]
END IF
END IF

LABEL contin
INPUT "Continue y/n  > " ,test$
test$= CHOP$(test$)

IF (NOT(LEN(test$))) THEN
END
ELSE

test$ = UCASE$(test$)
IF INSTR(test$,"Y") THEN

GOTO start
ENDIF
END IF


[vovchik]

Dear Alex,

Thanks. What about using "curl.bac"? It should be possible, too. I have been playing with it and can grab the bacon site but am getting errors (due to my ignorance) when I try https.

With kind regards,
vovchik

[alexfish]

Hi Vovchik

Yes , that is the target

If can help in that area then

since I have Just found what is happening with this demo.

Looks like the link to DuckDuckGo gets cached , in other words
adds more and more to the fist search list . not sure what the actual limit is
for now have corrected the counter.

will add pages to this demo later. also need to find a reset if one changes the search

this it where Demo is now

looking ahead if can get this each link through rsvg then pass the image to a widget with callback .. and so on
Added or pass it throw html2text & cross reference the bits

BR
Alex


' Simple Terminal Search engine > requires Curl
' this also requires default command "xdg-open"
LOCAL colselect
PROTO system
LOCAL links TYPE int
LOCAL lapptr =0
LOCAL x=0
LOCAL browser=0
LOCAL tags=0
LOCAL Search$= "curl https://duckduckgo.com/html/?q="
LOCAL ALLHTMLTAGS$
INPUT "See result in default browser y/n> " ,Inp$
Inp$=UCASE$(Inp$)
IF INSTR(Inp$,"Y")==1 THEN
browser=1
END IF

LABEL start
COLOR RESET
LOCAL Search$= "curl https://duckduckgo.com/html/?q="
tags=0
links=0
ALLHTMLTAGS$=""
CLEAR
INPUT "Search DuckDuckGo ......> " ,Inp$

IF Inp$ == "quit" THEN

INPUT " Quit y/n  > " ,test$
test$= CHOP$(test$)

IF (NOT(LEN(test$))) THEN
END
ELSE

test$ = UCASE$(test$)
IF INSTR(test$,"Y") THEN

END
ENDIF
END IF
END IF
' Must hav a result ------------------------------
IF LEN(Inp$) > 0 THEN

ELSE
Inp$ = "News"

END IF

PRINT "Searching for:: " & Inp$

SPLIT Inp$ BY " " TO Bits$ SIZE x


FOR n = 0 TO x-1

IF LEN(Bits$[n])>0 THEN
Bits$[n]= CHOP$ (Bits$[n])

IF n < x-1 THEN
Search$=Search$ & Bits$[n] & "%"
ELSE
Search$=Search$ & Bits$[n]
ENDIF

ENDIF

NEXT


HTML$ = EXEC$(Search$)
IF NOT(INSTR(HTML$,"<a")) THEN
PRINT " Can not Resolve Host : DuckDuckGo"
PRINT " Check Internet Connection"
END
END IF
'Clean Up    and save to file

SPLIT  HTML$ BY "\n" TO Res$ SIZE x

FOR n = 0 TO x-1

Res$[n]= CHOP$ (Res$[n])
IF LEN(Res$[n])>0 THEN
tem$ = LEFT$(Res$[n],1)
IF tem$ == "<" THEN
Results$ = CHOP$(Results$ & Res$[n]) & "\n"
END IF
END IF

NEXT




SAVE Results$ TO CURDIR$ & "/index.html"
IF FILEEXISTS(CURDIR$&"/index.html") THEN

PRINT "Results saved in --> " & CURDIR$&"/index.html"
Result$ = ""

OPEN CURDIR$&"/index.html" FOR READING AS myfile
WHILE NOT(ENDFILE(myfile)) DO
READLN txt$ FROM myfile
txt$ = CHOP$(txt$)
IF NOT(ENDFILE(myfile)) THEN
IF LEFT$(txt$,2) == "<a"  THEN
IF (INSTR (txt$,"Feedback"))  AND (lapptr>0) THEN
DECR tags
PRINT " LOOOP +++++++++++++++++++++++++++++++++++++++++++"
END IF
'http://baconbashtexas.com/%3Fproduct%3D45
'baconbashtexas.com/?product=45

txt$ = REPLACE$(txt$,"'","'")
txt$ = REPLACE$(txt$,"</b>","")
txt$ = REPLACE$(txt$,"<b>","")
txt$ = REPLACE$(txt$,"</a>","")
txt$ = REPLACE$(txt$,"%3A",":")
txt$ = REPLACE$(txt$,"%2F","/")
txt$ = REPLACE$(txt$,"%2D","-")
txt$ = REPLACE$(txt$,"%3D","=")
txt$ = REPLACE$(txt$,"%3F","?")
txt$ = REPLACE$(txt$,"%23","#")
IF INSTR(txt$,"nofollow") THEN
HTMLTAG$ = LEFT$(txt$,INSTR(txt$,">"))
END IF

IF (INSTR(txt$,"nofollow" )) THEN

INCR tags
IF ODD(tags)  THEN
COLOR FG TO YELLOW
PRINT STR$(tags) & " ..."
COLOR RESET
ELSE
COLOR FG TO RED
PRINT
PRINT STR$(tags) & " _"
COLOR RESET
END IF
ELSE
COLOR RESET
END IF

PRINT MID$(txt$,INSTR(txt$,">")+1)
IF INSTR(txt$,"result__url")  THEN
COLOR FG TO RED
HTMLTAG$ =MID$(HTMLTAG$,INSTR(HTMLTAG$,"http"))
HTMLTAG$= LEFT$(HTMLTAG$,LEN(HTMLTAG$)-2)
PRINT HTMLTAG$
ALLHTMLTAGS$=ALLHTMLTAGS$ &  HTMLTAG$ & "\n"
COLOR FG TO BLUE
PRINT
PRINT "**************************************************"
PRINT
COLOR RESET

END IF
Result$ = Result$ & txt$

END IF
ENDIF
WEND
CLOSE FILE myfile
END IF
tags = (tags -1)/2
IF browser THEN
PRINT " Select 1 TO " ,tags , "  or  " , 0 , "  continue"
INPUT "Select Link number Open Browser  > " ,links$
links=VAL(links$)
PRINT " Value ", links
IF links== 0 THEN

GOTO contin
END IF
IF ODD(links) THEN
links = links +1
END IF
PRINT links
SPLIT ALLHTMLTAGS$ BY "\n" TO HTP$ SIZE t
PRINT " total > " , t-1
IF  (links/2<=t-1) THEN
PRINT " SELECTED > " ,links/2
PRINT HTP$[(links/2)-1]
   'xdg$ = "xdg-open " &  HTP$[(links/2)-1] & " &"
'system(xdg$)
'xdg$=""
ELSE

GOTO contin
'PRINT HTP$[links]
END IF
END IF

LABEL contin
INPUT "Continue y/n  > " ,test$
test$= CHOP$(test$)

IF (NOT(LEN(test$))) THEN
END
ELSE

test$ = UCASE$(test$)
IF INSTR(test$,"Y") THEN
INCR lapptr
GOTO start
ENDIF
END IF

[alexfish]

Hi All

This is a base line

need to compile this as

baconduckduck

then place in the /usr/bin

/usr/bin/baconduckduck

the code::

' program "baconduckduck " A Simple Terminal Search engine > requires Curl
' this needs to be in /usr/bin
' sudo cp baconduckduck /usr/bin/baconduckduck
PROTO system
LOCAL Inp$
LOCAL tags=0
LOCAL links TYPE int
LOCAL lapptr =0
LOCAL browser=0


IF argc <2 THEN

Inp$="News"
ELSE

FOR y = 1 TO argc-1
Inp$=Inp$ & argv[y] & "%"
NEXT
END IF


LOCAL Search$= "curl https://duckduckgo.com/html/?q=" & Inp$

ALLHTMLTAGS$=""


' Must hav a result ------------------------------
IF LEN(Inp$) > 0 THEN

ELSE
Inp$ = "News"

END IF

PRINT "DuckDuckGo Searching for:: " & Inp$

SPLIT Inp$ BY " " TO Bits$ SIZE x


FOR n = 0 TO x-1

IF LEN(Bits$[n])>0 THEN
Bits$[n]= CHOP$ (Bits$[n])

IF n < x-1 THEN
Search$=Search$ & Bits$[n] & "%"
ELSE
Search$=Search$ & Bits$[n]
ENDIF

ENDIF

NEXT


HTML$ = EXEC$(Search$)
IF NOT(INSTR(HTML$,"<a")) THEN
PRINT " Can not Resolve Host : DuckDuckGo"
PRINT " Check Internet Connection"
END
END IF
'Clean Up    and save to file

SPLIT  HTML$ BY "\n" TO Res$ SIZE x

FOR n = 0 TO x-1

Res$[n]= CHOP$ (Res$[n])
IF LEN(Res$[n])>0 THEN
tem$ = LEFT$(Res$[n],1)
IF tem$ == "<" THEN
Results$ = CHOP$(Results$ & Res$[n]) & "\n"
END IF
END IF

NEXT


SAVE Results$ TO CURDIR$ & "/index.html"
IF FILEEXISTS(CURDIR$&"/index.html") THEN

PRINT "Results saved in --> " & CURDIR$&"/index.html"
Result$ = ""

OPEN CURDIR$&"/index.html" FOR READING AS myfile
WHILE NOT(ENDFILE(myfile)) DO
READLN txt$ FROM myfile
txt$ = CHOP$(txt$)
IF NOT(ENDFILE(myfile)) THEN
IF LEFT$(txt$,2) == "<a"  THEN
IF (INSTR (txt$,"Feedback"))  AND (lapptr>0) THEN
DECR tags

END IF
'http://baconbashtexas.com/%3Fproduct%3D45
'baconbashtexas.com/?product=45

txt$ = REPLACE$(txt$,"&;","'")
txt$ = REPLACE$(txt$,"'","'")
txt$ = REPLACE$(txt$,"</b>","")
txt$ = REPLACE$(txt$,"<b>","")
txt$ = REPLACE$(txt$,"</a>","")
txt$ = REPLACE$(txt$,"%3A",":")
txt$ = REPLACE$(txt$,"%2F","/")
txt$ = REPLACE$(txt$,"%2D","-")
txt$ = REPLACE$(txt$,"%3D","=")
txt$ = REPLACE$(txt$,"%3F","?")
txt$ = REPLACE$(txt$,"%23","#")
IF INSTR(txt$,"nofollow") THEN
HTMLTAG$ = LEFT$(txt$,INSTR(txt$,">"))
reset = 1
END IF

IF (INSTR(txt$,"nofollow" )) THEN

INCR tags

END IF

IF LEN( MID$(txt$,INSTR(txt$,">")+1)) THEN
IF reset THEN
PRINT  "title=" , MID$(txt$,INSTR(txt$,">")+1)
reset=0
ELSE
PRINT  "snippet=" , MID$(txt$,INSTR(txt$,">")+1)
END IF
END IF
IF INSTR(txt$,"result__url")  THEN
HTMLTAG$ =MID$(HTMLTAG$,INSTR(HTMLTAG$,"http"))
HTMLTAG$= LEFT$(HTMLTAG$,LEN(HTMLTAG$)-2)
PRINT "href=" & HTMLTAG$
ALLHTMLTAGS$=ALLHTMLTAGS$ &  HTMLTAG$ & "\n"
END IF
END IF
ENDIF
WEND
CLOSE FILE myfile
END IF


This is the calling program
IF have uo to date terminal , right click on the line beginning with "href=" to Copy URL

' program getsearch .. calls /usr/bin/baconduckduck

LOCAL  Inp$ = " "

LABEL start
CLEAR
Inp$ = " "

INPUT " q To Quit or Seach for >", Ino$
Ino$=CHOP$(Ino$)
IF Ino$=="q" THEN
END
END IF
IF Ino$=="Q" THEN
END
END IF

IF NOT(LEN(Ino$)) THEN
A$=EXEC$ ("baconduckduck")

ELSE
A$=EXEC$ ("baconduckduck " & Ino$)
END IF

PRINT A$
PRINT "******************************************************"
PRINT
INPUT "Press Key to continue >" ,key$
GOTO start

BR
+ have fun
Alex

[alexfish]

Hi All

It is easy to spot the ones to avoid

BR
Alex

[alexfish]

Hi Vovchik

Now looking at the curl.bac again . need to refresh my memory cells from API & demos as to what they did


on the raspberry pi

have try a few example & test with

libcurl4-openssl-dev ,, so need to also look at the header files..

libcurlpp-dev
c++ wrapper for libcurl (development files) yet that API was easy


also phantomjs , not work the effort, segfult,segfault ,,blagh blag , so not sure what is up there

BR
Alex

[alexfish]

Hi Vovchik

after a bit of fiddling with the encoded urls ,

IE

"https%3A%2F%2Fwww.matthartley.com%2Fbacon%2Dand%2Dlinux%2F"


did a test of replace the % with '&#x' & append ';'

svg example // test file is in the archive + have added HUG MARK example :: mark2.bac


<!--https%3A%2F%2Fwww.matthartley.com%2Fbacon%2Dand%2Dlinux%2F-->
<!--Replace '%' with '&#x' and append ';' -->
<svg width="600" height="600">
 <text x="0" y="15" fill="blue"> https://www.matthartley.com/bacon-and-linux/</text>
</svg>

BR
Alex

To All
need a svg compatible viewer like 'rsvg-view-3' from svg bin tools

Attachments:

text.svg.bz2 (231 B)

mark2.bac.bz2 (209 B)

[vovchik]

Dear Alex,

Yep, that works and looks perfect.

With kind regards,
vovchik

[alexfish]

Jul 6, 2017 17:10:58 GMT 1 vovchik said:

Dear Alex,

Yep, that works and looks perfect.

With kind regards,
vovchik

even better can get the translate text by
GRAB$

IE



A$ = GRAB$(mark)
PRINT A$

BR
Alex

[alexfish]

Hi All

did some further reading

in the archive is a simple example of the url & text snippet from a search
did not Know , can copy a label from a pop up menu ,

now need to look the Hug IMAGE:: that is if want graphics enable in a simple browser

BR
Alex

Attachments:

mark3.bac.bz2 (469 B)

[alexfish]

Hi Vovchik & Peter

had a few problems with Peter's curl example

in 1, Segfaults
in 2, not receiving all the data IE:: only part of the PAGE recieved

adapted code to see what I be looking at,

anyway finally arrived at this

hope it works

BR
Alex

' Include the CURL context
INCLUDE "curl.bac"

' We store our data in this variable
DECLARE mail$

' Callback to let Curl gather data
FUNCTION Save_Data(char * buffer, size_t size, size_t nmemb, void *userp)
   mail$ =  CHOP$(buffer)
mail$ = LEFT$(mail$,LEN(mail$)-1)
   RETURN size*nmemb

END FUNCTION

' Create the handle
easyhandle = curl_easy_init()

' Set the options
curl_easy_setopt(easyhandle, CURLOPT_URL, "https://duckduckgo.com/html/?q=BaconConverter")
curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, Save_Data)

' Perform the GET
success = curl_easy_perform(easyhandle)

' Cleanup
curl_easy_cleanup(easyhandle)
PRINT mail$

ADDED IF Wondering what the diff is in URL above IE the /html/ and what some of the problems lay

can try this as well

' Include the CURL context
INCLUDE "curl.bac"

'
We store our data in this variable
DECLARE mail$

' Callback to let Curl gather data
FUNCTION Save_Data(char * buffer, size_t size, size_t nmemb, void *userp)
   mail$ =  CHOP$(buffer)
mail$ = LEFT$(mail$,LEN(mail$)-1)
   RETURN size*nmemb

END FUNCTION

' Create the handle
easyhandle = curl_easy_init()

' Set the options
curl_easy_setopt(easyhandle, CURLOPT_URL, "https://duckduckgo.com/?q=BaconConverter")
curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, Save_Data)

' Perform the GET
success = curl_easy_perform(easyhandle)

' Cleanup
curl_easy_cleanup(easyhandle)
PRINT mail$

[vovchik]

Dear Alex,

This one works (in terms of looking like an html page):

"https://duckduckgo.com/html/?q=BaconConverter"

The other URL does indeed get something, but not, it seems, what we want...

And the links presented are invalid, for some reason. Wonder why. Are we not supplying some essential CURLOPT_?

With kind regards,
vovchik

[alexfish]

Jul 8, 2017 11:50:24 GMT 1 vovchik said:

Dear Alex,

This one works (in terms of looking like an html page):

"https://duckduckgo.com/html/?q=BaconConverter"

The other URL does indeed get something, but not, it seems, what we want...

And the links presented are invalid, for some reason. Wonder why. Are we not supplying some essential CURLOPT_?

With kind regards,
vovchik

MM! started think , as far as can work out there are recursive call backs , since the data is streamed in chunks

however the html call works in this case even if the html is placed in the '.com/html?q=BaconConverter"'

did a recursive test , found the beginning of file can , as in chunk's of data were missing, IE on getting Last Chunk of data

as far as the chunks are concern here is a fix



BR
Alex


' Include the CURL context
INCLUDE "curl.bac"

' We store our data in this variable
DECLARE mail$
DECLARE MAIL$=""

' Callback to let Curl gather data
FUNCTION Save_Data(char* buffer, size_t size, size_t nmemb, void *userp)

   mail$ =  CHOP$(buffer)
    MAIL$ = MAIL$ & mail$
'mail$ = LEFT$(mail$,LEN(mail$)-1)
   RETURN size*nmemb

END FUNCTION

' Clean end of html
SUB Clean()
IF RIGHT$(MAIL$,1) = "0" THEN
MAIL$ = LEFT$(MAIL$,LEN(MAIL$)-1)
END IF

END SUB


SUB Search()
' Create the handle
easyhandle = curl_easy_init()

' Set the options
curl_easy_setopt(easyhandle, CURLOPT_URL, "https://duckduckgo.com/html?q=BaconConverter")
curl_easy_setopt(easyhandle, CURLOPT_WRITEFUNCTION, Save_Data)

' Perform the GET
success = curl_easy_perform(easyhandle)

' Cleanup
curl_easy_cleanup(easyhandle)
CALL Clean
END SUB

CALL Search()


PRINT MAIL$

[alexfish]

Jul 2, 2017 20:13:11 GMT 1 vovchik said:

Dear Alex,

Thanks. What about using "curl.bac"? It should be possible, too. I have been playing with it and can grab the bacon site but am getting errors (due to my ignorance) when I try https.

With kind regards,
vovchik

now looking.

BR
Alex