|
Post by bigbass on Aug 2, 2022 22:28:50 GMT 1
Hello vovchik did some heavy editing to the latest version above
version 1.5
should work now
Joe
|
|
|
Post by vovchik on Aug 3, 2022 0:28:32 GMT 1
Dear Joe,
Thanks. Perfect on Pi, but still a bit weird on Mint.
I changed
DEF FN PRINTOUT(x) = qDebug(x)
and
DEF FN PRINTABLE(p) = qPrintable(p)
to
ALIAS qDebug TO PRINTOUT
and
ALIAS qPrintable TO PRINTABLE
which got me a running Mint binary, and when I added TRAP LOCAL at the top, it would run without a segfault error.
I am still investigating...
With kind regards, vovchik
|
|
|
Post by bigbass on Aug 3, 2022 2:22:19 GMT 1
Hello vovchik
I found the problem with the empty loops carefully stepping through the code I set i to 2 now it is correct and some more fixes
uncomment TRACE ON and step though the code if it doesnt work and give me a line number where it crashes
note that if the target directory is bad or missing you will get that error you are seeing also
your error reports forced me to go over the code more carefully and in the end we get improvements thanks for your patients
' added application/x-www-form-urlencoded downloads are much faster now
'---VERSION 1.7 Aug 2 2022 cli-downloader '---option -o ~/Downloads/ url url '---must have trailing slash when using the ~/target_dir/ ! '---ported https://code.qt.io/cgit/qt/qtbase.git/tree/examples/network/download/main.cpp?h=5.15 '---and removed class code heavily modified for bacon use by bigbass ' fmt$ = "%s%s\n" fix for mint thanks vovchik ' fixed empty arguments display help ' removed c++ code from bacons PRINT ' handle c++ with PRINTOUT and PRINTABLE ' changed i to start at 2 this fixed empty urls ' added application/x-www-form-urlencoded download are much faster now
PRAGMA INCLUDE <iostream> <QtCore/QDebug> <QtCore/QDir> <QtCore/QDir> PRAGMA INCLUDE <QtCore/QStringList> <QtCore/QFile> <QtCore/QFileInfo> PRAGMA INCLUDE <QtCore/QMimeDatabase> <QtCore/QMimeType> <QtCore/QTextStream> PRAGMA INCLUDE <QtCore/QUrl> <QtCore/QUrlQuery> <QtNetwork/QNetworkReply> PRAGMA INCLUDE <QtCore/QDateTime> <QtCore/QSize> PRAGMA INCLUDE <QtNetwork/QNetworkAccessManager> <QtNetwork/QNetworkRequest> PRAGMA INCLUDE <cstdio> <QtCore/QTimer> <QtCore/QCoreApplication> PRAGMA INCLUDE <QtCore/QCommandLineParser> <QtCore/QCommandLineOption> PRAGMA LDFLAGS -lQt5Network -lQt5Core -lpthread -latomic PRAGMA COMPILER g++ 'cli-downloader light version for cli only no widget dependencies added
'RPI3 PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wformat=1 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5/QtNetwork -fPIC
'x86 'PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith 'PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5 'PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -fPIC
'depends atleast qtbase5-dev libqt5websockets5
OPTION PARSE FALSE
DEF FN PRINTOUT(x) = qDebug(x) DEF FN PRINTABLE(p) = qPrintable(p)
DECLARE app TYPE QCoreApplication* DECLARE manager TYPE QNetworkAccessManager* DECLARE reply TYPE QNetworkReply* DECLARE request TYPE QNetworkRequest DECLARE currentDownloads TYPE QList<QNetworkReply *> DECLARE dlcounter = 0 TYPE int DECLARE fincounter = 0 TYPE int
DECLARE target_dir TYPE QString DECLARE filePath TYPE QString DECLARE cli_option TYPE QString
manager = new QNetworkAccessManager()
'had trouble globally seeing and comparing the option cli_option = "-o" '-------------------------------------------- SUB DownloadManager() '-------------------------------------------- '(1) COLOR FG TO INTENSE WHITE PRINT "(1) initialize the Download manager" COLOR RESET QObject::connect(manager, &QNetworkAccessManager::finished, downloadFinished) END SUB
'-------------------------------------------- SUB execute() '-------------------------------------------- COLOR FG TO INTENSE WHITE PRINT "(2) Process the command line urls" COLOR RESET LOCAL nameofapp TYPE const char* LOCAL str TYPE QString LOCAL arg2 TYPE QString QStringList args = QCoreApplication::instance()->arguments() PRINTOUT(args.at(0).toLocal8Bit()) '---use bacon PRINT as a char* HACK for nameofapp nameofapp = PRINTABLE(args.at(0).toLocal8Bit())
' skip the first argument, which is the program's name args.takeFirst() IF args.isEmpty() == true THEN COLOR FG TO INTENSE BLUE PRINT "Qt Downloader - downloads a single or multiple URLs by bigbass July 31 2022" PRINT "Usage: ",nameofapp, " url" PRINT "Downloads the URL or URLs passed in the command-line to the local directory" PRINT "If the target file already exists it will get over written with the new download" COLOR RESET END ELSE '---option homemade -o parser hack to a target_dir IF args.at(0).toLocal8Bit() == cli_option THEN 'get the second arg then remove it from the list later arg2 = args.at(1).toLocal8Bit() 'copy arg2 to the target_dir delete arg2 from the download list target_dir = arg2 COLOR FG TO INTENSE BLUE PRINT "-o DIR you want the download to go " PRINTOUT(PRINTABLE(target_dir)) COLOR RESET 'remove the option -o args.replaceInStrings(cli_option,"") 'remove the arg2 which is the target_dir args.replaceInStrings(arg2,"") args.replaceInStrings(QRegExp("^\\s+"),"") END IF 'changed i to 2 ver 1.6 LOCAL Multi_url TYPE QString LOCAL i = 2 TYPE int LOCAL num TYPE int num = args.size() REPEAT Multi_url = args.at(i).toLocal8Bit() IF (Multi_url.isEmpty()) THEN ' do nothing and stay here if a bad url happens ELSE CALL doDownload(Multi_url) END IF 'PRINTOUT(PRINTABLE(args.at(i).toLocal8Bit())) 'PRINT "i = ",i INCR i UNTIL num == i END IF END SUB
'-------------------------------------------- SUB doDownload(const QUrl &url) '-------------------------------------------- 'PRINT "(3)" IF (url.isEmpty()) THEN 'if you arrive here i needs to be set to 2 to fix it ELSE QList<QNetworkReply *> currentDownloads QNetworkRequest request(url) 'force a user agent request.setRawHeader( "User-Agent" , "Mozilla Firefox" ) request.setHeader(QNetworkRequest::ContentTypeHeader, "application/x-www-form-urlencoded") QNetworkReply *reply = manager->get(request) currentDownloads.append(reply) '--- use a dlcounter to see how many urls are used INCR dlcounter END IF END SUB
'-------------------------------------------- SUB downloadFinished(QNetworkReply *reply) '-------------------------------------------- 'PRINT "(4)" LOCAL url TYPE QUrl LOCAL fmt$ TYPE STRING url = reply->url() IF (reply->error()) THEN COLOR FG TO INTENSE RED EPRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT " failed: " PRINTOUT(PRINTABLE(reply->errorString())) COLOR RESET ELSE IF (isHttpRedirect(reply)) THEN COLOR FG TO INTENSE BLUE fprintf(stderr, "Request was redirected.\n") COLOR RESET ELSE '--- use a fincounter to see how many urls finshed and downloaded QString filename = saveFileName(url) filePath = target_dir + filename IF (saveToDisk(filePath, reply)) THEN COLOR FG TO INTENSE YELLOW PRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT "succeeded! " PRINT "saved as " PRINTOUT(PRINTABLE(filename)) INCR fincounter COLOR RESET END IF END IF END IF IF (dlcounter == fincounter ) THEN COLOR FG TO GREEN PRINT "All downloads finished !" COLOR RESET END END IF END SUB
'-------------------------------------------- FUNCTION isHttpRedirect(QNetworkReply *reply) TYPE bool '-------------------------------------------- 'PRINT "(5)" LOCAL statusCode TYPE int statusCode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt() RETURN (statusCode == 301 || statusCode == 302 || statusCode == 303 || \ statusCode == 305 || statusCode == 307 || statusCode == 308) END FUNCTION
'-------------------------------------------- FUNCTION saveFileName(const QUrl &url) TYPE QString '-------------------------------------------- 'PRINT "(6)" LOCAL path TYPE QString LOCAL basename TYPE QString
path = url.path() basename = QFileInfo(path).fileName()
IF (basename.isEmpty()) THEN basename = "download" ELSE
RETURN basename END IF END FUNCTION
'-------------------------------------------- FUNCTION saveToDisk(const QString &filename, QIODevice *data) TYPE bool '-------------------------------------------- 'PRINT "(7)" QFile file(filename) IF (file.open(QIODevice::WriteOnly)) == false THEN COLOR FG TO INTENSE RED EPRINT "Could not open " PRINTOUT(PRINTABLE(filename)) PRINT " for writing: " PRINTOUT(PRINTABLE(file.errorString())) COLOR RESET RETURN FALSE ELSE file.write(data->readAll()) file.close()
RETURN TRUE END IF END FUNCTION
'---main app = new QCoreApplication(argc, argv) 'special note we use QCoreApplication 'instead of the standard QApplication when its all cli CALL DownloadManager CALL execute app->exec()
|
|
|
Post by bigbass on Sept 8, 2022 16:44:56 GMT 1
Hello I am running mint 21 x86_64 live USB and doing some cleanup work testing on mint this is for x86_64 default I just added -Wno-format-security -Wformat=1 and it is working at least on mint 21 I will look at code that gave problems on mint still sorting through posts... *I want to deb package some things and test the installer 'depends atleast qtbase5-dev libqt5websockets5 another test note if you dont use the -o option it defaults to current directory ./cli-downloader -o ~/Downloads/ http://www.basic-converter.org/doc_frame.html \ http://www.basic-converter.org/hug.bac \ http://www.basic-converter.org/stable/CHANGES \ http://www.basic-converter.org/stable/bacon-4.5.tar.gz
Joe mkdir -p ~/maps/17/65489/
./cli-downloader -o ~/maps/17/65489/ https://tile.openstreetmap.org/17/65489/43588.png
'---VERSION 1.8 Sept 8 2022 cli-downloader '---option -o ~/Downloads/ url url '---must have trailing slash when using the ~/target_dir/ ! '---ported https://code.qt.io/cgit/qt/qtbase.git/tree/examples/network/download/main.cpp?h=5.15 '---and removed class code heavily modified for bacon use by bigbass ' fmt$ = "%s%s\n" fix for mint thanks vovchik ' fixed empty arguments display help ' removed c++ code from bacons PRINT ' handle c++ with PRINTOUT and PRINTABLE ' changed i to start at 2 this fixed empty urls ' added application/x-www-form-urlencoded download are much faster now ' -Wno-format-security tested on mint 21
PRAGMA INCLUDE <iostream> <QtCore/QDebug> <QtCore/QDir> <QtCore/QDir> PRAGMA INCLUDE <QtCore/QStringList> <QtCore/QFile> <QtCore/QFileInfo> PRAGMA INCLUDE <QtCore/QMimeDatabase> <QtCore/QMimeType> <QtCore/QTextStream> PRAGMA INCLUDE <QtCore/QUrl> <QtCore/QUrlQuery> <QtNetwork/QNetworkReply> PRAGMA INCLUDE <QtCore/QDateTime> <QtCore/QSize> PRAGMA INCLUDE <QtNetwork/QNetworkAccessManager> <QtNetwork/QNetworkRequest> PRAGMA INCLUDE <cstdio> <QtCore/QTimer> <QtCore/QCoreApplication> PRAGMA INCLUDE <QtCore/QCommandLineParser> <QtCore/QCommandLineOption> PRAGMA LDFLAGS -lQt5Network -lQt5Core -lpthread -latomic PRAGMA COMPILER g++ 'cli-downloader light version for cli only no widget dependencies added
'RPI3 'PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wformat=1 'PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5 'PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5/QtNetwork -fPIC
'x86 PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wno-format-security -Wformat=1 PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5 PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -fPIC
'depends atleast qtbase5-dev libqt5websockets5
OPTION PARSE FALSE
DEF FN PRINTOUT(x) = qDebug(x) DEF FN PRINTABLE(p) = qPrintable(p)
DECLARE app TYPE QCoreApplication* DECLARE manager TYPE QNetworkAccessManager* DECLARE reply TYPE QNetworkReply* DECLARE request TYPE QNetworkRequest DECLARE currentDownloads TYPE QList<QNetworkReply *> DECLARE dlcounter = 0 TYPE int DECLARE fincounter = 0 TYPE int
DECLARE target_dir TYPE QString DECLARE filePath TYPE QString DECLARE cli_option TYPE QString
manager = new QNetworkAccessManager()
'had trouble globally seeing and comparing the option cli_option = "-o" '-------------------------------------------- SUB DownloadManager() '-------------------------------------------- '(1) COLOR FG TO INTENSE WHITE PRINT "(1) initialize the Download manager" COLOR RESET QObject::connect(manager, &QNetworkAccessManager::finished, downloadFinished) END SUB
'-------------------------------------------- SUB execute() '-------------------------------------------- COLOR FG TO INTENSE WHITE PRINT "(2) Process the command line urls" COLOR RESET LOCAL nameofapp TYPE const char* LOCAL str TYPE QString LOCAL arg2 TYPE QString QStringList args = QCoreApplication::instance()->arguments() PRINTOUT(args.at(0).toLocal8Bit()) '---use bacon PRINT as a char* HACK for nameofapp nameofapp = PRINTABLE(args.at(0).toLocal8Bit())
' skip the first argument, which is the program's name args.takeFirst() IF args.isEmpty() == true THEN COLOR FG TO INTENSE BLUE PRINT "Qt Downloader - downloads a single or multiple URLs by bigbass July 31 2022" PRINT "Usage: ",nameofapp, " url" PRINT "Downloads the URL or URLs passed in the command-line to the local directory" PRINT "If the target file already exists it will get over written with the new download" COLOR RESET END ELSE '---option homemade -o parser hack to a target_dir IF args.at(0).toLocal8Bit() == cli_option THEN 'get the second arg then remove it from the list later arg2 = args.at(1).toLocal8Bit() 'copy arg2 to the target_dir delete arg2 from the download list target_dir = arg2 COLOR FG TO INTENSE BLUE PRINT "-o DIR you want the download to go " PRINTOUT(PRINTABLE(target_dir)) COLOR RESET 'remove the option -o args.replaceInStrings(cli_option,"") 'remove the arg2 which is the target_dir args.replaceInStrings(arg2,"") args.replaceInStrings(QRegExp("^\\s+"),"") END IF 'changed i to 2 ver 1.6 LOCAL Multi_url TYPE QString LOCAL i = 2 TYPE int LOCAL num TYPE int num = args.size() REPEAT Multi_url = args.at(i).toLocal8Bit() IF (Multi_url.isEmpty()) THEN ' do nothing and stay here if a bad url happens ELSE CALL doDownload(Multi_url) END IF 'PRINTOUT(PRINTABLE(args.at(i).toLocal8Bit())) 'PRINT "i = ",i INCR i UNTIL num == i END IF END SUB
'-------------------------------------------- SUB doDownload(const QUrl &url) '-------------------------------------------- 'PRINT "(3)" IF (url.isEmpty()) THEN 'if you arrive here i needs to be set to 2 to fix it ELSE QList<QNetworkReply *> currentDownloads QNetworkRequest request(url) 'force a user agent request.setRawHeader( "User-Agent" , "Mozilla Firefox" ) request.setHeader(QNetworkRequest::ContentTypeHeader, "application/x-www-form-urlencoded") QNetworkReply *reply = manager->get(request) currentDownloads.append(reply) '--- use a dlcounter to see how many urls are used INCR dlcounter END IF END SUB
'-------------------------------------------- SUB downloadFinished(QNetworkReply *reply) '-------------------------------------------- 'PRINT "(4)" LOCAL url TYPE QUrl LOCAL fmt$ TYPE STRING url = reply->url() IF (reply->error()) THEN COLOR FG TO INTENSE RED EPRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT " failed: " PRINTOUT(PRINTABLE(reply->errorString())) COLOR RESET ELSE IF (isHttpRedirect(reply)) THEN COLOR FG TO INTENSE BLUE fprintf(stderr, "Request was redirected.\n") COLOR RESET ELSE '--- use a fincounter to see how many urls finshed and downloaded QString filename = saveFileName(url) filePath = target_dir + filename IF (saveToDisk(filePath, reply)) THEN COLOR FG TO INTENSE YELLOW PRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT "succeeded! " PRINT "saved as " PRINTOUT(PRINTABLE(filename)) INCR fincounter COLOR RESET END IF END IF END IF IF (dlcounter == fincounter ) THEN COLOR FG TO GREEN PRINT "All downloads finished !" COLOR RESET END END IF END SUB
'-------------------------------------------- FUNCTION isHttpRedirect(QNetworkReply *reply) TYPE bool '-------------------------------------------- 'PRINT "(5)" LOCAL statusCode TYPE int statusCode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt() RETURN (statusCode == 301 || statusCode == 302 || statusCode == 303 || \ statusCode == 305 || statusCode == 307 || statusCode == 308) END FUNCTION
'-------------------------------------------- FUNCTION saveFileName(const QUrl &url) TYPE QString '-------------------------------------------- 'PRINT "(6)" LOCAL path TYPE QString LOCAL basename TYPE QString
path = url.path() basename = QFileInfo(path).fileName()
IF (basename.isEmpty()) THEN basename = "download" ELSE
RETURN basename END IF END FUNCTION
'-------------------------------------------- FUNCTION saveToDisk(const QString &filename, QIODevice *data) TYPE bool '-------------------------------------------- 'PRINT "(7)" QFile file(filename) IF (file.open(QIODevice::WriteOnly)) == false THEN COLOR FG TO INTENSE RED EPRINT "Could not open " PRINTOUT(PRINTABLE(filename)) PRINT " for writing: " PRINTOUT(PRINTABLE(file.errorString())) COLOR RESET RETURN FALSE ELSE file.write(data->readAll()) file.close()
RETURN TRUE END IF END FUNCTION
'---main app = new QCoreApplication(argc, argv) 'special note we use QCoreApplication 'instead of the standard QApplication when its all cli CALL DownloadManager CALL execute app->exec()
|
|
|
Post by vovchik on Sept 8, 2022 17:55:04 GMT 1
Dear Joe,
Thanks. It works fine on Mint 19 (x86_64) and on my Pi4. Great.
With kind regards, vovchik
|
|
|
Post by bigbass on Sept 8, 2022 18:20:17 GMT 1
Hello vovchik
thanks for the feedback I will clean up some of the demos that were giving errors on x86_64 and will test out code on a recent mint version to see if anything needed to be adjusted
some tech notes: I see that the gcc version in mint is 11 I will adjust some of my cmake cross scripts to allow for different versions
we all want to move forward to the next new thing but it is important to go back and clean up code update it and document somethings along the way
thanks for your report!
Joe
|
|
|
Post by alexfish on Sept 8, 2022 20:40:53 GMT 1
Hi Joe
I am getting compiler problem , raspberry pi4
Bacon fossil , 'BaCon version 4.5.1 on Linux armv7l - (c) Peter van Eerten - MIT License.'
WARNING: variable '*reply' in function header at line 179 in file '/home/pi/bigbass/cli3.bac' was defined previously!
WARNING: variable '*reply' in function header at line 228 in file '/home/pi/bigbass/cli3.bac' was defined previously!
'going to update fossil and see what happens.
BR Alex
|
|
|
Post by alexfish on Sept 8, 2022 20:57:40 GMT 1
Agh
just notice RPI3 bits this will compile and exec, but still shows those warnings
'RPI3 PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wformat=1 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5/QtNetwork -fPIC
previous would not compile,
ADDED:
if comment out the reply in the DECLARES , it compiles and exec works
DECLARE app TYPE QCoreApplication* DECLARE manager TYPE QNetworkAccessManager* 'DECLARE reply TYPE QNetworkReply* DECLARE request TYPE QNetworkRequest DECLARE currentDownloads TYPE QList<QNetworkReply *> DECLARE dlcounter = 0 TYPE int DECLARE fincounter = 0 TYPE int
result
-o ~/maps/17/65489/ https://tile.openstreetmap.org/17/65489/43588.png (1) initialize the Download manager (2) Process the command line urls ./cli3 -o DIR you want the download to go /home/pi/maps/17/65489/ Download of https://tile.openstreetmap.org/17/65489/43588.png succeeded! saved as 43588.png All downloads finished !
Possible global in function same 'QNetworkReply *reply = manager->get(request)'
SUB doDownload(const QUrl &url) '-------------------------------------------- 'PRINT "(3)" IF (url.isEmpty()) THEN 'if you arrive here i needs to be set to 2 to fix it ELSE QList<QNetworkReply *> currentDownloads QNetworkRequest request(url) 'force a user agent request.setRawHeader( "User-Agent" , "Mozilla Firefox" ) request.setHeader(QNetworkRequest::ContentTypeHeader, "application/x-www-form-urlencoded") QNetworkReply *reply = manager->get(request) currentDownloads.append(reply) '--- use a dlcounter to see how many urls are used INCR dlcounter END IF END SUB
Also tested openseamap/seamark , that works
-o ~/maps/16/32145/ https://t1.openseamap.org/seamark/16/32145/20402.png as does save as like so,
-o ~/maps/16/32145/S20402.png https://t1.openseamap.org/seamark/16/32145/20402.png
BR Alex
|
|
|
Post by bigbass on Sept 8, 2022 21:53:38 GMT 1
thanks Alex for the report I got those warnings too but it worked and thought to leave it alone but yes we should go for a clean compile and take risks lines #172 and #173 I renamed reply to dlreply (for download reply) and commented out the DECLARE it is working on mint 21 please let me know the results on RPI4 this is preset for x86_64 P.S I confirmed the other two links you posted worked so I hope all is well *I should have it make the directory if it doesn't exist will have some free time late tonight to look at that idea Joe
'---VERSION 1.9 Aug 2 2022 cli-downloader '---option -o ~/Downloads/ url url '---must have trailing slash when using the ~/target_dir/ ! '---ported https://code.qt.io/cgit/qt/qtbase.git/tree/examples/network/download/main.cpp?h=5.15 '---and removed class code heavily modified for bacon use by bigbass ' fmt$ = "%s%s\n" fix for mint thanks vovchik ' fixed empty arguments display help ' removed c++ code from bacons PRINT ' handle c++ with PRINTOUT and PRINTABLE ' changed i to start at 2 this fixed empty urls ' added application/x-www-form-urlencoded download are much faster now ' -Wno-format-security tested on mint 21 ' reply warning fixed
PRAGMA INCLUDE <iostream> <QtCore/QDebug> <QtCore/QDir> <QtCore/QDir> PRAGMA INCLUDE <QtCore/QStringList> <QtCore/QFile> <QtCore/QFileInfo> PRAGMA INCLUDE <QtCore/QMimeDatabase> <QtCore/QMimeType> <QtCore/QTextStream> PRAGMA INCLUDE <QtCore/QUrl> <QtCore/QUrlQuery> <QtNetwork/QNetworkReply> PRAGMA INCLUDE <QtCore/QDateTime> <QtCore/QSize> PRAGMA INCLUDE <QtNetwork/QNetworkAccessManager> <QtNetwork/QNetworkRequest> PRAGMA INCLUDE <cstdio> <QtCore/QTimer> <QtCore/QCoreApplication> PRAGMA INCLUDE <QtCore/QCommandLineParser> <QtCore/QCommandLineOption> PRAGMA LDFLAGS -lQt5Network -lQt5Core -lpthread -latomic PRAGMA COMPILER g++ 'cli-downloader light version for cli only no widget dependencies added
'RPI3 'PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wformat=1 'PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5 'PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5/QtNetwork -fPIC
'x86 PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wno-format-security -Wformat=1 PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5 PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -fPIC
'depends at least qtbase5-dev libqt5websockets5
OPTION PARSE FALSE
DEF FN PRINTOUT(x) = qDebug(x) DEF FN PRINTABLE(p) = qPrintable(p)
DECLARE app TYPE QCoreApplication* DECLARE manager TYPE QNetworkAccessManager* 'DECLARE reply TYPE QNetworkReply* DECLARE request TYPE QNetworkRequest DECLARE currentDownloads TYPE QList<QNetworkReply *> DECLARE dlcounter = 0 TYPE int DECLARE fincounter = 0 TYPE int
DECLARE target_dir TYPE QString DECLARE filePath TYPE QString DECLARE cli_option TYPE QString
manager = new QNetworkAccessManager()
'had trouble globally seeing and comparing the option cli_option = "-o" '-------------------------------------------- SUB DownloadManager() '-------------------------------------------- '(1) COLOR FG TO INTENSE WHITE PRINT "(1) initialize the Download manager" COLOR RESET QObject::connect(manager, &QNetworkAccessManager::finished, downloadFinished) END SUB
'-------------------------------------------- SUB execute() '-------------------------------------------- COLOR FG TO INTENSE WHITE PRINT "(2) Process the command line urls" COLOR RESET LOCAL nameofapp TYPE const char* LOCAL str TYPE QString LOCAL arg2 TYPE QString QStringList args = QCoreApplication::instance()->arguments() PRINTOUT(args.at(0).toLocal8Bit()) '---use bacon PRINT as a char* HACK for nameofapp nameofapp = PRINTABLE(args.at(0).toLocal8Bit())
' skip the first argument, which is the program's name args.takeFirst() IF args.isEmpty() == true THEN COLOR FG TO INTENSE BLUE PRINT "Qt Downloader - downloads a single or multiple URLs by bigbass July 31 2022" PRINT "Usage: ",nameofapp, " url" PRINT "Downloads the URL or URLs passed in the command-line to the local directory" PRINT "If the target file already exists it will get over written with the new download" COLOR RESET END ELSE '---option homemade -o parser hack to a target_dir IF args.at(0).toLocal8Bit() == cli_option THEN 'get the second arg then remove it from the list later arg2 = args.at(1).toLocal8Bit() 'copy arg2 to the target_dir delete arg2 from the download list target_dir = arg2 COLOR FG TO INTENSE BLUE PRINT "-o DIR you want the download to go " PRINTOUT(PRINTABLE(target_dir)) COLOR RESET 'remove the option -o args.replaceInStrings(cli_option,"") 'remove the arg2 which is the target_dir args.replaceInStrings(arg2,"") args.replaceInStrings(QRegExp("^\\s+"),"") END IF 'changed i to 2 ver 1.6 LOCAL Multi_url TYPE QString LOCAL i = 2 TYPE int LOCAL num TYPE int num = args.size() REPEAT Multi_url = args.at(i).toLocal8Bit() IF (Multi_url.isEmpty()) THEN ' do nothing and stay here if a bad url happens ELSE CALL doDownload(Multi_url) END IF 'PRINTOUT(PRINTABLE(args.at(i).toLocal8Bit())) 'PRINT "i = ",i INCR i UNTIL num == i END IF END SUB
'-------------------------------------------- SUB doDownload(const QUrl &url) '-------------------------------------------- 'PRINT "(3)" IF (url.isEmpty()) THEN 'if you arrive here i needs to be set to 2 to fix it ELSE QList<QNetworkReply *> currentDownloads QNetworkRequest request(url) 'force a user agent request.setRawHeader( "User-Agent" , "Mozilla Firefox" ) request.setHeader(QNetworkRequest::ContentTypeHeader, "application/x-www-form-urlencoded") QNetworkReply *dlreply = manager->get(request) currentDownloads.append(dlreply) '--- use a dlcounter to see how many urls are used INCR dlcounter END IF END SUB
'-------------------------------------------- SUB downloadFinished(QNetworkReply *reply) '-------------------------------------------- 'PRINT "(4)" LOCAL url TYPE QUrl LOCAL fmt$ TYPE STRING url = reply->url() IF (reply->error()) THEN COLOR FG TO INTENSE RED EPRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT " failed: " PRINTOUT(PRINTABLE(reply->errorString())) COLOR RESET ELSE IF (isHttpRedirect(reply)) THEN COLOR FG TO INTENSE BLUE fprintf(stderr, "Request was redirected.\n") COLOR RESET ELSE '--- use a fincounter to see how many urls finshed and downloaded QString filename = saveFileName(url) filePath = target_dir + filename IF (saveToDisk(filePath, reply)) THEN COLOR FG TO INTENSE YELLOW PRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT "succeeded! " PRINT "saved as " PRINTOUT(PRINTABLE(filename)) INCR fincounter COLOR RESET END IF END IF END IF IF (dlcounter == fincounter ) THEN COLOR FG TO GREEN PRINT "All downloads finished !" COLOR RESET END END IF END SUB
'-------------------------------------------- FUNCTION isHttpRedirect(QNetworkReply *reply) TYPE bool '-------------------------------------------- 'PRINT "(5)" LOCAL statusCode TYPE int statusCode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt() RETURN (statusCode == 301 || statusCode == 302 || statusCode == 303 || \ statusCode == 305 || statusCode == 307 || statusCode == 308) END FUNCTION
'-------------------------------------------- FUNCTION saveFileName(const QUrl &url) TYPE QString '-------------------------------------------- 'PRINT "(6)" LOCAL path TYPE QString LOCAL basename TYPE QString
path = url.path() basename = QFileInfo(path).fileName()
IF (basename.isEmpty()) THEN basename = "download" ELSE
RETURN basename END IF END FUNCTION
'-------------------------------------------- FUNCTION saveToDisk(const QString &filename, QIODevice *data) TYPE bool '-------------------------------------------- 'PRINT "(7)" QFile file(filename) IF (file.open(QIODevice::WriteOnly)) == false THEN COLOR FG TO INTENSE RED EPRINT "Could not open " PRINTOUT(PRINTABLE(filename)) PRINT " for writing: " PRINTOUT(PRINTABLE(file.errorString())) COLOR RESET RETURN FALSE ELSE file.write(data->readAll()) file.close()
RETURN TRUE END IF END FUNCTION
'---main app = new QCoreApplication(argc, argv) 'special note we use QCoreApplication 'instead of the standard QApplication when its all cli CALL DownloadManager CALL execute app->exec()
|
|
|
Post by alexfish on Sept 8, 2022 22:07:36 GMT 1
Hi Joe
clean compile on RPI4
BR Alex
out of curiosity I left both compile directives un-commented at
PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wno-format-security -Wformat=1
PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5/QtNetwork -fPIC
PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5 PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -fPIC
no problem here , it would be interesting what happens on X86
|
|
|
Post by bigbass on Sept 9, 2022 2:27:43 GMT 1
Hello Alex
yes,it seems to work if both RPI and x86 are un-commented but I wouldnt count on that working always maybe it gets passed as null if it doesn't find the include?
well a minor but important feature auto directory maker if you dont already have the directory '---found a way to mix bacon commands ! QString to char* hack '---MAKEDIR (char*) PRINTABLE(target_dir) QStrings dont mix well with bacon but the hack worked here
Joe
'---VERSION 2.1 Aug 2 2022 cli-downloader '---option -o ~/Downloads/ url url '---must have trailing slash when using the ~/target_dir/ ! '---ported https://code.qt.io/cgit/qt/qtbase.git/tree/examples/network/download/main.cpp?h=5.15 '---and removed class code heavily modified for bacon use by bigbass ' fmt$ = "%s%s\n" fix for mint thanks vovchik ' fixed empty arguments display help ' removed c++ code from bacons PRINT ' handle c++ with PRINTOUT and PRINTABLE ' changed i to start at 2 this fixed empty urls ' added application/x-www-form-urlencoded download are much faster now ' -Wno-format-security tested on mint 21 ' reply warning fixed '---found a way to mix bacon commands ! QString to char* hack '---MAKEDIR (char*) PRINTABLE(target_dir) '--- added default folder to Downloads and fixed the index many improvements
PRAGMA INCLUDE <iostream> <QtCore/QDebug> <QtCore/QDir> <QtCore/QDir> PRAGMA INCLUDE <QtCore/QStringList> <QtCore/QFile> <QtCore/QFileInfo> PRAGMA INCLUDE <QtCore/QMimeDatabase> <QtCore/QMimeType> <QtCore/QTextStream> PRAGMA INCLUDE <QtCore/QUrl> <QtCore/QUrlQuery> <QtNetwork/QNetworkReply> PRAGMA INCLUDE <QtCore/QDateTime> <QtCore/QSize> PRAGMA INCLUDE <QtNetwork/QNetworkAccessManager> <QtNetwork/QNetworkRequest> PRAGMA INCLUDE <cstdio> <QtCore/QTimer> <QtCore/QCoreApplication> PRAGMA INCLUDE <QtCore/QCommandLineParser> <QtCore/QCommandLineOption> PRAGMA LDFLAGS -lQt5Network -lQt5Core -lpthread -latomic PRAGMA COMPILER g++ 'cli-downloader light version for cli only no widget dependencies added
'RPI3 PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wformat=1 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5 PRAGMA OPTIONS -I/usr/include/arm-linux-gnueabihf/qt5/QtNetwork -fPIC
'x86 'PRAGMA OPTIONS -Wno-return-type -Wno-write-strings -Wno-pointer-arith -Wno-format-security -Wformat=1 'PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5 'PRAGMA OPTIONS -I/usr/include/x86_64-linux-gnu/qt5/QtNetwork -fPIC
'depends at least qtbase5-dev libqt5websockets5
OPTION PARSE FALSE
DEF FN PRINTOUT(x) = qDebug(x) DEF FN PRINTABLE(p) = qPrintable(p)
DECLARE app TYPE QCoreApplication* DECLARE manager TYPE QNetworkAccessManager* 'DECLARE reply TYPE QNetworkReply* DECLARE request TYPE QNetworkRequest DECLARE currentDownloads TYPE QList<QNetworkReply *> DECLARE dlcounter = 0 TYPE int DECLARE fincounter = 0 TYPE int
DECLARE target_dir TYPE QString DECLARE filePath TYPE QString DECLARE cli_option TYPE QString
manager = new QNetworkAccessManager()
'had trouble globally seeing and comparing the option cli_option = "-o" '-------------------------------------------- SUB DownloadManager() '-------------------------------------------- '(1) COLOR FG TO INTENSE WHITE 'PRINT "(1) initialize the Download manager" COLOR RESET QObject::connect(manager, &QNetworkAccessManager::finished, downloadFinished) END SUB
'-------------------------------------------- SUB execute() '-------------------------------------------- COLOR FG TO INTENSE WHITE 'PRINT "(2) Process the command line urls" COLOR RESET LOCAL nameofapp TYPE const char* LOCAL str TYPE QString LOCAL arg2 TYPE QString QStringList args = QCoreApplication::instance()->arguments() nameofapp = PRINTABLE(args.at(0).toLocal8Bit()) ' remove the first argument, which is the program's name args.takeFirst() '#NO ARGUMENTS GIVEN IF args.isEmpty() == true THEN COLOR FG TO INTENSE BLUE PRINT "Qt Downloader - downloads a single or multiple URLs by bigbass July 31 2022" PRINT "--------------------------------------------------------" PRINT "Usage: ",nameofapp, " url" PRINT "Downloads the URL or URLs passed in the command-line to Downloads by default" PRINT "If the target file already exists it will get over written with the new download" PRINT "--------------------------------------------------------" PRINT "Usage2: ",nameofapp, " -o ~/Downloads/ url url " PRINT "Must have the trailing slash when using the -o ~/target_dir/" PRINT "Note the target_directory will be created automatically" COLOR RESET END ELSE '#NO -o OPTION GIVEN '---we need to create a default directory when none is given ver 2.1 IF args.at(0).toLocal8Bit() ISNOT cli_option THEN ' you forgot the target directory ' make a default to ~/Downloads/ HOME$ = CHOP$(EXEC$("echo $HOME" )) target_dir = HOME$&"/Downloads/" PRINT "Automatically sent it to the Downloads folder" 'chop extra spaces args.replaceInStrings(QRegExp("^\\s+"),"") END IF '# -o OPTION GIVEN and your target directory also '---option homemade -o parser hack to a target_dir IF args.at(0).toLocal8Bit() == cli_option THEN 'get the second arg then remove it from the list later arg2 = args.at(1).toLocal8Bit() 'copy arg2 to the target_dir delete arg2 from the download list target_dir = arg2 COLOR FG TO INTENSE BLUE PRINT "-o DIR you want the download to go " PRINTOUT(PRINTABLE(target_dir)) '---found a way to mix bacon commands ! MAKEDIR (char*) PRINTABLE(target_dir) COLOR RESET 'remove the option -o args.replaceInStrings(cli_option,"") 'remove the arg2 which is the target_dir args.replaceInStrings(arg2,"") args.replaceInStrings(QRegExp("^\\s+"),"") END IF '#INITIALIZE INDEX 'changed i to 0 ver 2.1 LOCAL Multi_url TYPE QString LOCAL i = 0 TYPE int LOCAL num TYPE int num = args.size() '# PASS the urls to the doDownload function REPEAT Multi_url = args.at(i).toLocal8Bit() IF (Multi_url.isEmpty()) THEN ' do nothing and stay here for a moment if a bad url happens ELSE CALL doDownload(Multi_url) END IF 'PRINTOUT(PRINTABLE(args.at(i).toLocal8Bit())) 'PRINT "i = ",i INCR i UNTIL num == i END IF END SUB
'-------------------------------------------- SUB doDownload(const QUrl &url) '-------------------------------------------- 'PRINT "(3)" IF (url.isEmpty()) THEN 'if you arrive here i needs to be set correctly ELSE QList<QNetworkReply *> currentDownloads QNetworkRequest request(url) 'force a user agent request.setRawHeader( "User-Agent" , "Mozilla Firefox" ) request.setHeader(QNetworkRequest::ContentTypeHeader, "application/x-www-form-urlencoded") QNetworkReply *dlreply = manager->get(request) currentDownloads.append(dlreply) '--- use a dlcounter to see how many urls are used INCR dlcounter END IF END SUB
'-------------------------------------------- SUB downloadFinished(QNetworkReply *reply) '-------------------------------------------- 'PRINT "(4)" LOCAL url TYPE QUrl LOCAL fmt$ TYPE STRING url = reply->url() IF (reply->error()) THEN COLOR FG TO INTENSE WHITE EPRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT " failed: " PRINTOUT(PRINTABLE(reply->errorString())) COLOR RESET ELSE IF (isHttpRedirect(reply)) THEN COLOR FG TO INTENSE BLUE fprintf(stderr, "Request was redirected.\n") COLOR RESET ELSE '--- use a fincounter to see how many urls finshed and downloaded QString filename = saveFileName(url) filePath = target_dir + filename IF (saveToDisk(filePath, reply)) THEN COLOR FG TO INTENSE YELLOW 'PRINT "Download of " PRINTOUT(PRINTABLE(url.toEncoded().constData())) PRINT "succeeded! " 'PRINT "saved as " 'PRINTOUT(PRINTABLE(filename)) INCR fincounter COLOR RESET END IF END IF END IF IF (dlcounter == fincounter ) THEN COLOR FG TO GREEN PRINT "All downloads finished !" COLOR RESET END END IF END SUB
'-------------------------------------------- FUNCTION isHttpRedirect(QNetworkReply *reply) TYPE bool '-------------------------------------------- 'PRINT "(5)" LOCAL statusCode TYPE int statusCode = reply->attribute(QNetworkRequest::HttpStatusCodeAttribute).toInt() RETURN (statusCode == 301 || statusCode == 302 || statusCode == 303 || \ statusCode == 305 || statusCode == 307 || statusCode == 308) END FUNCTION
'-------------------------------------------- FUNCTION saveFileName(const QUrl &url) TYPE QString '-------------------------------------------- 'PRINT "(6)" LOCAL path TYPE QString LOCAL basename TYPE QString path = url.path() basename = QFileInfo(path).fileName()
IF (basename.isEmpty()) THEN basename = "download" ELSE RETURN basename END IF END FUNCTION
'-------------------------------------------- FUNCTION saveToDisk(const QString &filename, QIODevice *data) TYPE bool '-------------------------------------------- 'PRINT "(7)" QFile file(filename) IF (file.open(QIODevice::WriteOnly)) == false THEN COLOR FG TO INTENSE WHITE EPRINT "Could not open " PRINTOUT(PRINTABLE(filename)) PRINT " for writing: " PRINTOUT(PRINTABLE(file.errorString())) COLOR RESET RETURN FALSE ELSE file.write(data->readAll()) file.close() RETURN TRUE END IF END FUNCTION
'---main app = new QCoreApplication(argc, argv) 'special note we use QCoreApplication 'instead of the standard QApplication when its all cli CALL DownloadManager CALL execute app->exec()
|
|
|
Post by bigbass on Sept 10, 2022 0:41:31 GMT 1
Hello I bumped the last post to ---VERSION 2.1 today spent a few hours going over the code and now it automatically downloads to Home Downloads without any options needed if you want to use the -o option target directory thats ok too I had a problem with the index going out of bounds this should be the final version will package it as a deb cleaned up some of the output messages commented the code better also thanks Alex and Vovchik for your valuable feedback! Joe P.S will post a cmake build so that a deb package could be made for the RPI4 sourceforge.net/projects/bassix-gtk3-webkit-browser/files/CMAKE_BUILDS/CMAKE_CLI_DOWNLOADER_2.1.0.tar.gz/download------------------ cmake . cmake --build . cpack ------------------ and that will generate a new debian package
|
|
|
Post by alexfish on Sept 10, 2022 1:53:24 GMT 1
Hi Joe Thanks for the comment.. I remember, some time back, RE Raspberry PI posts ( not sure at which point ) but sure first instance was arm & gtk 1 .. later Peter Mention PI as a curious os , well yes it is yet no more than, here we can sit with an arm device or an x86 device , in short gcc g++ and the directives (Macro Functions) #ifdef blagh else same would apply #include , if the path/to/exists .. then... pi2 was a bit of a dilemma esp omx etc blagh native drivers, check the posts. so to back to directive #if worth reading sourceforge.net/p/predef/wiki/Architectures/BR Alex
|
|
|
Post by bigbass on Sept 10, 2022 17:55:32 GMT 1
|
|
|
Post by vovchik on Sept 10, 2022 18:38:04 GMT 1
Dear Joe, I too the latest source from sourceforge and compiled on Mint and Pi4. It works fine...as advertised. Many thanks. With kind regards, vovchik
|
|