fossil beta with new lexer

[bigbass]

Hello Peter

I know that this is "fossil beta testing" and you are doing many changes still
but just wanted you to know that I am getting a strange error
compiling code that I know and tested that it works correctly using the official stable bacon

BaCon version 4.2 on Linux armv7l - (c) Peter van Eerten - MIT License2020-11-22



--------------------------
using the latest fossil as of
---------------------------
2020-11-22
21:14:45
chiselapp.com/user/bacon/repository/bacon/info/2dfd580ecee5d52c

--------------------------
the bacon log gives this
--------------------------

https://chiselapp.com/user/bacon/repository/bacon/info//usr/bin/ld: bassix-browser-advanced-3.3.bac.o: in function `__b2c__server_init':
bassix-browser-advanced-3.3.bac.c:(.text+0x3424): undefined reference to `__b2c__delim_engine'
/usr/bin/ld: bassix-browser-advanced-3.3.bac.c:(.text+0x3468): undefined reference to `__b2c__delim_engine'
collect2: error: ld returned 1 exit status
make: *** [Makefile.bacon:4: bassix-browser-advanced-3.3] Error 1




the code example is here that compiles and runs correctly with the stable bacon but fails with fossil

sourceforge.net/projects/bassix-gtk3-webkit-browser/files/bassix-browser-advanced-3.3.tar.gz/download



I would be happy to provide logs or test anything
on my side if you have a suggestion

its not an emergency just some feedback you should know
how things are going on a different box
I do have flex installed (from apt-get )

and compiled other examples without any problems
a clue "seems it chokes" on this function __b2c__server_init
but that's as far as I got... its one very long unformatted line of code



If we can catch some problems in fossil beta
They won't pass into the stable branch


Joe

[Pjot]

Hi Joe,

Thanks for testing and your report! Indeed you have found an issue in the latest beta, but I have fixed it already.

If you fetch the latest version from fossil then you'll see that your program compiles now.

The reason for this issue is that the dependency check has become more thorough. Previously, M4 would scan both the BaCon source code and also the generated C sources. In some cases, this would add more functions to the resulting binary than necessary, resulting in a bigger binary.

With flex/lex, we can precisely define which parts of the original BaCon source needs to be scanned. And we can skip the generated C code.

So the lexer found the NETWORK keyword and added the corresponding C code. However, the C code for NETWORK also makes use of the delimited string engine, so this dependency had to be added as well.

Please feel free to report more issues if you run into them,

Thanks again
Peter

[bigbass]

Hello Peter

Thanks for the information on how the new checks work
I see how this fits into the plan to reduce
extra code from getting compiled into the bacon binary

also thanks for checking and fixing the fossil code
to compile the app I linked it compiles and runs correctly
with the latest fossil (this is really good news to me )
because I used a lot of macros and was worried they may crash and burn
with the lexer but all is fine


one question I already have flex installed will that become a required dependency
for new bacon installs?


thanks again
Joe




another note
the raspberrypi3 can build from official bacon source code
but to save time and make life easier for new or lazy bacon users
I ran your deb builder script and generated a raspberrypi3 stable package

and uploaded it here (I made no personal changes to the stable package)
============================================================
bacon does not have a registered package in apt-get but
You can manually install it in your system anytime using:


dependencies needed for bacon (dpkg will not auto install deps)
sudo apt-get install enscript xterm indent m4


sudo dpkg -i bacon_4.2-1_armhf.deb

The deb package generated from the script for the raspberry pi3 is here if anyone needs it 32 BIT
sourceforge.net/projects/bassix-gtk3-webkit-browser/files/bacon-raspberrypi3/bacon_4.2-1_armhf.deb/download

[Pjot]

Nov 25, 2020 18:13:29 GMT 1 bigbass said:

one question I already have flex installed will that become a required dependency for new bacon installs?

Well, in fact yes, so M4 will be replaced for flex (GNU) or lex (POSIX). The M4 macro processor has some unexpected side effects for the purpose of tokenization.

For example, if you would use a variable with the name 'len' in your code, then M4 will take this as an internal macro, and try to execute it as the 'len' macro. There are ways to get around this (like the -P option adding the "m4_" prefix) but then still there's a chance such situation happens. Besides, the -P option is not available in all M4 implementations.

Another downside is that M4 cannot skip comments or text between double quotes. We can use 'changecom' but what if a text contains an escaped double quote? M4 has no way to skip an escaped double quote. So it would end up taking part of the text and see unbalanced quotes, generating an error.

Furthermore, M4 cannot recognize keywords ending with a '$'. So none of the BaCon string functions can be recognized. The workaround is to let M4 parse all source files and generated C files. However, this also will cause M4 messing up the BaCon implementation of BaCon and adding a lot more stuff than necessary.

By using (f)lex, we can solve all these issues at once. Not surprisingly, because (f)lex has been designed especially for tokenization. It has a very long history (since end of the 80's), and is available everywhere, even on Android systems.

So for your installer, the new line will then be:

sudo apt-get install enscript xterm indent flex


Best regards
Peter

[bigbass]

Hello Peter

after taking a deeper look at flex with bacon
It produces cleaner code that will be easier to run compile and debug

thanks for your constant improvements
and risk taking adapting all your code to use another parser
because you want what's best even if it will cause you a lot of new work setting it all up

everything is looking really good as is !
thanks again for the m4 to to (f)lex info
I believe there will be no going back from here
(f)lex is here to stay and for the better!



Joe

[bigbass]

Not something new but there is still room
shaving the binary size
a low cost improvement for binary size
without any code changes or new deps is add

strip --strip-unneeded yourBaconApp


that gave me a 27.3% reduction in size

from 68.4k to 53.9k

tested on this file
# reduce binary size
strip --strip-unneeded bassix-browser-advanced-3.3



Joe

[bigbass]

Hello Peter
on my side I tested this to be working on the raspberry pi3
part of the binutils package so no new deps



adding only this one line to bacon.bac

WRITELN TAB$(1), "strip --strip-unneeded " , BASENAME$(g_SOURCEFILE$, 1) TO makefile



this code starts at line# 9500

' Creation of the Makefile
OPEN g_TEMPDIR$ & "/Makefile.bacon" FOR WRITING AS makefile
WRITELN "# Created with BaCon ", g_VERSION$, " - (c) Peter van Eerten - MIT License", NL$ TO makefile
WRITELN BASENAME$(g_SOURCEFILE$, 1), g_BINEXT$, ": ", BASENAME$(g_SOURCEFILE$), ".o" TO makefile
WRITELN TAB$(1), g_CCNAME$, " -o ", BASENAME$(g_SOURCEFILE$, 1), g_BINEXT$, " ", BASENAME$(g_SOURCEFILE$), ".o ", g_LDFLAGS$, " ", g_LIB_TLS$, " -lm" TO makefile
WRITELN TAB$(1), "strip --strip-unneeded " , BASENAME$(g_SOURCEFILE$, 1) TO makefile
WRITELN BASENAME$(g_SOURCEFILE$), ".o: ", BASENAME$(g_SOURCEFILE$), ".c" TO makefile
WRITELN TAB$(1), g_CCNAME$, " ", g_CCFLAGS$, " -c ", BASENAME$(g_SOURCEFILE$), ".c" TO makefile
WRITELN NL$, ".PHONY: clean" TO makefile
WRITELN "clean:" TO makefile




automatically
that gave me a 27.3% reduction in size


easy to add
smaller bacon binary


Joe

[vovchik]

Dear Joe,

Thanks for that little mod. When I compile bacon code, I always add the following:


bacon -l -no-pie -o -fno-pie -o -Wno-implicit -o -Os -o -fdata-sections -o -ffunction-sections -o -Wl,--gc-sections -l -s $myfile

and then pass the resulting binary through strip:

strip --strip-unneeded unstripped-bacon-binary

If I am particulary worried about size, I also do this:

upx -9 stripped-bacon-binary

We get a huge reduction using both strip and upx...

With kind regards,
vovchik

PS. I think a -S during linking can also do the strip job, but I have to check.

[Pjot]

Thanks Joe,

When trying at BaCon itself it doesn't seem to do much on my system. I mean, when installing BaCon from the source package, it is compiled using the '-O2' flag and then stripped. On my system, the resulting binary size is 740280 bytes.

And using your 'strip' command and parameter results to the same binary size of 740280 bytes.

But during linking, you can strip redundant (debug) symbols using the '-s' option for gcc as well:

# bacon -l -s program.bac


We can compile optimizing for size, and strip during linking in one swoop:

# bacon -l -s -o -Os program.bac


Even on my system these parameters do generate a smaller size. My BaCon binary now only has 646048 bytes.

The command line of vovchik takes it to the extreme, though I have seen mixed results. For me, on some systems, such extreme size reduction caused the binary to malfunction. But I never really investigated the reason for this...

Best regards
Peter

[bigbass]

Thanks vovchik for the smallest binary options
when we have to squeeze and compress


Thanks Peter your command line is very practical
with out source code modifications needed
and produces small binaries


I prefer clang and clang++
for error messages

the latest is called
clang-9
clang++-9

this works fine in general terms
PRAGMA COMPILER clang++-9
PRAGMA OPTIONS -Wno-write-strings -Wno-pointer-arith -Wno-deprecated



I wasn't thinking about compiling the bacon.bac (to make it smaller)
I was thinking about the resulting binaries I make on my side
However after using clang it said that -l -s are not used
so I shortened it for me


the bacon.bac modified code in bold text

' Creation of the Makefile
OPEN g_TEMPDIR$ & "/Makefile.bacon" FOR WRITING AS makefile
WRITELN "# Created with BaCon ", g_VERSION$, " - (c) Peter van Eerten - MIT License", NL$ TO makefile
WRITELN BASENAME$(g_SOURCEFILE$, 1), g_BINEXT$, ": ", BASENAME$(g_SOURCEFILE$), ".o" TO makefile
WRITELN TAB$(1), g_CCNAME$, " -o ", BASENAME$(g_SOURCEFILE$, 1), g_BINEXT$, " ", BASENAME$(g_SOURCEFILE$), ".o ", g_LDFLAGS$, " ", g_LIB_TLS$, " -lm" TO makefile
WRITELN TAB$(1), "strip --strip-unneeded " , BASENAME$(g_SOURCEFILE$, 1) TO makefile
WRITELN BASENAME$(g_SOURCEFILE$), ".o: ", BASENAME$(g_SOURCEFILE$), ".c" TO makefile
WRITELN TAB$(1), g_CCNAME$, " ", g_CCFLAGS$, " -c -Os ", BASENAME$(g_SOURCEFILE$), ".c" TO makefile
WRITELN NL$, ".PHONY: clean" TO makefile
WRITELN "clean:" TO makefile
WRITELN TAB$(1), "rm -f *.o *.c ", BASENAME$(g_SOURCEFILE$, 1), g_BINEXT$, " ", BASENAME$(g_SOURCEFILE$, 1), ".*.* Makefile.bacon" TO makefile
CLOSE FILE makefile
g_TMP_FILES$ = g_TMP_FILES$ & " " & g_TEMPDIR$ & "/Makefile.bacon"



and I get now
577.9kb for bacon.bac compiled
and no problems with clang or clang++

and automatically small binaries
for all programs that are compiled

Joe

BaCon version 4.3 on Linux armv7l - (c) Peter van Eerten - MIT License.