andrewchambers (andrewchambers) it might be useful to let users drop whatever modules they want in there to create a custom janet
I'm planning on something like this, a customized build framework I want to use for embedding Janet in my text editor I want it to be smart about only including what is needed and allow custom boot.janet if you want
I already have a replacement for the makefile that compiles the boot interpreter, runs boot.janet, compiles janet.c, and links it to the mainclient.c, so it shouldn't be too hard to make the process customizable
I can allow replacement of boot.c and/or boot.janet
the framework will cater to embedding the Janet core library (or a variant of it) alongside exact versions of libraries (not fetched with jpm, something with a fixed hash like a tarball, or a specific commit of a git submodule)
so I'll probably replace boot.c with my own implementation that is specialized for embedding a fine-tuned set of Janet modules
although intended to be used for Zig language bindings, by nature of the Zig build system it will support C projects as well (I do perform the full Janet bootstrapping process with this build system, after all)
I'm considering something that reads project.janet or some variant of it so you don't even need to write Zig code to customize the build process; you'll be able to use pure Janet code executed by my Zig impl of boot.c
also even if you don't want to customize anything, rebuilds are faster than the Makefile as it doesn't recompile C files unless headers, source files, or compiler version changes
in theory Makefile shouldn't do a full rebuild every time, but in practice it did in my initial attempt to make a script to automatically install the latest Janet release by compiling from source (so instead I only run make if the Janet executable is missing or I pull a new tag of Janet)
if you're curious, I build from source as I prefer installing dev tools in ~/.local so I don't have to compile/install modules as root, and I don't want to add yet another environment variable to ~/.profile (hence, I need to change PREFIX at build time to control the value of (dyn :syspath) in the marshaled Janet image used by the interpreter)
also the jpm shebang doesn't work on one of my systems as /usr does not exist, so I patch that to hardcode the shebang to use full path of Janet executable rather than relying on env, and my project.janet files copy jpm's shebang in the install step so I don't need to patch every module I create (patching the shebang is easy as running jpm guarantees that ((dyn :args) 0) is the full path to jpm, so I don't need to reimplement which in project.janet)
thanks for coming to my TED talk 🙂
Lue thanks for coming to my TED talk 🙂
Lololol
Sounds like that will be a handy project for zig folks
It could be handy for C folks too. Heck, I could release a standalone alternative to JPM that does not require a separately installed Zig or C compiler if I wanted.
doesn't seem worth the effort though
uvtc (John Gabriele) Regarding a batteries-included version of Janet... I remember a while back the Scheme community began its endeavor to spec Large and Small versions (R7RS-Large/Small). FWICT, it seems to work better to instead have some community-centered way to recommend modules, maybe including download counts and "qwalitee" checks (has-readme?, has-tests?, etc.) ... some kind of heuristics to help answer the new user's question, "Which module(s) would be good to use for ${x}?".
I don't mind having a semi-fluid list of Janet modules to install after a fresh Janet install, but it would be nice if there were an easy way to locate that nice list of module recommendations.
I see we already have pkgs (which is curated (is that the right word?)), and Awesome Janet.
Aside, I don't understand the discussion about having jpm be a module to install. I'm probably misunderstanding this ... it seems to be great as its own separate program.
It is very true.
Even the static generated list from the pkgs would be nice. But my current problem is that I have my own solution for this kind of work, which is obscure, yet I have very little motivation to use anything else :). So if bakpakin have nothing against me writting it ala https://sr.ht/~pepe/good-place/ I think we can have something till the end of the week, if someone helps me with setting up generation thru GH that is If not, which I could understand I will be happy to help.
Lue interesting idea, I have considered allowing a custom boot.janet (or custom parts) but I'm not sure there is really that much benefit in in a custom boot.c. The only thing boot.c does is run boot.janet. As for the custom shebang line thing you mentioned, the jpm installation procedure for Janet scripts does exactly as you mention with the shebang line hack
bakpakin Lue interesting idea, I have considered allowing a custom boot.janet (or custom parts) but I'm not sure there is really that much benefit in in a custom boot.c. The only thing boot.c does is run boot.janet. As for the custom shebang line thing you mentioned, the jpm installation procedure for Janet scripts does exactly as you mention with the shebang line hack
oh it does? I didn't see that when I did declare-binscript
for me it kept the /usr/bin/env which is why I did my own install step
Well I remember implementing that somewhere, perhaps it was just for the jpm script itself
If not, that would be useful functionality for jpm
You might remember it hardcoding syspath, modpath, etc. Which the Makefile does indeed do for jpm.
but I'm pretty sure the Makefile doesn't patch the shebang itself. and jpm's declare-binscript's :hardcode option simply adds a line to set the root env's syspath, it doesn't hardcode the shebang. It would be nice if it would set the shebang to the installed Janet interpreter.
Yes, that is likely the case
that should be quite easy to add, and if needed we can always add an option to disable adding an automatic shebang line
yeah :)
it was fairly easy in my own project.janet and it'd be even easier if it was natively supported by jpm. Maybe in tools/patch-jpm, add the path to where Janet will be installed within prefix so that scripts don't need to scrape ((dyn :args) 0) to copy the shebang
this would also let it be controlled by environment variable like the other paths, to allow for temporary override of shebang similar to modpath and whatnot
maybe JANET_SHEBANG so ppl don't accidentally rely on it as an exact path to the Janet interpreter (some users might want to keep it as /usr/bin/env janet)
Perhaps the best default is /usr/bin/env janet unless specifically overriden on Janet build? A shortcoming of hardcoded shebang is that you can't have spaces in the path, which could be an issue in environments such as cygwin.
I'd just like it to be configurable within Janet or JPM's build options rather than needing a custom install script to do it :)
@andrewchambers: Not sure if you have thoughts about the janet-lang/janet#656. I wanted to use it more for discussion than as a means to add path (although, I do think having that be part of core would be useful).
hmm im not sure - I am pretty happy just installing stuff with jpm
the biggest pain point is when you need stuff in the project.janet file
Yeah, I'm honestly not sure either. But I spent some time investigating it and wanted to show my work 😛
oh wait i thought this was an old issue
one sec
let me read it properly lol
@andrewchambers: Well, this relates to the issue.
It's just an example implementation to show how it could work in practice.
right
im pretty against this just because it increases the size of boot.janet
i suspect if you time how long a noop janet program takes this will increase it a fair amount
actually i can test it
Well, you mean boot.c, right? It specifically doesn't increase the size of boot.janet.
Actually, janet.c to be precise.
Message deleted
I don't see how it would increase it that much. The bootstrapping happens during compilation and then it's just part of the byte array that's at the end of janet.c but maybe I'm wrong.
yes you are right
that byte array gets unmarshaled every time janet runs for every program
one sec, im doing a benchmark
Thank you. Am genuinely curious :)
it adds slightly below 1 percent overhead
at least on my laptop in start time
and it allocates 60 extra kilobytes
Hmmm. I guess you can argue that if you keep doing this it's death by a thousand cuts. Although the counter would be that if you're really worried about this kind of thing, you can compile Janet and just disable it (which you can't do with things that are in boot.janet by contrast).
which is closer to about 8 percent
yes
also janet natives exes can disable pure janet code from boot.janet now
if its not used
Would that benefit this kind of thing as well? Since the module is pure Janet.
yes
Interesting.
theres another way too
which is including it in the binary but not importing it by default
so it takes space in the .exe but isn't loaded unless you import it
with a module loader that is capable of reading bytecode from a c variable or something
So add it as a separate byte array and then have import somehow check for that?
yes
not sure its a good idea or not, but its one way
That's an interesting approach. And the benefit of that is that you reduce the impact on unmarshalling?
It would still take up memory for the byte array, right?
yes, but its memory that doesn't affect the janet garbage collector and is processed only once by the OS
and can be paged out
I guess that would be a way to ship a 'standard library' that's part of the standard Janet binary but without much of an impact on start-up time/performance.
it might be useful to let users drop whatever modules they want in there to create a custom janet
or custom amalgamation
i dunno
my main concerns are two things
i don't want the minimum janet environment people expect to be bloated
and i also don't want to pay for features im not using
in terms of runtime cost
i agree path is a pretty decent module
Yeah, I share those concerns.
also an alternative is to just have make install just copy path.janet
along with a jpm manifest
and the same for the windows installer
then its just as if the user had run jpm install path
without them needing to
or all of spork
if you want want it you can run jpm uninstall spork
@andrewchambers: I reimplemented the pull request to use the 'standard library' approach. That is, the path module is marshalled into a byte array but it's not loaded into the core environment by default.
I'm pretty clueless when it comes to how the marshall and unmarshalling functions are supposed to work so I might have made a catastrophic mistake (the test suite passes).
@andrewchambers: I haven't tried benchmarking start-up times, but this approach results in a significantly larger binary than loading the module into the core environment during bootstrapping. On my system, current master Janet is 655KB, 'core modules' approach is 667KB and 'standard library' approach is 851KB. Not sure if I'm writing more to the byte array than is strictly required.
There is also the fact that the way path is implemented right now assumes that it can determine the OS at load time, and then dispatches to a different set of functions depending on which OS it detects. I think it's just POSIX vs Windows for now?
Which means that it will carry a little extra bloat there as well
Yeah, I'm not surprised there's a difference between current master and the 'core modules' approach but I am surprised how much bigger the 'standard library' approach is.
I assume I'm probably marshalling a copy of the core environment but am not sure if that's a limitation of trying to keep the environment separate or if there's some way I can avoid that overhead.
Yo
@pyrmont i guess some stuff is duplicated
btw, what is the problem with preinstalling jpm modules btw?
did you see that idea?
I don't think I followed that one.
Oh, wait, I see it now. Sorry >_<
If I'm understanding it correctly, I think one problem is it'll break if you want to put your modules separately from the 'standard library' modules.
Since jpm only really understands modules being in one location.
why does it break? they just need to jpm install them again
@pyrmont-545dcc43db8155e6700d2a64:gitter.im: I think that bakpakin stated couple times, that path should stay in spork. What is your reasons to put it in the core?
I have read the discussion above, but somehow I could not find the reason (maybe cognitive deficiency on my side)
@damnpepe: The OP explains that the purpose of the PR is really just for discussion. I used path because it's relatively simple and arguably the kind of module that you might want in a standard library (but not in core).
For instance, there's bits of jpm that would be cleaner if it had access to the path module.
OH, the PR I am gonna read it now. Sorry
@andrewchambers: I'm not sure I follow. The point of a standard library is that you don't need to separately install the elements of the library.
That said, I agree that if there's no way to avoid boating the binary by as much as this 'standard library' approach is doing then it's a non-starter.
Now I get it. Still I do not think it is good idea. One thing I like about Janet is the compactness. More code always means more bugs and more support. I am fine with the spork as this kind of stdlib
what if the default janet install also installed spork as a jpm module
that is approach I would actually like a lot andrewchambers (andrewchambers)
In my generator, there is the default deps with spork :)
@damnpepe: Yeah, I think it depends on how you want to approach things. I like the batteries-included approach but I concede it has trade-offs and those might not be worth taking.
I mainly wanted to experiment with what was possible.
yes experiments are good
Indeed. I do not want to say it is bad point of view, not at all, I am glad you do this experiments
there is not experience without the experience
OT yesterday when I was coding my data structures exploring/managing library I had that moment, where I wrote a code and it did what I wanted on the first try, and then something more unexpected, yet better than my idea was. Just magic
All of this was precipitated by @subsetpark wanting a way to use the functions that are defined in jpm from an external script. That's not possible at the moment and I think would require the functions defined in jpm to be pulled out into a module. If that module isn't included with the default Janet installation, then there are problems but including it causes its own problems.
yeah, I think the discussion is even older with Janet
the stdlib approach BTW is clever
I tried to open a discussion about this (in #652) but it was closed and I think would have benefited from something more practical to ground the discussion. Hence, #656.
Thanks! It's really @andrewchambers riff on the original idea but I thought it was clever, too, and wanted to see if you could make it work :)
So, why not split JPM into a module for the core, and a script that does a relative import on the module, but provides a self-update command?
Just another (possible) data point: Last time I built+installed Janet (Debian), one of the last lines after the make install had a commented-out rm -fr /usr/local/lib/janet comment pointing out that it might be desirable to do so. Since seeing that, I make a habit out of rm'ing that directory after upgrading+reinstalling Janet. It's not much trouble at the moment because all I have in there is spork and jaylib that gets reinstalled each time.
@yumaikas: I'm not sure I follow the suggestion.
Then the module could be installed (maybe as a small special case, or maybe the process of building Janet clones down jpm from a Known Place, and jpm is in pkgs?)
So, as I understand it, the crux of the issue with JPM is that it's one big script, built in a way that makes it hard to re-use?
Since it doesn't have a clean "use this URL to get jpm", and/or has a bit of a different execution mode?
@yumaikas: Well, I'm not sure there is an issue. As I said, this was precipitated by the 'jpm as module?' question but (to my mind at least) there is a broader question about the best way to add functionality to Janet over time.
Since I agree that just stuffing things into boot.janet has downsides.
So, I think that it's useful to think of jpm as a separate entity from Janet, imo
And maybe we should put a tiny bit more of a wall there? Idk
I don't know. I like the fact that Janet comes together with a package manager that you can rely on being there.
Right, I'm not saying to get rid of that property of Janet
Not sure I follow what more of a wall there could be, then. jpm is a completely separate script at the moment.
If anything, I think we're discussing there being less of a wall.
But that if JPM had a some attention given to it, it could probably bootstrap itself into an installable module.
Since it is the tool being used to install modules
To be clear, the PR doesn't touch on jpm directly. It's intended as a proof of concept of how you could incorporate modules written in Janet into the default installation in a way that limits the impact on those who don't want to use them. I use the path module as an example but it's really just for illustrative purposes.
So, I don't think path belongs in core myself? I think something in the direction of "install spork to the lib dir by default in Janet installs" makes more sense to me?
Idk
It depends on how many batteries we want in core vs spork
But, like, the fact that Janet can run in WASM, or on the Nintendo Switch, or potentially on (beefy) microcontrollers means that Janet should bias towards making it easy to add bits like this on, but away from including them in core, IMO.
We could have a "baterries-included" distribution of Janet (akin to Strawberry Perl or ActiveState TCL) that has Spork and/or PKGs locally cloned by default, in the installation download
(basically make a Thumbdrive friendly Janet, IMO)
That is also bright idea
Kinda like how we have the Janet C amalgamation in a tarball
I think that if we want to go "baterries-included", that might be a better direction for Janet. Could even add extra C or C++ modules to it, and make it a separate repo that uses core Janet as a submodule. (That might be going too far?)
But, if we do something like that's it'd work best with the blessing of the Janet org
if jpm itself was a jpm package
And not just Yet Another User Universe
and that jpm package came preinstalled
??? Ah
yes?
well, I'm just saying i have less of a problem shipping batteries
if they are just normal jpm packages and jpm itself could be one of those batteries
that is to say, they aren't magic blessed packages, just normal jpm packages that come with the janet installer
to make a virtual env or something you still need to add them to your project.janet deps
andrewchambers (andrewchambers): that makes very good sense to me
That's the direction I'd lean, IMO. And, like I said, build a distribution that makes it easy to bundle a bunch of deps into an installation would be nice too.
yumaikas-: oh yes good-place could be one of these big distributions. Thanks!
Here's the real question: what can't we ship via a JPM module, and/or, what is the best way to avoid the Haskell problem that people are recommending an alternative prelude?
@pp you do more project generator work of late?
Well I continue with Michael for the good place spcific projects
as I intended from the very beginning
What is your good place project for? It's been a while, and I've been busy outside of Janet, lol
From the upcomming web page: Create a free and open collection of code in Janet programming language.
I know it little too general, yet ...
So it's a tool for generating Janet distributions? Or is it more focused than that?
No at all it is collection of my libraries and programs
For areas I have found I am operating when coding my projects
With only connection being naming based on the Good Place TV series, which is small experiment on my side
Ah, kinda like my collection of x-libraries? But with a good branding/landing page?
well that we will see. And more high level abstractions I guess
And links up for perusal?
Here is the sourcehut project https://sr.ht/~pepe/good-place/sources. But it all need more polishing that is sure. But I think I see the light on the end of the tunnel
Regarding a batteries-included version of Janet... I remember a while back the Scheme community began its endeavor to spec Large and Small versions (R7RS-Large/Small). FWICT, it seems to work better to instead have some community-centered way to recommend modules, maybe including download counts and "qwalitee" checks (has-readme?, has-tests?, etc.) ... some kind of heuristics to help answer the new user's question, "Which module(s) would be good to use for ${x}?".
I don't mind having a semi-fluid list of Janet modules to install after a fresh Janet install, but it would be nice if there were an easy way to locate that nice list of module recommendations.
I see we already have pkgs (which is curated (is that the right word?)), and Awesome Janet.
Aside, I don't understand the discussion about having jpm be a module to install. I'm probably misunderstanding this ... it seems to be great as its own separate program.
JPM as a module is more for helping other people write things like project file parsers, linters and such, and would, for example, help make a more automated package registry be easier to pull off
Ah. Thanks, yumaikas- .