New Worlds And Lispy Solutions
Lispy
DISCLAIMER: I've just looked very briefly at the code of Lispy!
Matthew Kennedy's Lispy aims to be simple and easy to use and is based on concepts which are proven to work for Linux distributions. Lispy is not really centrally organized, but it doesn't seem to emphasize distributed facilities very much. It seems to be quite easy though to setup a Lispy repository - all you need is some web space and a mapping file which stores metadata about all provided packages. As far as I understood it, versions are handled as chronological markers and there is a distinction between versions which come from package authors and versions which are managed by the Lispy system. There are MD5 sums of the asdf-system tarballs to enable efficient ways for verification of integrity and authentication. Matthew's design is based on his experience from managing Lisp packages for the Gentoo portage project and it actually is designed as a system to manage package distributions. In regard to my find/acquire/verify/authenticate quadruple he decided not to conflate loading of packages and installation of packages into one operation (acquire); this is maybe indeed the better thing to do. I didn't understand really who is responsible for the creation of the mapping-files; particularly where the :depends-on information is coming from. My opinion about Lispy is, that it looks very well designed. On the first sight it seems to be more suited for package maintainers than for developers. That may sound a bit weird, what I actually mean is that it seems to be more suited to manage released versions of a package. What should I do if I need a particular (perhaps locally patched) version of a package for only a particular project? If I understand Lispy right, then I would define a repository (mapping-file) for each such project. It also doesn't seem to support fetching packages out of scm repositories (cvs, darcs, svn...) and defines asdf-packages as tarballs. I still think the MD5/SHA digest should be computed from a defined order of the files of a source package and not from a tarball. Perhaps there could be an ASDF operation which computes the digest instead of compiling the files - so the digest would be defined only by the files relevant to the ASDF-System.
Worlds Apart
To me a world in the common lisp system management
context is a logical bundle of systems used in a
particular project or setting. It defines the context
needed for development and deployment. A
system-version could be uniquely identified by its
system-name and a digest from the defined order of
the content files. This would also mean, that world
is actually defined by the a set of particular
versions of asdf-packages. If one changes one of the
packages it is not the same world anymore. This is
similar to the concept of identity in git (see e. g.
Nikodemus Siivola's explanation) Gary's command-set for
working with worlds (he called them
"environments") looks a bit to procedural to me. I
would prefer a more declarative syntax. So instead
of writing CREATE-WORLD and CLONE-WORLD as a user
I would rather use a DEFINE-WORLD macro which
defines the dependencies of a world. A world
should know anything needed to upgrade the systems
if needed. So if a system in my world is defined
to track the changes of a particular branch in a
scm system - so be it. I think worlds could be
quite similar to the shelfs of cl-librarian. The main difference
between "worlds" and "mapping repositories" (like
in lispy) is that the first is a set of
asdf-systems from the standpoint of a asdf-package
user and the latter is from the standpoint of a
package-publisher. The only reason to publish a
world definition is actually to readily deployable
(batteries included) projects. So the packages
within a world should in some way depend on each
other while the packages within a mapping
repository (a "distribution") may have the only
common point that they are maintained by the same
person.
ASDF Load Syntax Sucks?
An often raised complaint against ASDF is the quite unfriendly syntax of one of its most basic operations: Loading a system.
(asdf:operate 'asdf:load-op
:my-system)
I think this can not really be an issue, can it? How often does one actually have to type this thing? Systems are typically loaded as dependents of other systems. So I wouldn't actually use ASDF:OPERATE! I just type the system-name in the :depends-on clause of my system definition.
Interactive Development!
The sole exception to this is interactive development. It is quite common to load a system to try something out without officially adding it as an dependency to your system. Perhaps you do not even have a system for your code yet. The point here is: Loading a system while doing interactive development does not really need to be by directly using the programmatic interface of the ASDF-library; I'm using an editor command within the LispWorks IDE to interactively load a system. This even provides me with name-completion for asdf-systems. Writing something similar for slime should be trivial (if it doesn't exist yet).
Why Not Colon Commands?
A recent discussion on comp.LANg.lisp about colon commands on the lisp REPL showed that most common lisp implementations provide such a REPL command feature; why not just define an :asdf, :load-asdf or :la colon-command?
Ideas for Asdf and Asdf-install
1. It doesn't play well with Windows
Gary's first argument is that asdf-install doesn't work well with Windows, because of it's use of Unix utilities for tasks like archiving or signing/verifying. I agree that this is an issue, but nothing that couldn't be solved by just using ports of those utilities for windows.
2. It doesn't play well with itself
The second argument just describe some obvious short-comings of asdf-install. It doesn't handle updating your libraries in any way and it can only download and install tarballs. Gary is right here too; asdf-install is just not enough.
What do I think?
Gary is a smart guy and definitely is on the right track with his idea. I do not agree is his idea to make asdf-install better, but only because I think that asdf-install isn't a good base for anything. I think managing asdf-packages is too important to be left outside of lisp. I think there are enough high-quality libraries to handle the necessary tasks in common lisp and without airy callouts to shell utilities.
Well... but asdf-install is written in lisp!
Technically yes, but if you look into other general purpose lisp-libraries, you will see that a good lisp library should be flexible enough to adapt to many different use cases. A good lisp package manager needs to be extensible in at least four directions:
- How to find packages (cliki, cl-user.net...)
- How to acquire packages (e. g. tarball, zip, CVS, Darcs, SVN, git, e-mail, local folder)
- How to verify integrity of packages
- How to authenticate the identity of the supplier of a package
Finding a package should be a decentralized task. It should be possible to easily setup your own package registry. There could be different technologies used, but I think there should be at least a web-based one which doesn't demand more than some static web space. An interesting point here is, that finding a package might not necessarily be a proactive step. Maybe one could register to some typical package feeds, and as soon as a new version is announced, it will be acquired by the system. If the protocols of this component are done right, than there is no perceivable difference in finding a package somewhere in the internet or already installed on your harddrive. There should be no need for symlinks or *central-registry* hacking. The installed packages happen just to be cached on your harddrive
There is nothing special behind the technologies to acquire the packages. One important thing is the change of notion for an asdf-package. Up to now it was actually a tarball with a certain structure. This does not hold anymore if we want to acquire packages through other channels like version control systems. Maybe an asdf-package should defined as the decompressed folder. The handling of a compressed asdf-package seems obvious to me, regardless of it being archived using tar/gz, zip or whatever.
The most obvious way to verify the integrity of an asdf-package seems to be a digest algorithm like MD5 or SHA. But if we define asdf-packages to be just folders and not tarballs anymore, we have the problem, that there is no obvious way how to compute the digest (because there is no defined order of its components). One idea may be to include a manifest-file with each asdf-package and use the order defined by this manifest to compute the digest; there may certainly be different ideas. A manifest-file could be useful for other things too though. It should be a very easy one step action to bless a package as released and create the necessary stuff like the digest and perhaps the manifest.
I've not thought very much about authentication. There are some evident solutions possible; one of it actually used already by asdf-install. The crux is to find a way which will actually be used, instead of just switched off by most users.
The package manager should understand different versions of a package. In the past the version-info was always encoded in the system-definition. I'm not sure if this is the right place for it. Regardless of this; the package manager has to handle packages with and without versions. The version-number is probably a good idea to order the different package-versions by age. But time is only one dimension in which different versions of a package could be placed. I think there should be something like a unique identifier for each version of a package and the best candidate for this would be the digest for the concrete asdf-package.
The package manager needs to handle dependencies between asdf-packages in a better way than just trying to load it and acquire the package on fail (like it is done in asdf-install). Actually this approach seems caused by a misfeature of asdf itself; there doesn't seem to be a way to ask an asdf-system for its dependencies.
My wishlist in very short: A new package-manager for asdf should:- be written in common lisp
- be able to use different package sources
- be written in common lisp
- not mandate airy-fary use of OS-dependent stuff like symlinks
- be written in common lisp
- neither makes downloading nor finding packages a centralized task
- be written in common lisp
- have well-defined CLOS protocols for future extension
- and of course: It should be written in plain common lisp!
Thanks for your time! I really want to hear what you think about this ideas.
The Darkside is Here
For most of my projects I'm using the distributed revision control system darcs. I particularly like how easy and streamlined it is. Creating repositories, branches or pushing single patches around is just a matter of a few minutes in learning time. Another nice feature is, that you can commit only parts of the changes in a single file; darcs will just ask you about each change. Choosing the changes one wants to record on the command line can be a tedious process though. This is why I wrote us an internal tool called "Darkside" which allows us to easily pick all changes using a straightforward dialog. The following screenshot shows the "Patch Recorder" tool of Darkside. An editor pane below the change-tree shows which lines were removed (red) and which were added (green). Darkside is written in LispWorks using CAPI.
The next screenshot show the second tab-pane of the interface. Here one can write the log-message for the current record. I extended darcs' notion of a patch name a little bit by providing a combo-box with certain tags like "Feature", "Change", "Bugfix" and so on. The selected tag gets enclosed in brackets and then prepended before the patch-name. This is actually just a convention we developed using darcs. The list-pane below the log-editor shows other patches which have been recorded before. It is possible to double-click those lines to paste the log-text of this patch into the log-editor.
CL-Darcs: A Common Lisp Implementation
The original darcs (which is used in Darkside too) is written in Haskell, a statically typed functional programming language. The installation is simple: just copy the executable somewhere into your path. Wouldn't it be nice to have an implementation of darcs in common lisp? Well, we have luck! Thanks to Magnus Henoch's cl-darcs there is already a common lisp library which aims to be compatible with the original darcs implementation. The code is nicely structured and good readable. One really missing thing for a cl-darcs to ensure compatibility to the original darcs is an extensive test-suite. But Magnus implements the command-line interface of darcs on top of the common lisp API so that one can use the official darcs test suite. I think cl-darcs is a very promising project.
Server completely rebuilt
Steve is Not an Acceptable Lisper
Steve said in his introduction:
I guarantee you there isn't one single Lisp programmer out there who uses exclusively Lisp. Instead we spend our time hacking around its inadequacies, often in other languages.
Well I've used Lisp almost exclusively in the last 6 years. I do not think it has to be this way - I have absolutely no problem using another programming language if it fits better to the demands of my tasks. I also learned many other languages in this six years; but for most problems that came up I did use Lisp and it never was a real problem.
Steve seems a bit obsessed about "length" in all kinds of shades so I will try to give an elaborate overview of the points he raised:
Problem 1: Which Lisp
In short this section talks about how difficult it is to choose a Lisp because there are so many implementations of it. I never quite understood what the problem was with this. I'm quite happy to have a multidimensional choice out of commercial/non-commercial, bytecompiled, native, Mac, Windows, Linux, Solaris, big, small, slow, fast, embeddable, whatever implementations of Lisp.According to Steve, choosing a Lisp seems difficult too, because Paul Graham planned to create his own Lisp (Arc) and this would mean something like "don't use Lisp!". I always thought choosing a programming language should be done by analysing ones needs and what is offered from the available options.
Steve mentioned another difficulty arising, when a hypothetical newbie chooses a Common Lisp implementation:
Well - it seems Steve himself is not yet at the point were he realizes that exactly this adaptability is what makes Lisp the language it is. For me "CL" means not only "Common Lisp", but also "Compiler Language". I know no other language which makes it so easy to adapt it to that kind of language you want and need.Eventually, if they stick with Lisp at all, they learn they can override most of these defaults in nonportable ways, which makes things infinitesimally more bearable.
Problem 2: Worthless Spec
This section argues that the Common Lisp spec (ANS X3.226-1994) is ancient. Well it's twelve years from that time - maybe thats really a long time for a language spec. He claims that everytime someone proposes to update the standard someone else does not agree to that. That's true - there are many people who think redoing the official standards process which lead to ANSI Common Lisp might be to much work for the gain. What I do not understand with Steve's concerns about this topic is, why other languages like Ruby or Python are allowed to evolve without any official standard at all and Common Lisp is only allowed to change through a labour intensive ANSI process?
Some years ago someone suggested creating a COMMON-LISP-2006 package (I've updated the year 'cause I cannot remember when the proposal was) and extending the language the way one wants within this package. This technique is used by many Lispers to adapt CL to their needs and it could also get used to adapt "Common Lisp the Language". Even better would be an approach which would be modular; allowing CL2006 implementations to implement only parts of the CL2006 package tree. But I think such an approach is not really needed. As with any current programming language: Solutions grow out of actual problems. If there is no solution for a particular problem, someone will write a solution for it if he needs it. Interoperability and reuse are programming problems too and people will write solutions.It's not a new thing for Lisp - not even for any other Programming Language - that there are people who like more to talk, argue and discuss than to simply sit down and solve their problems. Fortunately the Lisp community has quite a lot talented programmers who have contributed in a constructive and helpful way to create useful solutions for Lisp-using people.
Problem 3: CLOS
Steve claims some things which are factually not true with CLOS. I recommend any Lisper to get a copy of the book "The Art of the Metaobject Protocol" - IMHO this is an essential lecture for any programmer who wants to understand object oriented programming; not only in Lisp. For the curious: you can read what Steve got wrong with CLOS (and other things) by reading the comments to his blog entry. I don't think its necessary to duplicate what already was said.
Problem 4: Macros
Steve builds up an analogy between CL Macros and duct tape. While being entertaining to read, I don't see were this story fits with the topic of Lisp being no acceptable Lisp. The points raised are old: "hygienic" vs "not-hygienic" macros, whining about the syntax - the only new and innovative one is Steve's complete misunderstanding that CLOS is made as just a set of macros.
Problem 4: Type System(Yes the numbering is quoted right - Steve has chosen to use this for an analogy between type systems and incorrectly numbering headings) In this section Steve promises to talk more about type systems - but not now. There is nothing more than a loose mention that the CL type system may be flexible but not enough. After his claim that Haskell and OCaml are faster because of their type systems my intention to read up his next posting about type systems dropped below zero.
I think this whole exercise in trying to write a rant about something you do not even fully comprehend is entertaining but useless.
