I'm a heavy user of Common Lisp, and I only dabble in Racket from time to time. While Common Lisp is my tool of choice for a lot of reasons stated in the post, since the post largely skews favorably toward Common Lisp, I'll offer two things that Racket shines at:
1. Documentation
Racket has a beautiful documentation system called Scribble [0]. Almost all documentation written for Common Lisp is ugly, inconsistent, and doesn't follow any sort of standard. Common Lisp has a gazillion "documentation generators" which do little more than slurp up all the documentation strings and barf them back out as e.g. HTML. Common Lisp users tend to just rely on READMEs + jump-to-definition to learn how a project works. A few Lisp programmers (e.g., Shinmera, Edi Weitz) have made their own documentation systems that look better than average, but are still at least an order of magnitude less capable than Racket's offering.
Racket's documentation system is aesthetically pleasing, very well organized, and makes the programmer want to write long-form documentation. Moreover, Racket (the language) sets a superlative example of what its users can strive for in terms of quality of clear, no-nonsense writing.
2. Languages and DSLs
One of Lisp's (any dialect) defining features is the ability to build domain-specific languages, syntactic extensions, and the like by way of a special kind of function that can run prior to the code's execution. These special functions are called "macros", and across Lisp dialects, there are lots of different takes on them.
Common Lisp gives you a grab bag of facilities for writing code that lets you change syntax at different levels: DEFMACRO (scoped variant: MACROLET), DEFINE-COMPILER-MACRO, DEFINE-SYMBOL-MACRO (scoped variant: SYMBOL-MACROLET), SET-MACRO-CHARACTER, and a few others. At best, I'd consider these only loosely related, and they all work on really "raw" data representations (parsed S-expressions and character streams). In part because of this, ANSI Common Lisp doesn't provide access to any notion of environment, source location, etc. when writing new macros.
Macros defined by the above operators "live" in different places and are compartmentalized in different ways. There are some Lisp libraries that help you tame definitions and keep them fenced in, like NAMED-READTABLES which helps you organize where reader macros (macros which change how Lisp is parsed at the character-level) live and how to turn them off and on in a sane way.
Contrast with Racket which has explicit documentation on how to write macros [1,2], how to debug them [3], and how to encapsulate them a new DSL embedded in the system [4]. Racket—as a technical system—clearly defined the phases of compilation, what objects are manipulated during the phases (e.g., "syntax objects"), how to provide debugging information, and so on.
In the end, Common Lisp's system for making DSLs feels a little runny. It's supremely hacker-friendly, but it's a bit more difficult to deliver, in one fell swoop, a language as a library. Racket, on the other hand, at the expense of having increased rigidity and needing to read two or three complete chapters of a manual, you get a way to neatly bundle up a language. This is why Racket had nice DSLs like Typed Racket, Hackett, Scheme RnRS, etc.
I use Common Lisp and I develop relatively large DSLs in the language, and even if I want to, it's hard—if not impossible—to write large DSLs that have nice error messages, source locations, etc. without writing an entirely new "reader" (lisp's term for a code parser) myself every time. Smaller DSLs where user code doesn't exceed more than a few lines? Examples like:
- regex syntax
- string interpolation syntax
- a pattern matching feature
Common Lisp is great, and you get a ton of mileage out of the facilities it provides. But larger ones? Examples like:
- an embedded language for writing documentation with markup
- a complete language with lazy evaluation semantics
- a language for non-deterministic programming (like SCREAMER, Prolog)
- a language for strictly typed functional programming (like Coalton, Hackett)
Tackling these in Common Lisp requires ambition and a very steady hand, whereas Racket's overall system (language, implementation, documentation) makes that a comparatively painless exercise.
[0] https://docs.racket-lang.org/scribble/
[1] https://docs.racket-lang.org/guide/macros.html
[2] https://www.greghendershott.com/fear-of-macros/
As someone trying to understand what people find special about Lisp, this was helpful. I get it now about macros, but doesn't modifying the language in the compiler phase just complicate solving the actual runtime problem? I feel I would get distracted into writing my own DSL. I've never wanted to modify the language as I was writing in it. Now I'm curious.
Don't think of it as modifying the language. Think of it as adding new ways of expressing ideas in the existing language.
These sorts of languages/language extensions pop up all the time in "ordinary" programming. For example, common DSLs we see often.
- regex is a language for string patterns
- JSON is a language for simple nested data
- SQL is a language for expressing DB queries
- OpenMP contains (essentially) a language for parallelizing loops
- (this might sound crazy because it's so common and basically built in standard in almost all programming languages:) infix notation for math
The problem is that many of these are absolute kludges in existing languages. Anybody who has had the displeasure of trying to interpolate DB table names into SQL SELECT strings knows this. Most languages try to make it a bit better by adding preprocessors, using builder-patterns, abusing reflection, or otherwise. Looking under the hood of some languages' implementations of these ideas will often lead you to see the most cursed of programmer things in existence.
Each of those languages (or close syntactic cousins of them) can be written directly in Lisp, as a library, without shoving them into strings, or using external tools, or making custom file formats. They call can exist as a part of the Lisp language as a syntax extension.
Making DSLs requires exercising that muscle a bit; it's non-trivial to design a new and useful language that fits orthogonally into an existing system.
It's like being introduced to OO programming where, at least for me, it made sense but it wasn't immediately obvious what a good object model for a problem might be. It took me lots and trial and error to understand how OO works in "the real world", where it's good, and where it falls short.
Once you get they hang of making DSLs—or even just small but powerful syntax extensions—it's an amazing tool to have at your disposal.
My second thought, which I excluded for brevity, was aren't functions and libraries language extensions? I mean technically you could write C without malloc, but it wouldn't be very useful. Regex is a great example of an embedded DSL, but I've never thought "I wish I could write a macro to change it's behavior on the fly in this special case". The last time I looked at this issue I saw that sure, if you don't have generics, it would be helpful to have a language that could write typed functions for you. That makes sense. Changing the parser and compiler behavior before any runtime code executes could be very interesting. There are certainly enough people waxing poetic about the expressiveness of it to make it worth learning.
https://github.com/norvig/paip-lisp/search?l=Markdown&q=defm... - all references to defmacro in the markdown files
Chapter 3 shows a simple macro, just adding a while loop to the language.
Chapter 9 shows some more complex ones, including a with- macro and a grammar compiler macro.
Chapters 11 and 12 show the development of a Prolog implementation in CL using defmacro to aid in compilation again in Chapter 11.
Chapter 12 shows adding an OO system to the language. Technically not needed with CLOS, but a good demonstration of what can be done with macros.
There are other examples (why I included that search link). Macros let you change the language in ways large and small. Many uses could probably be replaced with functions, though you'd end up having to throw a bunch of quotes about or closures in order to delay processing things. You'd also be delaying that to runtime, which incurs its own penalties compared to macro expansion and compile time (if a compiled CL).