I know so little about Smalltalk and its way of software development. A year ago, I read about VisualAge, and asked this question on the lisp subreddit:

https://www.reddit.com/r/lisp/comments/4saada/curious_about_...

It has an interesting response from lispm. Having only worked in languages where the source is stored in files, it's hard for me to get a grasp on this kind of development. I catch rare glimpses of this. Here's another example, from http://dept-info.labri.u-bordeaux.fr/~strandh/Teaching/Langa...

> A major problem with Scheme that does not exist in Common Lisp is that the Scheme language does not define an internal representation for programs, although to be fair, all Scheme implementations do.

> For instance, in Scheme, there is a difference between:

    (define (f x y . (z))
       ...)

     (define (f x y z)
        ...)

(You'll probably have to read that short section "Program/data equivalence"; I couldn't find the right excerpt)

Scheme code is stored in text files. Common Lisp code is a sequence of lisp objects. This is a strange difference to me; I'm an unexceptional web programmer. But it's "the very basis of the Common Lisp macro system, which uses Lisp data to represent and manipulate Lisp code." I am condemned to be intrigued!

The distinction is that Common Lisp source code is not text; it's Lisp values. What you see in a text file of "Lisp source code" isn't actually Lisp source code; it's a text serialization of Lisp source code. Other serializations are possible.

As a practical matter, most of the time the distinction doesn't matter. You edit text files. Your Lisp reads the text files and the first thing it does with the contents is convert them to Lisp source code. The source code is then processed as you would expect.

One place where the distinction matters is in macro processing: Lisp macro expansion operates on Lisp source code, not on text. If it operated on text, as, for example, C preprocessors do, then macro expansion would be a matter of string substitution. Instead, Lisp macro expanders operate on abstract syntax trees represented as convenient Lisp data structures, which means that macros are not limited to what can be conveniently done with simple string substitution. You have the full Lisp language available, operating on a convenient Lisp representation, to compute the expansion.

More generally, when source code is represented as convenient data structures in the language, it's easier to build tools that operate on it. C compilers do not operate directly on text input; they convert it to internal data structures more convenient for the various stages of the compiler to work with. So does Lisp. The difference is that in a C compiler the representations used by the compiler are private implementation details, and in Lisp they are standard surface features of the language, available to every user.

Another example: Go has a parser and AST that comes with the standard library, which again is just using regular Go data structures (structs and interfaces). If you wanted to write a macro preprocessor, using the built-in parser seems like a much better idea than string manipulation.

So, I'm wondering if there's anything more to the way Common Lisp does it?

[1] http://esprima.org/ [2] https://github.com/estree/estree [3] http://esprima.org/demo/parse.html