I’m still doing things

It’s been a while.

The credulity of my tagline — where I’m still writing but you’re not reading — seems to be a bit stretched lately, as it’s been months since I last posted anything about anything.  That’s not for a lack of interesting things to write about, but mostly for a lack of time in which to do it:  I most often think of things to say while driving to or from work, or while lying in bed at night having spent my evening feeding and cleaning up after small children, and neither scenario is especially conducive to posting on one’s blog.  Even when I do get a little private time, I tend to spend it working on Smile, rather than posting here about Smile, which doesn’t especially help with anybody else knowing what’s going on with it.

So let’s talk about Smile.

Status Report

I’ve been doing my best to “code in the open,” as an open-source project on GitHub, so that the world can hold me accountable for the things I say about Smile.  It’s real; and it exists; and you can count the lines of source code yourself to prove it.  Excluding Intel’s Binary Integer Decimal library and Boehm’s Garbage Collector library, it’s 50,000 lines of C, and growing steadily.

I’d like to think the fact that you can click a “Download” button means that it’s not vaporware.

The current implementation is simultaneously really powerful and really weak:  In some places, it feels like a programming language that fell back in time through a portal from the future, while in other places, you stare at it and wonder how it could possibly be so big and powerful yet not yet have basic data structures like arrays.  That’s the paradox of having a single person with limited time working on it: Some parts are very complete, and some parts are very incomplete.  I’m working to remedy the incomplete parts, but that simply takes time.

There are no release builds available to download, but if you have Windows with Visual Studio or a Linux box with GCC, you can build the source code yourself and give it a spin.  Of course, the complete lack of documentation doesn’t help.

— but that’s also something I’m slowly remedying.  I own smile-lang.org, and slowly but surely, I’m writing the introductory programmer’s manual there.  Once again, some parts are really well-documented, and some parts are so undocumented that their omission is almost criminal.

Whither Smile?

One of the hardest questions to answer about Smile is “What is it?”  When I started in the 1990s, I had a vague sort of idea as to what I wanted, but I didn’t know why I wanted it.  I just knew something was missing in existing mainstream languages, and that Lisps seemed to have that undefinable something, and nothing else did.  Over the years since I started, lots of new languages have been created, and lots of new languages have become popular, but the something is still missing.  Finally, over last summer, I slowly realized what the something was, the something that Smile was designed to be and that few other languages are, and I gave the concept a name:  Domain-Oriented Programming.

Smile, you see, is designed first and foremost to be a programming language in which nothing is truly “built-in.”  For contrast in, say, Java, there’s a fundamental notion of a class.  Java is built around classes, and everything it does is based on that.  Class is a reserved keyword, even when it’s inconvenient for that to be a keyword.

This doesn’t mean that Java is good or bad, only that in Java, certain constructs are baseline constructs without which the language simply doesn’t exist.  Class.  Method.  If-then-else.  That’s fine — if the built-in constructs are what you need to build your software, but if they fall short, you spend huge amounts of time and energy attempting to express what you wish you could say, but using only the language’s built-in concepts.

For example, how does your language of choice interact with a SQL database?  Again, in Java, the most common ways to express it are by just embedding the text of a SQL query inside a string, or by some kind of “fluent syntax” like SqlConnection(“localhost”).Select(“field”).From(“Table”).Join(“OtherTable”).  Neither of these describes what you likely want to express — which is simply SQL as a first-class citizen in your language:

public List<Foo> SomeFunction(int id) {
    if (id > 0)
        return select * from table inner join otherTable where id = @id;
    else
        throw new InvalidArgument(“id”);
}

The experienced programmers reading this are likely shaking their heads.  In order to write that code, you need to extend your language definition, and get the committee to agree on it, and change the compiler, and deploy it, and a whole host of other logistical problems, assuming you’re even willing to have select and from and join become keywords in your language.  At the same time, many other programmers are looking at that and simply saying, “Huh, that sure would be nice.”

Interestingly, the few Lisp programmers in the audience are probably laughing right now.  Extending Lisp to support something like SQL syntax isn’t an especially difficult exercise; it’s no different than adding any other significant construct has been.  When object-orientation became popular, several Lisp packages were created to support it, and eventually the CLOS package became part of the Common Lisp standard.  Nothing’s really “built in” with Lisp, other than a handful of special core forms, and everything else is bolted on after the fact.  Even if-then-else and logical and/or were latecomers to Lisp — all just more things built out of Lisp macros, and that we eventually agreed ought to be included out-of-the-box on most installations.

Lisp, then, allows a kind of evolutionary growth that most other languages don’t:  Consider how long it took for C# to support async/await, or how long it too for Java to support lambda.  Those constructs had to go through design committees, and be added to the compiler, and deployed and supported.  You couldn’t just grab the source code of some GitHub repo somewhere and then suddenly lambda exists.

This ability to freely mutate the language, then, was one of the things I wanted most in Smile.  I wanted the power of a Lisp but with none of the look or feel — no piles of parentheses, no inscrutable names like cons and mapcar, but the freedom to arbitrarily mutate and expand an easily-readable language, on a whim, whenever you feel like it.  Need async/await?  Just write some code.  Need inline SQL?  Just write some code.

The New Goal

Thing is, when you start with “ability for the programmer to (re)design the language on the fly,” you end up in weird and unexpected places.

First off, if you can make the most common tools in the language look like anything — after all, they can be redesigned on the fly if nobody likes them! — you might as well make them simple.  Easy-to-read.  Like executable pseudocode.  As my friend Ben commented when he first read some Smile code, “It’s like BASIC!”  And why not make it a little like BASIC?  Simple things should be simple.  So Smile’s if-then-else reads like if-then-else.  Smile’s while loop reads like a while loop.  Declarations are mostly avoided, and simple iterative code reads like simple iterative code.

Second, you need Lisp as your foundation.  Lisp is the only language I know of where nothing is truly built-in, where everything only has meaning relative to how it’s interpreted in context.  In Lisp-land, that also means you’re stuck with parentheses to describe your syntax, but as I’ve realized (and implemented), there are ways around that restriction.

Smile uses Lisp the same way C uses assembly language:  It’s just under the hood, right at the ready, and everything you read in the source code is more-or-less a one-to-one mapping to some deeper construct.  There is a translator that is part of the parser, and it turns a pretty if-then-else into a (if x y).  So you could just write (if x y), but why would you?  The pretty if-then-else is easier to read, and it means the same thing, so write the easy-to-read version.  The translator is a high-speed dynamic syntax engine (currently limited to LL(1), plus a few bits of trickery to get around LL(1) limitations) — so if you don’t like my if-then-else, you can make your own if-then-else.

Or you can make SQL as a first-class syntax.  Or vector/matrix algebra.  Or async/await.  Or a goto.  Or a come-from.  It’s not up to me to tell you what your code should look like, only that you should strive for readability.

And that brings up the third notable point about Smile:  Anybody can solve a problem in a given language.  You can implement a SQL query in some way in every programming language, from assembly on up, but anymore, programmer time is the most important resource in computing — not just development time but maintenance time as well.  The goal of a Smile programmer isn’t to solve a problem — anybody can do that — but to find the best solution, the solution that is the most readable, which sometimes means least code but more often means matching the domain of the thing it’s expressing.

SQL should look like SQL.  Vector algebra should use “·” and “x”, not “.DotProduct()” and “.CrossProduct()”.  Every domain has a way of expressing its concepts, and the closer the code gets to the target domain, the fewer bugs creep in.

I remember reading a tweet on Twitter a year or so ago where an AI expert said she was “constantly trying to avoid creating domain-specific languages,” and I thought that was among the silliest things I’ve read in ages.  “.DotProduct()” and “.CrossProduct()” really are a domain-specific language in disguise:  You haven’t reduced the mental overhead required to understand vectors just because you expressed things as classes and methods; if anything, you’ve increased the mental overhead, because now you need to understand vectors and Java and how vector concepts are mapped to Java in order to understand the code.  A mathematician can’t understand the Java code, and even a Java programmer can’t understand the Java code even if he knows vector algebra without also knowing the mapping between the two.

Everything is a domain-specific language, so why not use a programming language that makes DSLs themselves easy, first-class citizens?

Domain-Oriented Programming

Smile, then, is my attempt to fix the domain-mapping problem.  I have no idea if it succeeds, but I think it gets closer to fixing it than most existing languages.  You can directly express your domain in a wide variety of ways in Smile, from adding and redefining operators to adding new keywords to simply bolting in whole new domain-specific languages.

(For the record, I know Racket has a DSL system, and a few other languages offer macros, but this is something different:  You don’t say explicitly which DSL to switch to; they just coexist, and you can use them all simultaneously!)

That could be somewhat chaotic, without rules.  Obfuscated code is trivial to write when you can make black into white and vice versa!

So Smile includes a bunch of tools to keep the chaos to a minimum:

• There is a “core” syntax for common programming techniques we all tend to agree on, like algebra. You can opt out of it at any time, but you shouldn’t without a good reason.
• The syntax-translator is lexically scoped, so introducing a new syntactic construct here doesn’t necessarily break code over there; just because I might tweak the language to make async a keyword in my code doesn’t mean it has to be a keyword in yours.
• You can reuse syntax rules just like you reuse anything else, so just because you’ve introduced a new construct like await that depends on an expression doesn’t mean you now need to also define what an expression is too.

Under the hood, of course, this all turns into Lisp-like forms, which themselves get compiled down to an efficient optimized bytecode, so when you execute your fancy new micro-DSL, it’s no worse for performance than, say, Python (or an older JavaScript engine), and depending on how you define it, it might run substantially faster than an equivalent Python program — if you can translate anything into anything, why not translate into the fastest possible representation?

Smile is currently a bytecode-interpreted dynamically-typed language.  It could be “compiled” the same way that V8 compiles JavaScript, but that’s an exercise for somebody else to tackle — my goal is to make Smile work in the first place, and to work well enough that people don’t object to at least trying it.  If your primary goal is stupid-crazy-fast execution times, Smile’s not really the right language for you — C and C++ and Rust and Go and D might be better fits.  But if your goal is elegant, simple, maintainable code, Smile’s got a lot going for it.

Everything But The…

…and all that said, my friend Ben thinks I’m going through a bit of a midlife crisis right now, and he might be right.

In my early forties, I’m about the right age for such an issue, and over the summer, I had a strike from a thunderbolt hit me that after thirty-plus years of writing code, software is dumb.

Nobody cares if code is good.

The average lifespan of code in the business world is maybe ten years these days.  Whatever is built today, it’ll have a project to replace it starting about five years from now.  So most code is just good enough to limp to production.  Unit tests?  Documentation?  Quality?  What do those matter if code is as disposable as Kleenex?

Worse than that, you used to have to at least attempt to write passably acceptable code to not waste resources.  You had, say, ten servers the code was going to run on, and buying more servers cost money, so if something was slow or bad, you had to optimize it at least well enough for it to reach that ten-year lifespan.

But now, everything is moving to the cloud.  Computers are cheap.  Sudo terraform AWS go make me a server!  It doesn’t matter if your code eats CPU; as long as your code is somewhat-parallelizable, the businesspeople can open their wallets, turn up the more knob, and work around the quality problem with money.

Smile is a language for beauty — for writing pretty, elegant, readable, maintainable code.  But that doesn’t matter.  Nobody wants elegant, maintainable code.  The only thing people want is shipped code, and nobody cares if your code is God’s gift to programming — or is the XKCD comic parody of itself.

Add onto that the fact that JavaScript has eaten the world, for good, bad, or otherwise; and that “good enough” in programming languages seems to have become good enough; and that most programming language research seems to now be around ever-more-complex layers of type theory, which is so very alien to Lisp’s and Smile’s way of thinking; and that even with WebAssembly, browsers will likely never really run other languages; and that software development for “personal computers” is a thing of the past.  I like making things that make peoples’ lives better, and making servers talk to each other doesn’t do that, and that’s the only environment where new languages seem to have room to exist anymore.

Which has all left me in a weird place.  I’m building the language I always wanted, a programming environment with few rules, an environment designed to let my mind soar, an environment designed to make software gorgeous — but I have this weird feeling like none of those are things that anyone really, truly wants, and that depresses me more than a little bit.  There’s stuff in Smile that I can confidently say is a new direction in computer science or at least in software engineering — but I’m just not sure anyone cares.

So I’m still working on Smile, but now it feels more like I’m working on it just to finish it rather than any other reason.  Maybe someday my own website will run on it, and it’ll be the one website in the world running on Smile.

All That Said…

If you do want Smile, feel free to go grab a copy.  It’s on GitHub, and it compiles on Windows (VS2017) and Linux (GCC, and maybe Clang).  The latest work is going on in the feature/bootstrap branch, and I highly recommend that over the stable master branch right now.  At some point, I’ll open up more of the documentation on smile-lang.org so you can better understand how to use it.  — assuming there’s anyone out there who wants the documentation enough that it’s worth publishing.

But for now, I’m going to mostly keep quietly coding on my little GitHub research project, working slowly toward finishing it, on the theory that if you build it, they will eventually come.  Maybe.

 

Update:  I apparently wrote nearly 3,000 words about metaprogramming without thinking to use the word metaprogramming.  I may need a vacation.