First, thank you for writing this. I would ask that you continue to think & refine and share back what you discover, prove, or disprove.

To me, it seems quite likely that B will have a lot of regularity to it. It will not be good code from the human perspective, but there will be a lot of structure I think, simply because that structure is in T and the environment.

I'm interested to see if we can (i) do more than claim this is likely and (ii) unpack reasons that might require that it be the case.

One argument for (ii) would go like this. Assume the generating process for A has a preference for shorter length programs. So we can think of a A as a tending to find shorter description lengths that match task T.

Claim: shorter (and correct) descriptions reflect some combination of environmental structure and compression.

• by 'environmental structure' I mean the laws underlying the task.
• by 'compression' I mean using information theory embodied in algorithms to make the program smaller

I think this claim is true, but let's not answer that too quickly. I'd like to probe this question more deeply.

1. Are there more than two factors (environmental structure & compression)?
2. Is it possible that the description gets the structure wrong but makes up for it with great compression? I think so. One can imagine a clever trick by which a small program expands itself into something like a big ball of mud that solves the task well.
3. Any expansion process takes time and space. This makes me wonder if we should care not only about description length but also run time and space. If we pay attention to both, it might be possible to penalize programs that expand into a big ball of mud.
4. However, penalizing run time and space might be unwise, depending on what we care about. One could imagine a program that start with first principles and derives higher-level approximations that are good enough to model the domain. It might be worth paying the cost of setting up the approximations because they are used frequently. (In other words, the amortized cost of the expansion is low.)
5. Broadly, what mathematical tools can we use on this problem?

thereby writing directly into your brain’s long-term storage and bypassing the cache that would otherwise get erased

What do we know about "writing directly" into long-term storage versus a short-term cache? What studies? Any theories about the mechanism(s)?

See also Nomic, a game by Peter Suber where a move in the game is a proposal to change the rules of the game.

I grant that legalese increases the total page count, but I don't think it necessarily changes the depth of the tree very much (by depth I mean how many documents refer back to other documents).

I've seen spaghetti towers written in very concise computer languages (such as Ruby) that nevertheless involve perhaps 50+ levels (in this context, a level is a function call).

In my experience, programming languages with {static or strong} typing are considerably easier to refactor in comparison to languages with {weak or dynamic} typing.*

* The {static vs dynamic} and {strong vs weak} dimensions are sometimes blurred together, but this Stack Overflow Q&A unpacks the differences pretty well.

No source code

I get the intended meaning, but I would like to made the words a little more precise. While we can find the executable source code (DNA) for an organism, that DNA is far from a high-level language.

I got minimal value from the article as written, but I'm hoping that a steel-man version might be useful. In that spirit, I can grant a narrower claim: Smart people have more capability to fool us, all other things equal. Why? Because increased intelligence brings increased capability for deception.

• This is as close to a tautology as I've seen in a long time. What predictive benefit comes from tautologies? I can't think of any.

• But why focus on capability? Probability of harm is a better metric.

• Now, with that in mind, one should not assume a straight line between capability and probability of harm. One should look at all potential causal factors.

• More broadly, the "all other things equal part" is problematic here. I will try to write more on this topic when I have time. My thoughts are not fleshed out yet, but I think my unease has to do with how ceteris paribus imposes constraints on a system. The claim I want to examine would go something like this: those constraints "bind" the system in ways that prevent proper observation and analysis.

If instead you keep deliberating until the balance of arguments supports your preferred conclusion, you're almost guaranteed to be satisfied eventually!

Inspired by the above, I offer the pseudo code version...

loop {
    if assess(args, weights) > 1 { // assess active arguments
        break; // preferred conclusion is "proved"
    } else {
        arg = biased_sample(remaining_args); // without replacement
        args.insert(arg);
        optimize(args, weights); // mutates weights to maximize `assess(args, weights)`
    }
}

... the code above implements "the balance of arguments" as a function parameterized with weights. This allows for using an optimization process to reach one's desired conclusion more quickly :)

Thanks for your quick answer -- you answered before I was even done revising my question. :) I can personally relate to Dan Luu's examples. / This immediately makes me want to find potential solutions, but I won't jump to any right now. / For now, I'll just mention the ways in which Jacob Collier can explain music harmony at many levels.

Preface: I feel like I'm wearing the clown suit to a black tie event here. I'm new to LW and respect the high standards for discussion. So, I'll treat this an experiment. I'd rather be wrong, downvoted, and (hopefully) enlightened & persuaded than have this lingering suspicion that the emperor has no clothes.

I should also say that I personally care a lot about the topic of communication and brevity, because I have a tendency to say too much at one time and/or use the wrong medium in doing so. If anyone needs to learn how to be brief, it is me, and I'll write a few hundred words if necessary to persuade you of it.

Ok, that said, here are my top two concerns with the article: (1) This article strikes me as muddled and unclear. (i) I don't understand what "get" five words even means. (ii) I don't understand how coordination relates to the core claims or insight. My confusion leads to my second concern: (2) what can I take from this article?

Let's start with the second part. Is the author saying if I'm a CEO of a company of thousands I only "get" five words?

A quick aside: to me, "get" is an example of muddled language. What does the author mean w.r.t. (a) time period; (b) ... struggling for the right words here ... meaning? As to (a), do I "get" five words per message? Or five words some (unspecified) time frame? As to (b), is "get" a proxy for how many words the recipient/audience will read? But reading isn't enough for coordination, so I expect the author means something more. Does the author mean "read and understand" or "read and internalize" or "read and act on"?

Anyhow, due to the paragraph above, I don't know how to convert "You only get five words" into a prediction. In this sense, to me, the claim it isn't even wrong, because I don't know how to put it into practice.

Normally I would stop here, put the article aside, and move on. However, this article is featured here on LW and has many up-votes which suggests that others get a lot of value out of it. So I'm curious: what am I missing? Is there some connection to EA that makes this particularly salient, perhaps?

I have a guess that fans of the article have some translation layer that I'm missing. Perhaps if I could translate what the author means by get and coordination I would have the ah-ha moment.

To that end, would someone be so kind as to (a) summarize the key point(s) as simply as possible; with (b) clear intended meanings for "coordinate" and "get" (as in you only "get" X words) -- including what timeframe we're talking about -- and (c) the logic and evidence for the claims.

It is also possible that I'm not "calibrated" with the stated Epistemic Status:

all numbers are made up and/or sketchily sourced. Post errs on the side of simplistic poetry – take seriously but not literally."

Ok, but what does this mean for the reader? The standards of rationality still apply, right? There should still be some meaningful, clear, testable takeaway, right?