遞歸式模擬

*早排同朋友吹水，對話大致如下

A: 世界局勢變幻不定，究竟點先可以知會發生咩事呢？

B: 而家科技倡明，行 simulation 咪得囉。

A: 但如果個 simulation 入面啲人都打算行 simulation 咁點算？

B: 咁無窮無盡㗎喎 👻 

(*按：我唔記得同exactly邊個同幾時。應該都係分靈體之一)

我個問題係：點樣避免呢個問題？係咪可以模擬一個唔識整模擬嘅世界出嚟？

山西經

「河水出焉，而南流東注于無達」

已知崑崙 = Khum (Khufu)

佢話「河水注于無達」就肯定知係地中海

一查 The Ancient Egyptians called the Mediterranean Wadj-wr/Wadj-Wer/Wadj-Ur

咁啱又啱音...

Austere Instincts

Austerity is soon to be a thing in AI.

Chatted with an old classmate today. One of the things we discussed was how the DeepSeek team started to work on architecture and operations optimizations way way back (a couple years, that's prehistory in AI timescales).

This doesn't seem like a big deal yet. But with the end of Moore's law looming on the horizon, monetary conditions gradually tightening, it seems quite obvious that the prevalent attitude of "don't worry about the money, we can just buy more hardware from Nvidia" can't be sustained for much longer.

When US firms realize capital (money) can't provide a sufficient edge, they *will* have to pivot to focus more on efficiency. And of course they will. But cultural inertia is a thing in large and proud organizations, and I'm actually not very bullish on the idea that a bunch of python coders can somehow be repurposed to hand writing low level PTX or optimizing bits of the pipeline 0.x% at a time.

The effect isn't linear either. Frontier model training is more of a series of scientific experiments than an engineering process, and lowering the cost of doing experiments allows people to experiment more and learn more. The reduced time and money costs of running experiments might be the difference between a subpar model and SoTA.

Unfortunately I don't have an intuition of how much inefficiency there is in training large models today. What we do know is that US trade embargoes of advanced chips to China has forced Chinese firms to focus on efficiency. It's pretty obvious Chinese firms are leading in the efficiency department especially after recent DeepSeek revelations. *If* there is much room for improvement in the efficiency space, US AI firms are going to be royally fucked in the coming 1-2 years.

Long context LLMs

Around mid-2024 I tried running feeding Hong Kong court cases to long context (local) LLMs and see how they fared.

Didn't work well.  Although most of them claimed to support long contexts, they kind of just failed (got repetitive, etc.).

Since I wasn't really using them for important stuff I just set it aside.

[ Fast forward an eternity later (i.e. a couple months) ]

I'm pleasantly surprised to find that more recent models (as ancient as Llama-3.2 3B) actually performed pretty well on such long contexts. Llama-3.2 3B was actually the worst of the bunch, and apparently the recent gemma-3 models did really well.

The only issue is that gemma-3 27B is a bit slow.

Didn't really bother to check whether the difference was due to model performance or llama.cpp bugs. I suspect more of the former than the latter (Llama3.1 8B occasionally glitches out too).

Anyway, that's kinda good news, maybe I actually can create a comprehensive database of HK court case summary and commentary....

Fear

Fear isn’t just fear

It is the logical invitation of possibility

The imagining of madness requires madness itself.

What would God do? Solving the Paradox of Tolerance and why "evil" or "bad things" happen in the world.

There's a weird trick when dealing with various spiritual or philosophical questions -- just ask: what would God do?

It's a very useful trick because it immediately allows you to throw away bad assumptions and put things into the right perspective.

Take for example the so-called "Paradox of Tolerance". Basically it's the idea that "if a tolerant society allows intolerant ideologies to grow unchecked, those intolerant forces may eventually suppress tolerance itself" (GPT4o)

To most people, that sounds like a perfectly reasonable concern. So the popular solution in modern "tolerant" societies is to become intolerant (of intolerance) so that society can ensure it is tolerant. (Now *this* is a strange paradox!)

But if you switch perspective from being a human being victimized by evil intolerant forces, to that of divine nature, you'll see something totally different. God not only tolerates, but also loves. And thus through its creations, whether we call it "good" or "evil", all things exist and are treated the same. This is why we have "evil" in the world -- God tolerates and loves it as much as it loves "good". There is no paradox -- point to anything that exists or is conceivable to exist, and God loves it since it is its creation.

The paradox returns only when we return to the human perspective. Where is the difference? First, consider the scenario where the intolerant suppresses tolerance in society. This of course does not hinder your *personal* freedom to tolerate this intolerant society -- but those who self-proclaim to be tolerant do not tolerate this. Now, which came first, the intolerant people, or the stance that intolerance cannot be tolerated? From the story telling perspective it is the former, but in fact it is the latter that was planted first into the hearts of the self-proclaimed tolerant.

It is a logical fallacy to think that one cannot be tolerant of the intolerant. It might seem to be pointless, but it is not something illogical or logically impossible. Once this is accepted, the paradox also goes away. You have people who proclaim to be tolerant, but discreetly hold an exception to that rule. The intolerant people show up, trigger that exception, and now there's a "paradox" because the self-proclaimed tolerant can't fully explain why they are so intolerant of the intolerant.

Now, people may object that, if the intolerant gain power, bad things will happen. That may be the case, but that just also shows that tolerance is not really without exceptions. It simply implies that those who proclaim tolerance will be intolerant to things that lead to bad outcomes. (Btw, God is not intolerant of bad outcomes...)

I have put "evil" and "bad things" in quotes because there is no objective way to define these concepts that is orthogonal to subjective value systems. Those that are truly tolerant realize that these concepts are never absolute.

-------------------

Zzz

Those that are in conflict will be separated.

Objective Function of Intelligence

After a decade or so of "machine learning" in the 21st century, arguably the only important lesson is what they call the "Bitter Lesson", which tells us that, with the current level of computation power we have, just saying what you want and performing gradient descent is more than enough. As long as we have an objective function that is composed of differentiable functions, we can just shove FLOPS into it and make it work.  

As such, the most valuable thing on Earth right now, something that would be worth trillions of USD, is an objective function of intelligence.

The fact that we can't define intelligence, but depend on it, is a really really interesting fact.

What does it mean? It means that intelligence is not a simple thing we can objectively (as opposed to subjectively) define. Instead, it depends on human judgments. In particular, value judgments.

This is not to say that value judgments are the only thing important for intelligence. But it is a dependency.

In contrast, the kind of intelligence that does not require human value judgments is simply what we call computation. We have formal frameworks to define and validate computation. The reason we can't apply those things to general intelligence is the dependency on human judgment.

This kind of means "we" (collectively) get to choose. Choice. Free Will.

Ah yes. The most valuable thing in the 21st century is our collective free will. 

Of course this also means that we don't have an "objective" function of intelligence in that the function must be subjective to the user's choices.

To me, this is almost a fulfillment of the idea of "ask, and it will be given to you; seek, and you will find; knock, and it will be opened to you." (Matthew 7:7)

To those who have not yet experienced providence, it is difficult to accept that the limit on ourselves is not lack of substance, but that we don't yet know what we truly desire. And now the trillion dollar objective function is staring directly at us. This is not "heaven on earth" yet for sure, but divine concepts do manifest in such similar worldly forms.