Humans are the future of vibe coding

I'm wondering what the end game of this LLM vibe coding is. There's been chatter of how even professional developers are shifting towards not reading the code and instead testing the correctness of the output of these LLMs. Often when grilled they admit that the LLM writes the tests, which are subject to the same hallucinations. It probably has a compounding effect on the correctness of the resulting code...

For webdev, these LLMs can adjust their output by running a headless browser. These LLMs seem to be really good at iterating at problems. If you run a headless browser, a human could also see what the test runner sees, meaning you could monitor these LLMs as they make inferences on the resulting code. But I think it will be important to distinguish different types of tests.

• There are unit or integration tests that humans have not written or read and can't be bothered to. I'm skeptical of the utility of these. It's an "if a tree falls in a forest" situation. If a llm writes a test and it passes, is the product is stable? I don't know, do you?

• There are e2e tests that humans can monitor in some way, written by an llm or not, and the human can see the test run and step through the screens. These seem useful for both humans and LLMs.

• There are tests that are measurable, like performance or A/B tests, which provide a ground truth that is optimized for. Are there compromises LLMs will make if you do not fully specify the design well? Humans are subject to perverse incentives when rewarded for a poorly defined metric, I'd assume llms would be too.

• There are tests against historical data. Any idiosyncracies your historical data has, your LLM is also subject to reproducing.

• Then there test that are written by humans and then forgotten about while diligently running in the CI/CD. I can see value in having these, just like I could see value in hiring a detail oriented human to pore over the historical data like an accountant, providing a baseline of 'correctness'. Asking the LLM for its opinion on your tests could help but then you're in a ship of theseus situation where if your tests needed significant overhaul or the test case changes you end up just letting the LLM do the hard part.

Are there other types of tests? that's probably where the future's at.

There is a sort of ground truth that LLMs don't really have about the world, due to being purely digital beings. Humans also struggle to make sense of the ground truth, but providing a ground truth in the form of tests and guardrails only makes sense if you can steer the ground truth to be the truth, right? Just like bad training data negatively impacts the performance (usually measured by tests) of an LLM, so could bad guardrails alters the final product. Will robots give these LLMs a better ground truth? Will they still hallucinate in familiar and unfamiliar ways?

CAP Theorem explained with a horse and carriage

You are king or queen of a vast realm. Your subjects depend on your wise decision making, and your law to maintain order.

Therefore you've written down all laws so that every citizen knows what is lawfully permitted, and what is a criminal offense.

It would hardly be just if somebody was punished for a crime they did not know existed. You decide it's important for all people to be able to see these laws. You send your messengers across the kingdom to the many cities in your realm, each with a tablet inscribing all of the laws, to be placed in the public square and announced for those who can't read.

Unfortunately, some of your messengers return with reports that a major road has been blocked by a flood. The river is uncrossable. There is simply no way to deliver the tablets until the flood waters recede, which coud be weeks.

This puts you in a conundrum. You could hold off on adopting these new laws on principle that everybody must see the same sets of laws. But it could be some time.

The Yellow River Breaches Its Course by Ma Yuan, 1160-1225

The CAP theorem can be used to explain this situation. It was originally envisioned as a pick any two:

• Consistency (Every city has the same laws)
• Availability (There are laws on display, even if they are out of date)
• Partition Tolerance (Roads being blocked)

The more modern thinking on the CAP theorem is that partitions are unavoidable. Just like a King or Queen can't stop a flood, network engineers can't always stop a broken router or a server from going dark.

Consistency suggests that all computers in the network have the same data, just like all cities in our kingdom see the same sets of laws. Due to blocked roads, you're forced to hide the laws from everybody until you can be sure that every city has seen the new set of laws.

If you relaxed that requirement, you allow some cities to show maybe an older version of the laws even when the roads are blocked, you're choosing to prioritize availability. Prioritizing availability means that for some cities, the laws may be out of date, but at least there's some laws on display, potentially important to keep the peace. While the flood is ongoing, cities outside of the reach of the Capital operate on their own set of laws.

The only way to stay consistent (to have all laws be the same) and available (the laws are on display) is to have clear roads and no partitions. To make sure everybody is on the same page, you'd want riders returning from each city confirming that the new laws are in fact on display, at which point you can confidently say the laws are universal.

Databases make these tradeoffs when replicating or horizontally scaling. Postgres recommends many ways of replicating your data. Some of them are consistent:

Shared Disk Failover

Uses a single disk array that is shared by multiple servers. If the main database server fails, the standby server is able to mount and start the database as though it were recovering from a database crash. This allows rapid failover with no data loss.

One significant limitation of this method is that if the shared disk array fails or becomes corrupt, the primary and standby servers are both nonfunctional

File System (Block Device) Replication

All changes to a file system are mirrored to a file system residing on another computer. The only restriction is that the mirroring must be done in a way that ensures the standby server has a consistent copy of the file system

SQL-Based Replication Middleware

A program intercepts every SQL query and sends it to one or all servers. Each server operates independently. Read-write queries must be sent to all servers, so that every server receives any changes. But read-only queries can be sent to just one server, allowing the read workload to be distributed among them.

Care must also be taken that all transactions either commit or abort on all servers, perhaps using two-phase commit (PREPARE TRANSACTION and COMMIT PREPARED).

Notice how all of these methods require the replica to be online. That's the price of consistency. Other Postgres replications relax the consistency, and focus more on availability:

Write-Ahead Log Shipping

Warm and hot standby servers can be kept current by reading a stream of write-ahead log (WAL) records. If the main server fails, the standby contains almost all of the data of the main server, and can be quickly made the new primary database server.

Logical Replication

Logical replication allows a database server to send a stream of data modifications to another server.

On the other hand, if there are other writes done either by an application or by other subscribers to the same set of tables, conflicts can arise.

Trigger-Based Primary-Standby Replication

Operating on a per-table basis, the primary server sends data changes (typically) asynchronously to the standby servers.

Because it updates the standby server asynchronously (in batches), there is possible data loss during fail over.

Asynchronous Multimaster Replication

Each server works independently, and periodically communicates with the other servers to identify conflicting transactions. The conflicts can be resolved by users or conflict resolution rules.

Notice how all of these availability first strategies all involve some form of data loss (not consistent) when a failover happens and if writes are allowed on replicas there are conflicts that need to be resolved. That's the price of availability.

Once the roads clear, your messengers deliver the laws to the furthest parts of your kingdom. You can rest assured knowing that nobody will be punished for a law they were unaware of. That is, until you need to make changes to the law. You're getting nervous about any blocked roads as you're starting to realize it's the same problem all over again.

Calculating Pi in 5 lines of code

Known as the Madhava–Leibniz method of calculating Pi. Originally known as the Leibniz formula for calculating Pi, it was discovered that Indian mathmematician and astronomer Madhava of Sangamagrama (or perhaps his followers) had formalized this method of calculating Pi in the 14th-15th century. A naming improvement since Leibniz already has too many formulas named after him.

It's beautifully simple. 1 - 1/3 + 1/5 - 1/7 + 1/9... continued to infinity gets you 1/4th of Pi.

And it's very easy to write in code. Here it is using python:

quarterPi = 0
for i in range(0, 1000000):
  numerator = 1 if i % 2 == 0 else -1
  denominator = i * 2 + 1
  quarterPi += numerator / denominator

print(quarterPi * 4)

If we run this code, we get 3.1415916535897743, which is surprisingly close to Pi, accurate to the 6th decimal.

Infinite series can't really be calculated to completion using a computer, but that for i in range(0, 1000000): line controls how many iterations you calculate for (currently 1 million).

If we crank it up to 100 million iterations we can really make our CPUs work:

3.1415926525880504

100 million iterations took 180 seconds on my computer, and it's now accurate to the 8th decimal. Now, what about if we only loop 100 times?

3.1315929035585537

Not very accurate. It took my computer less than half of a millisecond to compute 100 iterations.

It's incredible to think that mathematicians were able to come up with this formula in a time when they couldn't necessarily check the results by hand.

User /u/Another_moose came up with this one line version of the algorithm:

sum(-(i*8%16-4)/(i*2+1) for i in range(10**6))

Mathematics can really illustrate how powerful programming can be, often with very little effort. Code representing math often ends up being some of the most concise, maybe because these two fields of study share some deep parallels.

Networking explained with a horse and carriage

Imagine you're a king or queen ruling over a vast realm.

Key to your power is knowledge. You've maintained a network of horse riders and stables for your messengers to use as they bring news from far parts of the kingdom to you.

One day, you have an incredible proclamation, a wedding invitation for the grandest ceremony and party.

UDP

You pay several local villagers to ride their horse to a city in the realm, each carrying a "save the date" letter. Since the date is far off you tell the messengers that their job is done after delivering the message.

This would be analogous to sending UDP packets. Messages are sent without confirmation of delivery. After giving these instructions to the riders you realize you don't know when or if the letters arrived at their destination.

After sending the letters you get some unfortunate news. One of the most important guests cannot attend that date, and so the wedding date will have to be changed. You repeat the process, sending out messengers with the new date.

Some of the letters with the new date arrive sooner than the letters with the old dates. You end up getting confused envoys asking which date is the real date. This is known in computer networking as Out-of-order_delivery.

Determined not to create any more chaos you hire the wisest thinkers to implement a system that can deliver messages in order reliabily. Here is a system they came up with. They call it TCP.

TCP

You use a trusted courier that will deliver the message, then return with confirmation that the message has been delivered. Easy enough. Your thinkers also tell you to write down a number on each message corresponding to the number of letters sent to that destination. That way the recipient knows if a message comes out of order.

TCP is like this trusted courier. TCP is a protocol that provides reliable, ordered delivery of data.

Once the courier returns, they get sent out again, this time to acknowledge (ACK packet) that the confirmation was received by the original sender. Although it's a lot of travel, missing messages can be noticed when the acknowledgement has gaps in the numbering.

Some computer terminology: The time it takes for the messenger to travel from the capital to its destination is called Latency. Instead of messengers you would call them "packets". Ping would be how long it takes for a full round-trip. In this analogy some of the messages may be time-sensitive (especially as the wedding date gets closer). If a message doesn't arrive to its destination for any reason (bandits? treacherous cliffsides?), it might not be noticed until the next message is received, and then it would still require a messenger to be sent back to the capitol in order to request a copy of the missing of letters. This increased number of round-trips and additional latency is one of the big reasons TCP is not a great system for fast communication (like in fast paced games).

In real life before trains, latency would measure in days, weeks, or even months for very far off destinations. But in computer networks, latency is anywhere between a few milliseconds to... well, actually there is no upper limit.

Light could theoretically circumnavigate the Earth within 133 milliseconds, which means our best fiber optic cables could feasibly see latency as low as 66ms for somebody on the other side of the world. This number is optimistic because there's always slow down from hops along different routers to and along the internet backbone, and from wifi or satellite. There can also be lots of internet "traffic" which bottlenecks some routes around the internet causing lag spikes.

When you navigate to a website or play an online game, you can imagine signals traveling back and forth between your computer and the server that hosts that website or game. Every time you see the page load or the contents update, some data must have been exchanged. Imagine if a message is lost because a line went dead or one of the many routers on the route had a bug or reset. In UDP those messages would be lost forever. With TCP, there is a way for the sender and receiver to request the missing messages but with a big latency penalty equal to the round trip time. The basic principles of message passing still are followed in the same way they were when the word was spread by horse and carriage. Many complex networking related topics like the Cap Theorem can be explained by the horse and carriage analogy – because at its core it boils down to simple message passing.

Custom made wood lapdesk

I use my laptop often and deal with neck strain, so I thought I'd build myself a lapdesk capable of raising my laptop closer to eye level.

The design is pretty simple. A stand and base connected by a basic doorhinge, and some legs that can lock into slots along the base.

I had originally tried using a CNC router to cut out the piece but found it took too long to cut. This time I used a laser cutting machine.

Using the lightburn software, I mocked up the designs. Lightburn is somewhat cumbersome, I would recommend using a different software like illustrator and export to SVG.

This is the laser cutter. Shout to Seattle Makers.

The cut was pretty quick, less than a half hour. It's also quite precise. The edges are burnt uniformly, giving it a pretty unique appearance. The plywood also smells nice and woody while lasering.

Once cut, I glued pieces together, essential to have gaps for the legs to slot into.

You can see the design taking shape. I sourced regular old door hinges and assembled the final product.

Here's a side profile of an earlier design (the CNC routed attempt).

The completed product

Ask Sirius business cards!

I made this using a laser cutter, lightburn software, and an anodized aluminum card.