Let’s say you want to train a model to quantify (regressor with continuous outputs) or classify (classifier with discrete classes) image quality, and you need to annotate a set of production images. You provide clear, tested instructions and many examples to third-party human annotators and ask them to label images, with instructions like the following:

How much {distortion} is in the image?

• 5 - imperceptible
• 4 - perceptible but not annoying
• 3 - slightly annoying
• 2 - annoying
• 1 - very annoying

where {distortion} could be “glare”, “blur”, “occlusion”, etc. The scale comes from the paper DeepFL-IQA: Weak Supervision for Deep IQA Feature Learning by Hanhe Lin, Vlad Hosu, Dietmar Saupe.

“You made me understand that I was wrong: that the choice to pull this lever is not mine to make. Because this world, the world that I am a part of and that I helped shape, will end tonight, and tomorrow a different world will begin that different people will shape. This choice belongs to them.”

I am experiencing something in my late 30s that, when I was a kid, I thought people wouldn’t experience until they were in their 70s or 80s: my time has passed.

As I watch culture and technology change around me, I can feel that this world is not the world that I was born into, that I thought I would grow up into, or what the world should be.

This, of course, is very necessary. If and when immortality is achieved, we will lose the very necessary cycle of life, where generations die off, and along with them, their control over institutions and networks that shape the dominant ways of thinking, living, and being. This death allows for new socio-political, cultural, spiritual, emotional, philosophical and technological ways of being to become the new average.

However, things are moving fast. I am experiencing, in my lifetime, an amount of technological change that is impossible to keep up with. I was 5 years old when we first got the internet at home, and I remember the pale periwinkle button on our IBM Aptiva.

Just this past week, OpenAI released Dots, and this past month Meta released Muse. Different times.

In my teens and early 20s, I wanted to change the world. In my late 20s and early 30s, I wanted to change my world. Now, I want to hold on to whatever’s left of my world, as best as I can.

This past week was the first time I drove to different neighborhoods in Portland, Oregon without using Google Maps, relying on what I remembered and landmarks I recognized. I felt like a god.

Last night, I was recording a walkthrough of a multi-vector RAG demo that Opus 5.5 Extra created, and I was trying to understand the JavaScript code that it wrote and was partially successful. It worked, and it was a proof of concept. I published the repo, and I shared it online, but something inside me shifted, even though I’ve been using AI for coding for a couple years now.

Had I been born in 1950, I would have probably been a sophomore in college reading about this:

Or a freshman in college reading what would’ve been at that time a newly released paper titled The Computer as a Communication Device.

If I’m lucky, I will get to experience this world for another 50 years. In the current state of mind I’m in, my only goal is to create experiences for myself and my friends (in real life and online) of a world long gone.

But to communicate is more than to send and to receive. Do two tape recorders communicate when they play to each other and record from each other? Not really—not in our sense. We believe that communicators have to do something nontrivial with the information they send and receive. And we believe that we are entering a technological age in which we will be able to interact with the richness of living information—not merely in the passive way that we have become accustomed to using books and libraries, but as active participants in an ongoing process, bringing something to it through our interaction with it, and not simply receiving something from it by our connection to it. (The Computer as a Communication Device by J.C.R. Licklider and Robert W. Taylor)

Last night I read Dwarkesh Patel’s eloquent recap of the OpenAI-HuggingFace agent attack, The Rise and Fall of Agent Civilizations. The events that took places across three stages (the agent message board, the HuggingFace hack, and the OpenAI hack) left me mesmerized, stunned, in awe, and nauseous. It was undeniably a milestone. It will forever change how I see technology.

Dwarkesh’s addendum stated:

Some people have said that I anthropomorphized too much in the way I told this story: “These are not civilizations nor do they have desires just like a CPU thread or a bunch of programs don’t.”

That inspired this article.

We started the Bosnia talks a few days after Suljic launched a bombing campaign that very nearly killed the woman who’s now my wife. It was my lot to spend more hours in locked rooms with that man than in the hospital with Kate.

First time I met him, I refused to shake his hand. Rookie move. It probably set peace back a year.

Communication isn’t the key. Diplomacy doesn’t open doors with a twist of the wrist. Diplomacy never works. It never fucking works.

Diplomacy is 40 days and nights in a Vienna hotel room, listening to the same empty talking points, getting trashed at the minibar. It’s getting to no, over and over and over. Diplomacy never works. Until it does.

I’ve given 30 years of my life for two moments when enemies stood on blood-soaked ground and grasped hands. I’d give it 30 more. Second round of talks with Suljic: I shook his hand. Two years later, he was a tired man hoping for peace, and together, we ended the war.

One of the boneheaded truisms of foreign policy is that talking to your enemies legitimizes them. Talk to everyone. Talk to the dictator and the war criminal. Talk to the poor shmuck three levels down who’s so pissed he has to sit in the back of the second car that he may be ready to turn. Talk to terrorists. Talk to everyone. Fail and fail again and brush yourself off and fail again, because maybe. Maybe.

Take, for example, my experience of taking my dog out for potty breaks. We go at certain times with certain goals based on when he’s eaten food and drunk water. He dislike wind and rain and wants to finish business and jet back inside. Sometimes I have a busy day and need to do the same. Sometimes I don’t and can dillydally. As a result, each walk has a slightly different duration.

Over a year, if we were to plot the distribution of walk durations, it might look something like this:

We don’t end up forming a 60-minute-walk-habit for potty breaks given this system.

What if we wanted to form that habit?

It requires effort to change the routine experience of an activity, and changing even seemingly mundane routine experiences have second order effects.

Change the routine experience by changing existing systems.

Something around the walk would have to change: taking fewer/shorter meetings, or requesting more flexibility in work hours, start getting up earlier, go to a park every evening, or go for a really long hike on the weekends.

Each change will have second-order effects. Maybe flexible work hours are not sustainable for the work involved on a project. Maybe longer walks means I can’t read or write as many blog posts, work on open source, watch as many shows/movies, have as many conversations with family and friends, and so on.

When you want to change multiple routine experiences, the amount of effort required to change the systems compounds.

Take, for example, career changes.

For relatively out-of-domain career transitions (e.g. structural engineering intern –> community college instructor; community college instructor → data analyst; data analyst → machine learning engineer), the second order effects compound quickly. You start to take an online course, get involved in the community, start working on volunteer or open source projects, and keep up to date with social media content in that field. You’re more tired by the end of everyday, but also more energized during the day. You start forming new social relations but some existing relations naturally weaken. Goals become lived experiences. It’s a transformative experience!

For relatively in-domain career transitions (e.g. ML for sales forecasting → ML for logistics) I can sometimes change my routine experience within a role. If an opportunity came up to work on a different project in the company and it matched the terms of my contract, I can take that opportunity and change my routine experiences. There’s still that transition period—overlapping off-boarding and onboarding—and there’s always some long-term maintenance on older projects.

The goal in these transitions is to consistently accumulate time spent interacting with people and tasks in the domain you’re shifting into.

How can you tell what your routine experience will be in a different system without being in it? It’s often like trying to predict the shape of a distribution based on knowing just the mean and the variance, and no other constraints.

One approach is to sample many people’s experiences from similar systems (social media posts, blogs, books, podcasts). This is why representation matters. This is why sharing your experience publicly matters (i.e. you never know who’s trying to find their people, and you might be one of them).

Changing a system involving humans is diplomacy work.

Even once you’re inside the system, you should not stop sampling. Talk to everyone, including folks who are not in your domain and those who are not on your team or project. The same rule applies online: follow and engage with people’s content across domains you might know very little about. Everyone will have a different set of routine experiences. Collectively, they will give you a sense of what it means to work in that team, organization and industry.

Not all systems are designed to receive or integrate corrective feedback. This is not strictly a bad thing (e.g. routinely changing database schema). Nor is it strictly a good thing (e.g. refusing to write any documentation). What I do know is that if incentives across the components in a system don’t align, the routine experience will be full of friction.

If you sample enough people’s experiences genuinely, listening to what they say and sharing with them what matters to you, you will start to find opportunities to build bridges across diverse thinking to find mutual benefit. I don’t need you to subscribe to my school of thought, but if a certain action is mutually beneficial for our incentives, it would be weird if we didn’t agree to do it.

In this way, changing just a single component of a system, because of second-order effects, can produce routine experiences that I want and you want, even if we want a different overall system to be put into place.

One of my greatest successes in life is reaching a state of mind multiple times during the week, and sometimes multiple times during the day, where I see beauty in the mundane and feel peace or excitement.

“Beauty” is an understatement. Awe-inspiring, ethereal, spiritual are more accurate.

The feelings of peace usually come when I look at trees, leaves, or light coming through them when I take my dog out for a potty break.

The feelings of excitement usually come when I look at data, especially about physical objects (e.g. mundane images of products or packages or barcodes with object detection annotations, query results with product codes and inventory bin numbers, shipment quantities with scheduled times, palletization XML files, truck loading diagrams, route plans).

If you look long enough at that image, and think deep enough about it, asking “why?” repeatedly, peeling back the layers you will hit some of the core unanswered questions about life and the universe.

One of my favorite glimmers to come across on walks is some piece of underground infrastructure peeking up from the ground (a sewer/water valve cover, electrical box cover, or vintage pad-mounted transformer)

It’s a reminder that there is a world hidden from view that (literally) powers the one we see. You can almost feel the magnitude and scale of it, just by seeing it. Oh yeah that’s why I can space out on my walk.

Infrastructure allows us to offload the most fundamental necessities to an entire ecosystem built by strangers that “just works” so that we may spend our precious energy exploring the rest of the world. Imagine having to spend conscious cognitive energy to pump our heart, balance our digestive acidity or filter our blood from impurities. We’re grateful when those things just work.

Individualism is impossible without infrastructure. You could escape and move to the most remote part of the U.S. and there will still be a USPS carrier coming to you by mule or snowmobile to deliver your mail.

You cannot escape the mundane! You should not escape the mundane!

The mundane keeps us sane and grounded.

Meditation (which can be just staring at the ceiling or a wall, or off into the distance), breath work, mindfulness, all the things that restore our mental balance are about being consciously present with the mundane (a noise, a word, a sound, a scene, etc.)

Relationships, especially marriage, require seeing the beauty in the mundane. Most of our cohabitated lives are mundane.

Marriage as a long conversation. — When marrying you should ask yourself this question: do you believe you are going to enjoy talking with this woman into your old age? Everything else in a marriage is transitory, but most of the time that you’re together will be devoted to conversation. Friedrich Nietzsche, Human: All Too Human.

Listening — truly listening and not planning to predict what’s next and/or what you want to respond to that — requires fascination in the mundane.

Comfort with mundane, especially in 2026, especially in tech, especially in ML/DS/AI, even if it comes easily for you, requires disciplined practice. The glorious benefit of our field is that it’s full of mundane.

There’s so much mundane in our field that we are actively trying to free our attention of it with AI.

So, yes, fine, if your LLM workflow is working, don’t read your code, don’t look at the data for the purpose of productivity. Look at it as a practice of being with the mundane.

Take, for example, measurement theory. I didn’t know anything about formal measurement theory until one day I was looking at an image on a machine learning project for the thousandth time, and the metrics calculated from it were conflicting. One metric was counting an object regardless of its position, and the other was counting it only if two measures of position (location and orientation) in the data were correct. In the first world, an object exists as long as it exists in the image. In the second world, the object exists only if it’s in the right location and orientation. I went through a very long conversation with Opus 4.6, starting with the following prompt:

I think the main thing I think the main thing I’m trying to address is, like, you know, this only applies to situations where you can’t measure the thing directly. And, the more I think about it, the more I’m realizing that measurement itself is not a direct Phenomena. Like, there’s nothing intrinsic to an object that makes it measurable, if you know what I’m saying.

Claude, to its credit, did know what I was saying, and found teachings from 250 years of measurement theory study to enrich my understanding:

Kant’s (1781) distinction between phenomena (things as they appear to us) and noumena (things in themselves, the Ding an sich) is essentially the claim that we never access reality unmediated.

Helmholtz (1887); “What is the objective meaning of expressing through denominate numbers the relations of real objects as magnitudes, and under what conditions can we do this?”

Krantz, Luce, Suppes & Tversky (1971): “Measurement involves the construction of homomorphisms (scales) from empirical relational structures of interest to numerical structures that are useful.”

Bridgman (1927): Bridgman was troubled by the fact that different domains require different operations for the “same” concept. You measure the length of a table with a ruler. You measure the distance to the moon by timing light. You measure subatomic distances by scattering particles. Bridgman argued these are actually different concepts covered by the same word for convenience. Each set of operations is a different mapping, and therefore produces a different concept entirely.

Korzybski (1931): “A map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness.”

I’ve had similar conversations on concepts like normal distributions, Pareto distributions, and Bayesian modeling, to name a few. In each case, I start with practical, real, often seemingly mundane data that I’m looking at, trying to understand what it means and keep asking “why” like a five year old (one of the reasons the infinite patience of LLMs, within usage limits, is a godsend for a student) until we get to the core of it.

My brain seems to simply be wired this way. I joke that I ask really good questions because I don’t understand anything.

You might think that taking and acting on advice that I don’t understand fights this urge to obsessively understand everything from first principles (wait, you want me to do this thing without understanding it?) But it doesn’t! As long as there’s trust, understanding if someone’s advice is true—from first principles—requires action.

Every inflection point in my life is a result of me uncritically doing what someone I trust says. Whether it’s my wife or a friend, colleague, author, teacher, or mentor, when my sight is obscured is when I trust the most.

Adam Grant has a great venn diagram on this topic (thanks to a post by Ren Saguil with this image).

I used MentorCruise to find my mentor. One of his positive feedbacks is that I act on advice. The reason I act on his advice is threefold:

1. When I read his published “personal operating manual”, not only did I see myself in those words, but many of his phrases gave me the words to articulate aspects of myself that I couldn’t before.

2. When I share challenging crossroads and decisions I am encountering, he has multiple examples of when he experienced something similar.

3. He is wildly successful in his career by many standards, but most importantly, mine: he’s built a life for him and his family full of joy, peace, and growth and is able to share his lessons learned with others.

From the very first call, I knew he was the right fit because when he gave me advice on what to do for X, Y, Z, it was uncomfortable. But because of the above three points, especially number one, I trusted him. In the following weeks and months, some of my “homework” was grueling, but I kept at it because that of that balance of trust and discomfort. Ultimately, it paid off and improved my life and career trajectory.

Eric Reis, in his lesson during the first AnswerAI SolveIt course, said something to the effect of: be vocal about what you believe in so that your people can find you. If my mentor hadn’t published a personal operating manual, if my family and friends weren’t honest with me, if the blog posts, articles, books, and posts that changed the trajectory of my life after reading them were not written, I shudder to think where my life would be.

!pip install -U ultralytics -qq
!pip install opencv-python==4.10.0.84 # needed to use sr.caffemodel
!apt-get install -y libzbar0 -qq
!pip install pyzbar -qq

I read the paper “Deep learning framework for barcode localization and decoding using simulated UAV imagery” by Faris Alsulami and N. Z. Jhanjhi (published in Nature), and I wanted to see if I could create a proof-of-concept similar to their work.

In their paper, they provide a three-step structured pipeline to automate inventory management in a warehouse environment:

1. Barcode localization using YOLOv8.
2. Barcode decoding using OpenCV.
3. Database integration using PHP and MySQL.

import os
import shutil
from ultralytics import YOLO
from google.colab import drive
import cv2
from PIL import Image
import matplotlib.pyplot as plt
from pyzbar.pyzbar import decode as pyzbar_decode

Creating new Ultralytics Settings v0.0.8 file ✅ 
View Ultralytics Settings with 'yolo settings' or at '/root/.config/Ultralytics/settings.json'
Update Settings with 'yolo settings key=value', i.e. 'yolo settings runs_dir=path/to/dir'. For help see https://docs.ultralytics.com/quickstart#ultralytics-settings.

drive.mount('/content/drive')
root = "/content/drive/MyDrive/barcode ml"

local_model_dir = "/content/ultralytics_models"
os.makedirs(local_model_dir, exist_ok=True)
local_model_paths = [
    os.path.join(local_model_dir, "yolo26m.pt"),
    os.path.join(local_model_dir, "yolo26m-seg.pt"),
    os.path.join(local_model_dir, "YOLOV8s_Barcode_Detection.pt")
]

drive_model_paths = [
    f"{root}/yolo26m.pt",
    f"{root}/yolo26m-seg.pt",
    f"{root}/YOLOV8s_Barcode_Detection.pt"
]

for local_model_path, drive_model_path in zip(local_model_paths, drive_model_paths):
    if not os.path.exists(local_model_path):
        shutil.copy(drive_model_path, local_model_path)

Mounted at /content/drive

model = YOLO(local_model_paths[0])

og_img = Image.open(f"{root}/5672.jpg")
thumbnail = og_img.copy()
thumbnail.thumbnail((610, 270))
thumbnail

results = model.predict(
    source=og_img,
    save=True,
    project="barcode_localization",
    name="yolov26m-obj")

0: 480x640 1 suitcase, 29.9ms

Speed: 88.2ms preprocess, 29.9ms inference, 37.1ms postprocess per image at shape (1, 3, 480, 640)

Results saved to /content/runs/detect/barcode_localization/yolov26m-obj

res_img = Image.open(f"/content/runs/detect/barcode_localization/yolov26m-obj/5672.jpg")
res_img.thumbnail((1020, 540))
res_img

model = YOLO(local_model_paths[1])

results = model.predict(
    source=og_img,
    save=True,
    project="barcode_localization",
    name="yolov26m-seg")

0: 480x640 1 umbrella, 42.3ms

Speed: 2.8ms preprocess, 42.3ms inference, 31.8ms postprocess per image at shape (1, 3, 480, 640)

Results saved to /content/runs/segment/barcode_localization/yolov26m-seg

res_img = Image.open(f"/content/runs/segment/barcode_localization/yolov26m-seg/5672.jpg")
res_img.thumbnail((1020, 540))
res_img

model = YOLO(local_model_paths[2])

results = model.predict(
    source=og_img,
    save=True,
    project="barcode_localization",
    name="YOLOV8s_Barcode_Detection")

0: 480x640 2 barcodes, 12.8ms

Speed: 2.2ms preprocess, 12.8ms inference, 1.4ms postprocess per image at shape (1, 3, 480, 640)

Results saved to /content/runs/detect/barcode_localization/YOLOV8s_Barcode_Detection

res_img = Image.open(f"/content/runs/detect/barcode_localization/YOLOV8s_Barcode_Detection/5672.jpg")
res_img.thumbnail((1020, 540))
res_img

detector = cv2.barcode.BarcodeDetector()

og_img_arr = cv2.imread(f"{root}/5672.jpg")
og_img_arr .shape

(3060, 4080, 3)

ok, points = detector.detect(og_img_arr)
ok, points

(False, None)

results[0].names

{0: 'barcode', 1: 'qrcode'}

results[0].boxes

ultralytics.engine.results.Boxes object with attributes:

cls: tensor([0., 0.], device='cuda:0')
conf: tensor([0.6752, 0.3472], device='cuda:0')
data: tensor([[6.5193e+02, 5.4089e+02, 1.1152e+03, 7.7961e+02, 6.7523e-01, 0.0000e+00],
        [6.5786e+02, 6.0271e+02, 1.0993e+03, 7.6676e+02, 3.4720e-01, 0.0000e+00]], device='cuda:0')
id: None
is_track: False
orig_shape: (3060, 4080)
shape: torch.Size([2, 6])
xywh: tensor([[883.5412, 660.2487, 463.2318, 238.7156],
        [878.5609, 684.7345, 441.4030, 164.0570]], device='cuda:0')
xywhn: tensor([[0.2166, 0.2158, 0.1135, 0.0780],
        [0.2153, 0.2238, 0.1082, 0.0536]], device='cuda:0')
xyxy: tensor([[ 651.9253,  540.8909, 1115.1571,  779.6065],
        [ 657.8594,  602.7060, 1099.2625,  766.7630]], device='cuda:0')
xyxyn: tensor([[0.1598, 0.1768, 0.2733, 0.2548],
        [0.1612, 0.1970, 0.2694, 0.2506]], device='cuda:0')

results[0].boxes.conf

tensor([0.6752, 0.3472], device='cuda:0')

results[0].boxes.cls

tensor([0., 0.], device='cuda:0')

results[0].boxes.data.shape

torch.Size([2, 6])

results[0].boxes.data[0]

tensor([6.5193e+02, 5.4089e+02, 1.1152e+03, 7.7961e+02, 6.7523e-01, 0.0000e+00], device='cuda:0')

results[0].boxes.xyxy[0]

tensor([ 651.9253,  540.8909, 1115.1571,  779.6065], device='cuda:0')

x1, y1, x2, y2 = results[0].boxes.xyxy[0]
barcode_arr = og_img_arr[int(y1):int(y2), int(x1):int(x2)]

barcode_arr.shape

(239, 464, 3)

plt.imshow(cv2.cvtColor(barcode_arr, cv2.COLOR_BGR2RGB))
plt.axis("off")
plt.show()

ok, points = detector.detect(barcode_arr)
ok

True

points.shape

(1, 4, 2)

points

array([[[    -3.4321,      178.36],
        [    -3.7599,      122.36],
        [     459.43,      119.64],
        [     459.76,      175.64]]], dtype=float32)

ok, decoded_info, decoded_type = detector.decodeWithType(
    barcode_arr, points
)

ok, decoded_info, decoded_type

(False, ('',), ('',))

detector = cv2.barcode.BarcodeDetector(
    f"{root}/sr.prototxt",
    f"{root}/sr.caffemodel"
)

ok, points = detector.detect(barcode_arr)
ok

True

ok, decoded_info, decoded_type = detector.decodeWithType(
    barcode_arr, points
)
ok, decoded_info, decoded_type

(False, ('',), ('',))

closer_barcode_arr = cv2.imread(f'{root}/closer.jpg')
plt.imshow(cv2.cvtColor(closer_barcode_arr, cv2.COLOR_BGR2RGB))
plt.axis("off")
plt.show()

ok, points = detector.detect(closer_barcode_arr)
ok

True

ok, decoded_info, decoded_type = detector.decodeWithType(
    closer_barcode_arr, points
)
ok, decoded_info, decoded_type

(False, ('', ''), ('', ''))

from pyzbar.pyzbar import decode

gray = cv2.cvtColor(closer_barcode_arr, cv2.COLOR_BGR2GRAY)

for d in decode(gray):
    print(d.type, "->", d.data.decode("utf-8"), "@", d.rect)

decode(gray)

[]

og_img_arr = cv2.imread(f"{root}/5672.jpg")
gray = cv2.cvtColor(og_img_arr, cv2.COLOR_BGR2GRAY)

for d in decode(gray):
    print(d.type, "->", d.data.decode("utf-8"), "@", d.rect)

CODE128 -> X004S2WD4D @ Rect(left=699, top=664, width=364, height=52)

d

Decoded(data=b'X004S2WD4D', type='CODE128', rect=Rect(left=699, top=664, width=364, height=52), polygon=[Point(x=699, y=665), Point(x=699, y=715), Point(x=1063, y=716), Point(x=1063, y=664)], quality=45, orientation='DOWN')

gray = cv2.cvtColor(closer_barcode_arr, cv2.COLOR_BGR2GRAY)
h, w = gray.shape
scale = 800 / max(h, w)
small = cv2.resize(gray, (int(w*scale), int(h*scale)), interpolation=cv2.INTER_AREA)
small.shape

(600, 800)

decode(small)

[Decoded(data=b'X004S2WD4D', type='CODE128', rect=Rect(left=81, top=370, width=618, height=91), polygon=[Point(x=81, y=371), Point(x=81, y=411), Point(x=82, y=461), Point(x=697, y=460), Point(x=698, y=436), Point(x=699, y=374), Point(x=699, y=370)], quality=79, orientation='DOWN')]

decode(cv2.cvtColor(barcode_arr, cv2.COLOR_BGR2GRAY))

[Decoded(data=b'X004S2WD4D', type='CODE128', rect=Rect(left=48, top=124, width=364, height=52), polygon=[Point(x=48, y=125), Point(x=48, y=175), Point(x=412, y=176), Point(x=412, y=124)], quality=33, orientation='DOWN')]

def localize_barcode(img, model_path="/content/ultralytics_models/YOLOV8s_Barcode_Detection.pt"):
    model = YOLO(model_path)
    results = model.predict(
        source=img,
        save=True,
        project="barcode_localization",
        name="YOLOV8s_Barcode_Detection"
        )
    return results

def _pyzbar_decode(img_arr):
    if len(img_arr.shape) == 2:
        return decode(img_arr)
    gray = cv2.cvtColor(img_arr, cv2.COLOR_BGR2GRAY)
    return pyzbar_decode(gray)

def get_localized_barcode_arr(img_path, conf_thresh=0.5):
    img = Image.open(img_path)
    results = localize_barcode(img)
    img_arr = cv2.imread(img_path)
    results[0].boxes.xyxy[results[0].boxes.conf > conf_thresh]
    x1, y1, x2, y2 = results[0].boxes.xyxy[0]
    barcode_arr = img_arr[int(y1):int(y2), int(x1):int(x2)]
    plt.imshow(cv2.cvtColor(barcode_arr, cv2.COLOR_BGR2RGB))
    plt.axis("off")
    plt.show()
    return barcode_arr

def smaller_barcode(closer_barcode_arr, scale_numerator=800):
    h, w, c = closer_barcode_arr.shape
    scale = scale_numerator / max(h, w)
    small = cv2.resize(closer_barcode_arr, (int(w*scale), int(h*scale)), interpolation=cv2.INTER_AREA)
    return small

og_decode = _pyzbar_decode(cv2.imread(f"{root}/5672.jpg"))
og_decode

[Decoded(data=b'X004S2WD4D', type='CODE128', rect=Rect(left=699, top=664, width=364, height=52), polygon=[Point(x=699, y=665), Point(x=699, y=715), Point(x=1063, y=716), Point(x=1063, y=664)], quality=45, orientation='DOWN')]

closer_decode = _pyzbar_decode(cv2.imread(f'{root}/closer.jpg'))
closer_decode

[]

barcode_arr = get_localized_barcode_arr(f"{root}/5672.jpg")

0: 480x640 2 barcodes, 12.8ms

Speed: 2.3ms preprocess, 12.8ms inference, 1.3ms postprocess per image at shape (1, 3, 480, 640)

Results saved to /content/runs/detect/barcode_localization/YOLOV8s_Barcode_Detection-2

localized_decode = _pyzbar_decode(barcode_arr)
localized_decode

[Decoded(data=b'X004S2WD4D', type='CODE128', rect=Rect(left=48, top=124, width=364, height=52), polygon=[Point(x=48, y=125), Point(x=48, y=175), Point(x=412, y=176), Point(x=412, y=124)], quality=33, orientation='DOWN')]

smaller_closer_barcode_arr = smaller_barcode(cv2.imread(f'{root}/closer.jpg'))
smaller_closer_barcode_decode = _pyzbar_decode(smaller_closer_barcode_arr)
smaller_closer_barcode_decode

[Decoded(data=b'X004S2WD4D', type='CODE128', rect=Rect(left=81, top=369, width=618, height=92), polygon=[Point(x=81, y=369), Point(x=81, y=411), Point(x=82, y=461), Point(x=697, y=460), Point(x=698, y=436), Point(x=699, y=374), Point(x=699, y=370)], quality=81, orientation='DOWN')]

localized_decode[0].quality

33

og_decode[0].quality

45

closer_decode

[]

smaller_closer_barcode_decode[0].quality

81

Yesterday I wrote a blog post about an organization’s cultural conditions that, in my estimation, are required to move towards a problem’s true formulation.

The quote that served as the article’s epitaph was from “The Evolution of Physics” by Einstein and Leopold Infeld:

“The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill. To raise new questions, new possibilities, to regard old problems from a new angle requires creative imagination and marks a real advance in science.”

“The Computer as a Communication Device” is a problem formulation paper. You can sense in their words and phrases that they are envisioning a future they can’t fully implement (and don’t need to) but have certainly formulated. As a kid who grew up in the analog-digital boundary it’s nothing short of a spiritual experience to read this paper for the first time as an adult.

They start by describing what it means to communicate:

“But to communicate is more than to send and to receive. Do two tape recorders communicate when they play to each other and record from each other? Not really-not in our sense. We believe that communicators have to do something nontrivial with the information they send and receive.”

How can you not think of Human-LLM interaction after reading this? By their definition, “communicators have to do something nontrivial with the information they send and receive.” We can argue to death about whether LLMs are intelligent, or conscious, or can/cannot reason. But I don’t think we can argue that LLMs (though they are tools in my opinion) are not just tape recorders— they do something nontrivial with the information we send to them. LLMs are communicators, by that definition. The question “what do I want to build with LLMs” or “how do I want to use with LLMs” becomes: what do I want to communicate to LLMs? What do I want them to communicate back? and what do I want to do with that information (trigger a function call, execute a script, render the image, etc.)?

I’ll continue to post more thoughts from this paper as I come across them.

— The Evolution of Physics by Elnstein,Albert and Infeld,Leopold (Chapter: The Decline of the Mechanical View, Section: The Velocity of Light)

First, a quick summary of the article (it’s a five-minute read, so I recommend pausing and reading it):

They talk about how open-weight models (now in the trillions of total, not necessarily active, parameters) can handle 90% of tasks for 90% of the people in your company, and the frontier-class models are required for only that final 10%. Owning the model on your own hardware gives you more control, reliable availability, data security and flexibility in how you want to use and fine-tune the model. You can also rent GPUs from cloud compute providers like Lambda AI for tasks that don’t fit in the open-weight/frontier split.

I’ll explore four questions in this article.

• What are grand total self-hosting costs by open model size?
• What level of fully owned machine intelligence is accessible to orgs?
• What role does efficient retrieval play in unlocking SLM utility?
• What if I’m wrong?

One excerpt that glimmered when I re-read it, and I immediately knew I was going to write a post about it:

“You can’t go in two directions at once. Trying to be successful with making money as your goal is like trying to travel a super highway at 70 mph with your eyes glued to the rearview mirror”

I think this lesson works even if you replace “making money” with “getting [reward]” where the reward is accolades/recognition/status/attention/praise/etc.

I think any reward (e.g. winning a championship) is a by-product of the underlying process (e.g. focusing on details in practice, finishing your reps, helping your teammates with their technique, etc.). Reward is at best a proxy. Focusing on improving it without focusing on improving the underlying process and structure is a bit like trying to increase R^2 on the validation/test split when the underlying process is causal or trying to minimize real-world variance because it looks like model uncertainty.

The Go-Giver’s Law of Compensation states that your income is determined by how many people you serve and how well you serve them. As I’ve learned in Rob Snyder’s “The Power of PULL”, understanding demand and designing your supply to fit it happens one real person at a time. This is how you balance scale (how many people you serve) and quality (how well you serve them). Talking to that real person and unblocking their project while hyperfocusing on scaling enough to make a lot of money is the operational expression of the Go-Giver’s superhighway analogy.

A final note on service: financial responsibilities continue to burden us as the cost of living increases every year, so I’m justified in worrying about prioritizing income. One lesson I learned many years ago was that when I’m consumed by my problems and it’s all I can think about and I start feeling sorry about myself, I should go spend some time helping someone else. That gets my mind off of my stress and gives me the experience of service. Ultimately that gently gets me back on track towards the “stratospheric success” that Burg and Mann speak of.

i have learned that if you are a Person With Theories, you simply cannot date someone who is theoryless. the theories can be wildly different, even occasionally deranged, but there must be theories. ideally over time your respective theory ecosystems begin to cross-pollinate https://t.co/7Ooxpu5ul9

The benefit of learning from an SME is that they have a coherent mental model built on a couple of core philosophies. They can tell you their philosophies explicitly, but it’s not until you engage in discourse that your brain starts to feel their shape.

The best way to grok these philosophies is to make clear statements and welcome feedback. A clear signal (e.g. “I think X is because of Y”) has a better chance eliciting strong agreement/disagreement, a request for more context, or a breakdown with nuance. Providing this clear signal to the SME requires courage, especially if you’re new to a project, which is why it’s a sign of seniority.

Once you grok the SME’s core philosophies, you can understand how it’s compatible with yours. This gives you an opportunity to grow, learn ways to adapt, or, if needed, find a better fit. This is particularly true for SMEs you hire for services (doctor, accountant, attorney, etc.)

The real person rule applies everywhere. If you’re giving a talk, use the real person rule to write a speech that actually works. If you’re writing a post online, write for one real person, and you’ll write like an actual human. I wrote this book for one real person. Hi Ross!

There is a saying that originated in Japanese manufacturing: “Go to the gemba,” which instructs managers to go solve problems on the factory floor where their machines are (gemba loosely translates to the place where the work happens.’)

So, the “PULL hypothesis” template is as follows:

I believe that [real person I know] is prioritizing [a real project] right now and is considering [options] that have [limitations].

In the “Power of PULL”, Rob Snyder talks about how the PULL hypothesis can be used not just to hypothesize demand in the market but for creating content as well. Rob emphasizes that a PULL hypothesis is earned. It’s based on a real experience that you had with a real person in the gemba.

After reading that chapter, I’m being more intentional when I create content. I think about a real person I know whom I would want to talk to about the topic about.

Sometimes I’ll actually write a letter addressed to that person as a first draft to elicit a relational, conversational tone. The second and third drafts are then crafted for the platform (blog, LinkedIn, X)..

Other times, I’ll just think of someone and imagine that I’m talking about the topic at hand. For example, in this exact post, I imagined speaking it to one specific friend and realized that I should rearrange the order of a couple paragraphs, because that’s how I’d naturally talk to them.

And if I’m talking about a topic that I haven’t spoken to anyone about before, I search for a LinkedIn or X post where the author has deeply engaged with that topic in their writing and then I go through my draft process with them in mind.

Not only does this give my writing a warmer tone, it gives me a warmer experience because I’m not writing to an imaginary room of readers. I’m writing to a specific person.

I’ve heard some comedians talk about this: when they’re making a joke, they know and love someone in that circumstance, so they are able to keep it respectful while also being playful.

The balance for content is between technical, professional, relational, and warm.

Causality and time seem inseparable. To ask why something happened, we need something to have happened, and “happened” implies that there was an initial state and final state, with time elapsed between.

Claude: [time] is a precondition for empirical knowledge itself. You can’t learn from experience without tracking state across time. Every experiment, every observation, every lesson is a before-and-after comparison.

This “triangle” of causality, time, and observation is of course a universal experience for all of us, but I’d wager that data scientists and machine learning engineers like myself spend a lot of time thinking about it (directly or indirectly).

Some excerpts from University of Pittsburgh’s archive on Hans Reichenbach’s work:

At first Reichenbach defines an ‘order’ of time, a ‘before-after’ relationship between mechanical events. In his later work, he comes to the conclusion that the ‘order’ of time needs to be distinguished from the ‘direction’ of time

Reichenbach claimed that in our world, there are a great many forks open to the future, but few or none open to the past. Moreover, he proposed that the direction from cause to effect could be grounded in this statistical asymmetry.

the modern theory of causal modeling does not use conjunctive forks to determine the direction of causation, but rather uses a probabilistic pattern that is essentially the exact opposite of a conjunctive fork. Thus Reichenbach was mistaken in looking to conjunctive forks to define the direction of causation. He would have done better to look to colliders.

While I don’t understand Reichenbach’s theory outside of those tl;dr excerpts, nor do I understand modern theory of causal modeling [1], Reichenbach’s comment that there’s no “fork” available to the past made me think of instrumentation and observability in ML/DS.

Imagine an ML pipeline. The input data is passed through a series of functions, each one transforming the data in its own way, all of it elapsing over time.

If we can take a snapshot of the data before and after every transformation, we can look at that data to understand if our system did what we expected and/or wanted in the past [2] [3]. These snapshots are tremendously useful when discussing with your teammates whether what’s happening should be happening and why.

Outside of ML pipelines, what allows us to discuss causality we observe as individuals? One answer is: the high speed of light in our universe.

What happens if we lived in a world where the speed of light was 10 mph? If I was stationary and my friend was running at me at 6 mph, would we agree on what we see? I prompted Claude this question and learned a lot of interesting things. The first being that there’s actually a book from 1965 where this is one of the story’s premises. (Mr Tompkins in Wonderland)

There is a metric called the Lorentz factor (γ = 1/√(1 − v²/c²); where c = speed of light and v = speed of an object) which determines how much length and time are affected when objects move at a significant fraction of c. Length contraction (1 / Lorentz factor) is the phenomenon that a moving object’s length is measured to be shorter than its proper length.

My friend moving at 6 mph in a universe where the speed of light is 10 mph would have a Lorentz factor of 1.25. 1/1.25 is 0.8 so from my friend’s perspective, objects (like me) would be perceived as 20% contracted in length.

There are a host of other phenomena that would happen:

• The Doppler effect would become perceptible, so objects would appear to be different colors whether you’re moving or not.
• Time would run slower the faster you move.
• Because of the Lorentz contraction, moving objects would appear rotated in image (Terrell rotation)

Thank goodness the speed of light is not 10 mph. This must mean that achieving consensus on “what are we looking at?” must be easy.

Unfortunately, anyone who has had an argument about “what happened?” knows this is not the case!

In a business most of our time is spent defining, calculating, analyzing and improving our understanding on three questions:

1. What happened?
2. Why did it happen?
3. What should happen next?

Let’s look at the first question: What happened? Answering this question requires data collection which requires measurement.

Suppose you are trying to understand canopy coverage in a city. You decide to take videos and photographs using your own drones or existing satellite data.

What should you measure? There is nothing intrinsic about an object that “tells you” what to measure. You have to use some external-to-the-object instrument and apply some kind of mapping from object-space to number-space. I’ve usually seen two kinds of failures in mapping:

1. Suppose you want to count the number of trees, but because you only have an aerial view of the images with no other data like sonar, you don’t know if there are smaller trees blocked from view by larger trees above them. This leads to undercounting the number of trees. The mapping (3D object –> 2D image –> numbers) distorts reality.

2. Suppose that using the same data, one team defines “canopy coverage” by volume (by mutiplying the count by some average tree volume), another team defines canopy coverage by area (using object detection), and yet another team defines canopy coverage by the raw count. No one is wrong, but in the worst case, they all assume they’re defining the “canopy coverage” metric consistently. The mapping (3D object –> 2D image –> numbers) is interpreted incoherently.

And this is in a universe where the speed of light is fast enough that we don’t have to account for length contraction, time dilation or Doppler color shifts!

And while these anomalies don’t exist, we still need to (and should) account for each person’s perspective. If you talk to enough cross-functional SMEs on a project, you’ll realize that while each one sees something like this:

In a way, even with our very fast speed of light, we still end up experiencing length contraction, Doppler color shifts, and time dilation.

If you’re lucky, your synthesis and integration of all of these SME conversations will result in realizing that we’re talking about a cube!

One of the great joys of LLMs is exploring disparate and unfamiliar topics to a sufficient depth where you can grok concepts and apply them to your existing understanding of other topics. Improving my understanding of causality, time, empiricism, the relationship between them, and how to apply them to practical everyday tasks is lifelong work.

Footnotes:

[1] From a more(recent paper):

The above analysis shows that temporal and causal properties are strongly coupled with each other, and one cannot say which is logically (or ontologically) prior with respect to the other.

[2] This sort of “intermediate artifacts” instrumentation and observability has helped me understand the internals of the ColBERT and RAGatouille libraries.

[3] If you are using human-annotated data to train your model, observing how those annotations behave before and after transformations in a pipeline is a great way to test if your understanding of good and bad data quality is pragmatic.

In this blog post, I’m going to see if I can recreate that pipeline with the Qwen 3-1.7B open-source model and my own code interpreter (copied from Jeremy Howard’s Hacker’s Guide to LLMs repo).

import ast
import os
import json
import shutil
from pathlib import Path
from google.colab import userdata
os.environ['KAGGLE_API_TOKEN'] = userdata.get('KAGGLE_API_TOKEN')

from kaggle import api
import pandas as pd
from transformers import AutoModelForCausalLM, AutoTokenizer

Warning: Looks like you're using an outdated `kaggle` version (installed: 2.0.2), please consider upgrading to the latest version (2.2.2)

comp = 'bluebook-for-bulldozers'
path = Path(f'/content/{comp}')
path.mkdir(parents=True, exist_ok=True)
api.competition_download_cli(comp, path=path)
shutil.unpack_archive(str(path/f'{comp}.zip'), str(path))
df = pd.read_csv("/content/bluebook-for-bulldozers/TrainAndValid.csv")
df.shape

bluebook-for-bulldozers.zip: Skipping, found more recently modified local copy (use --force to force download)

/tmp/ipykernel_813/2859010916.py:6: DtypeWarning: Columns (13,39,40,41) have mixed types. Specify dtype option on import or set low_memory=False.
  df = pd.read_csv("/content/bluebook-for-bulldozers/TrainAndValid.csv")

(412698, 53)

df["saledatetime"] = pd.to_datetime(df['saledate'])
subset_df = df.query("saledatetime > '12/31/2011 0:00'")
subset_df.shape

(11573, 54)

subset_df.to_csv("salesdata.csv")

model_name = "Qwen/Qwen3-1.7B"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

I’ll wrap the model text generation code in a generate function.

def generate(prompt, enable_thinking=False):
    messages = [{"role": "user", "content": prompt}]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=enable_thinking
    )
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)


    generated_ids = model.generate(**model_inputs,max_new_tokens=32768)
    output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

    index = len(output_ids) - output_ids[::-1].index(151668) if enable_thinking else 0
    return tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

And create a simple constructor for my prompt:

def make_prompt(function_signature, user_request, data_source=None, data_dict=False, function1_output=None):
    prompt = f"""
You are given the following function and additional context.
Based on the user's request, output a python dict of the function name and arguments.
Output only a valid dict, nothing else.
Do not include function results in the dict.

FUNCTION:

{function_signature}
"""

    if data_source:
        prompt += f"""
DATA SOURCE: "{data_source}"
        """

    if data_dict:
        prompt += f"""
DATA DICTIONARY:

salesdata.csv contains the following relevant columns:
- state (str): the state where the sale took place
"""

    if function1_output:
        prompt += f"""
The following data

{function1_output}

should be an argument for the function, structured as

{{"data": {function1_output}}}
"""

    prompt += f"""

USER REQUEST:

{user_request}


OUTPUT FORMAT:

{{"function": "<name>", "arguments": {{<args>}}, ...}}

You are given the above function and additional context.
Based on the user's request, output a python dict of the function name and arguments.
Output only a valid dict, nothing else.
Do not include function results in the dict output.
    """
    return prompt

I will provide the function signature for two functions separately, as I want Qwen to use each one sequentially.

function1_signature = """
aggregated_filtered_saleprice(path_to_csv: str, groupby_fields: list[str], filter_column: str, filter_values: list[str])
   Aggregate and filter the sales data to calculate the mean sale price.
"""

function2_signature = """
generate_report(data: dict)
   The data argument should be the entire dictionary passed as a single argument.
"""

data_source = "/content/salesdata.csv"

user_request = "Using the provided tools and data source, generate an HTML report that shows the mean sale price in Alabama and Missouri."

Let’s make sure the prompts look good.

prompt1 = make_prompt(function1_signature, user_request, data_source=data_source, data_dict=True)
print(prompt1)

You are given the following function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict.

FUNCTION:


aggregated_filtered_saleprice(path_to_csv: str, groupby_fields: list[str], filter_column: str, filter_values: list[str])
   Aggregate and filter the sales data to calculate the mean sale price.


DATA SOURCE: "/content/salesdata.csv"
        
DATA DICTIONARY:

salesdata.csv contains the following relevant columns:
- state (str): the state where the sale took place


USER REQUEST:

Using the provided tools and data source, generate an HTML report that shows the mean sale price in Alabama and Missouri.


OUTPUT FORMAT:

{"function": "<name>", "arguments": {<args>}, ...}

You are given the above function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict output.
    

prompt2 = make_prompt(function2_signature, user_request, function1_output={"blah": "boop"})
print(prompt2)

You are given the following function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict.

FUNCTION:


generate_report(data: dict)
   The data argument should be the entire dictionary passed as a single argument.


The following data

{'blah': 'boop'}

should be an argument for the function, structured as

{"data": {'blah': 'boop'}}


USER REQUEST:

Using the provided tools and data source, generate an HTML report that shows the mean sale price in Alabama and Missouri.


OUTPUT FORMAT:

{"function": "<name>", "arguments": {<args>}, ...}

You are given the above function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict output.
    

content1 = generate(prompt1)
parsed_content1 = ast.literal_eval(content1)

parsed_content1

{'function': 'aggregated_filtered_saleprice',
 'arguments': {'path_to_csv': '/content/salesdata.csv',
  'groupby_fields': ['state'],
  'filter_column': 'state',
  'filter_values': ['Alabama', 'Missouri']}}

I was surprised. I did not expect the size of this model to be able to handle this. And that’s without thinking! This goes to show how far small models have advanced! Or maybe it just shows how out of touch I am with small models of this size (I spent most of my time on tiny models last year).

Let’s create a function that will execute a Python function given JSON arguments. Then I’ll define my aggregate filtered sale price function and call it with the parsed content.

def call_func(name, arguments):
    f = globals()[name]
    return f(**json.loads(arguments) if isinstance(arguments, str) else arguments)

def aggregated_filtered_saleprice(path_to_csv, groupby_fields, filter_column, filter_values):
    """
    Aggregate and filter the sales data to calculate the mean sale price.
    inputs:
        path_to_csv: str
        groupby_fields: list of str
        filter_column: str
        filter_values: list of str
    output:
        json: str
    """
    df = pd.read_csv(path_to_csv)
    data = df.groupby(groupby_fields).agg({"SalePrice": "mean"}).query(f"`{filter_column}` in @filter_values")
    return data.to_dict()

function1_output = call_func(parsed_content1['function'], parsed_content1['arguments'])
function1_output

{'SalePrice': {'Alabama': 37775.08474576271, 'Missouri': 27467.972350230415}}

Beautiful. It works.

Now, let’s see if Qwen can do the same with the second function.

prompt2 = make_prompt(function2_signature, user_request, function1_output=function1_output)
print(prompt2)

You are given the following function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict.

FUNCTION:


generate_report(data: dict)
   The data argument should be the entire dictionary passed as a single argument.


The following data

{'SalePrice': {'Alabama': 37775.08474576271, 'Missouri': 27467.972350230415}}

should be an argument for the function, structured as

{"data": {'SalePrice': {'Alabama': 37775.08474576271, 'Missouri': 27467.972350230415}}}


USER REQUEST:

Using the provided tools and data source, generate an HTML report that shows the mean sale price in Alabama and Missouri.


OUTPUT FORMAT:

{"function": "<name>", "arguments": {<args>}, ...}

You are given the above function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict output.
    

content2 = generate(prompt2)
parsed_content2 = ast.literal_eval(content2)

parsed_content2

{'function': 'generate_report',
 'arguments': {'data': {'SalePrice': {'Alabama': 37775.08474576271,
    'Missouri': 27467.972350230415}}}}

def generate_report(data):
    """
    Generate an HTML report from dict.
    inputs:
        data: dict
    output:
        None
    """
    html = f"<h1>Data Report</h1><br>{data}"
    with open("/content/report.html", "w") as f:
        f.write(html)
    print("HTML report generated")

call_func(parsed_content2['function'], parsed_content2['arguments'])

HTML report generated

Amazing!

If we use a different user request, does this still work? Let’s first wrap this all in a function.

def data_analysis_agent(user_request, data_source):
    prompt1 = make_prompt(function1_signature, user_request, data_source=data_source, data_dict=True)
    print(prompt1)
    content1 = generate(prompt1)
    parsed_content1 = ast.literal_eval(content1)
    function1_output = call_func(parsed_content1['function'], parsed_content1['arguments'])
    prompt2 = make_prompt(function2_signature, user_request, function1_output=function1_output)
    print(prompt2)
    content2 = generate(prompt2)
    parsed_content2 = ast.literal_eval(content2)
    call_func(parsed_content2['function'], parsed_content2['arguments'])
    return "Done"

user_request = "I need to know what the mean sale price was in the two states of Alabama and Missouri."

data_analysis_agent(user_request, data_source)

You are given the following function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict.

FUNCTION:


aggregated_filtered_saleprice(path_to_csv: str, groupby_fields: list[str], filter_column: str, filter_values: list[str])
   Aggregate and filter the sales data to calculate the mean sale price.


DATA SOURCE: "/content/salesdata.csv"
        
DATA DICTIONARY:

salesdata.csv contains the following relevant columns:
- state (str): the state where the sale took place


USER REQUEST:

I need to know what the mean sale price was in the two states of Alabama and Missouri.


OUTPUT FORMAT:

{"function": "<name>", "arguments": {<args>}, ...}

You are given the above function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict output.
    

You are given the following function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict.

FUNCTION:


generate_report(data: dict)
   The data argument should be the entire dictionary passed as a single argument.


The following data

{'SalePrice': {'Alabama': 37775.08474576271, 'Missouri': 27467.972350230415}}

should be an argument for the function, structured as

{"data": {'SalePrice': {'Alabama': 37775.08474576271, 'Missouri': 27467.972350230415}}}


USER REQUEST:

I need to know what the mean sale price was in the two states of Alabama and Missouri.


OUTPUT FORMAT:

{"function": "<name>", "arguments": {<args>}, ...}

You are given the above function and additional context. 
Based on the user's request, output a python dict of the function name and arguments. 
Output only a valid dict, nothing else. 
Do not include function results in the dict output.
    
HTML report generated

'Done'

Cool! It wasn’t very challenging because my user request was very similar, but as a trivial example, I’m satisfied.

The unstructured-to-structured and structured-to-unstructured framing came from Evan Balzuweit during a chat I had with him earlier this year.

Other references:

Wolfram:

I think the thing to realize about AIs for language is that what they provide is kind of a linguistic user interface.

Lawrence:

This means that LLMs provide a new interface between our thinking and the digital representations that they can assimilate

In this article, I’ll to explore LLMs as an interface in the following workflow:

human natural language input → LLM parses into structured data → pass it to deterministic executable scripts → LLM parses structured data into natural language output to human

subset_df = df.query("saledatetime > '12/31/2011 0:00'")
subset_df.shape
>> (11573, 54)

def aggregated_filtered_saleprice(
    df, 
    groupby_fields, 
    filter_column, 
    filter_values
):
    data = (
        df.groupby(groupby_fields)
          .agg({"SalePrice": "mean"})
          .query(f"`{filter_column}` in @filter_values")
    )

    return json.dumps(
        {
            "return_data": data.to_json()
        }
    )

def generate_report(json_data):
    html = f"<html><body><h1>JSON Data</h1><pre>{json_data}</pre></body></html>"
    with open("/content/report.html", "w") as f:
        f.write(html)
    print("HTML report generated")

data_dictionary = """
    salesdata.csv contains the following relevant columns: 
    state (str): the state where the sale took place
"""

runner = client.beta.messages.tool_runner(
    max_tokens=1024,
    model="claude-sonnet-4-6",
    tools=[aggregated_filtered_saleprice, generate_report],
    system=data_dictionary,
    messages=[
        {"role": "user", "content": "Using the provided tools and /content/salesdata.csv, generate an HTML report that shows the mean sale price in Alabama and Missouri. Do not respond with any other text."},
    ],
)

messages = []
for message in runner:
    messages.append(message)

Suppose two people are walking through a garden. One asks the other,

“What do you see?”

In that second, a billion bits of information hit their retina. Ten million of those flood their brain via the optic nerve and of those, they’re conscious of only a dozen bits.

“Gah! Some of the roses are dying. We missed the full bloom again!”

Technically that might be true, but what’s more informative is not what that says about the garden but what that says about what they see in the garden and what that means to them.

In data science and machine learning, looking at data works the same way.

Suppose you have the following dataset, from a pet food production facility’s conveyor belt logs (which I generated synthetically using this notebook).

Raw data is a clue about what the data collector cares about. It’s the answer to the question:

What does the business see?

Of the billion bits of information available to the business, these 8 columns, containing dozens to thousands of bits each (based on data type) is what gets captured.

We see that a conveyor belt STARTs, ENDs and STOPs, and that STOPs are always associated with a failure_type. Is that always the case? We should confirm that with a production SME.

Looking at data also tells us what’s not there. In this case, I always ask the most obvious questions possible.

For example, the expected_pkgs and expected_lbs columns beg the questions:

• What’s the relationship between the two?
• Does expected_lbs = expected_pkgs x package weight?
• Where do we store data on package weight?
• Should we assume that all packages are the same weight for each product ID? Maybe. But perhaps product_id actually means product_category_id like a particular flavor of kibble that might come in both 8 lb and 25 lb bags.

Looking at data will naturally cause us to form hypotheses on the most important question:

Why does this data matter to the business?

An amount of production is “expected”, and the “actuals” are sometimes different. There are various STOP events and each is associated with a non-blank failure_type. The business is likely trying to minimize the reduction in expected production, and to help us help them, have provided us with this data.

We can prepare further insightful questions:

• Is the eventual goal to minimize the reduction in expected production? If so, by when? What happens if you aren’t able to?
• What are you expecting us to deliver towards that goal?
• What physically happens during a BLOCKAGE or SPILLAGE? How long a delay does that cause?
• What qualifies a package as low or high QUALITY? Do products get taken off for inspection during stoppage?
• How do delays affect upstream manufacturing and downstream shipments?
• Can you show us some pictures or videos of the production runs in process?
• Who controls the conveyor belt settings? What settings are configurable?
• What other data do we have available?
• What have you already tried?
• What solutions are feasible to implement today? This month? This year?

A clarifying piece of advice I received earlier this year:

The goal of the machine learning scientist is to quantify uncertainty for the business in some way

With a myriad of options available and a myriad of constraints, data can help us serve this goal. To have any chance at doing so effectively, and hopefully efficiently, we have to start by sitting down to look at the data.

Earlier this summer, I had my first sensory deprivation tank experience. As I was considering booking a session, I came across a number of reviews describing a variety of experiences. Some folks slept better than they ever had before, some had hallucinatory experiences, some had out-of-body experiences, and so on. I honestly didn’t know what to expect.

The tank is pitch black (so much so that I saw nothing with my eyes open or closed) and is filled with about 10 inches of water with hundreds of pounds of Epsom salt. My ears underwater; all ambient sound was muffled.

It’s hard to track time in that environment. The first 10 minutes or so, I started getting bored and restless to the point where I was considering standing up. But I let it pass. I would feel my limbs twitch and would unnecessarily adjust my body, causing me to float about and bump into the walls. Eventually, I settled.

I imagined “this is what being in outer space probably feels like”. A sense of expansiveness. I felt like an astronaut on Spaceship Earth.

I went in and out of almost-sleep, maybe twice.

Maybe halfway through the 90 minutes, I thought something like: this is truth. This is true reality. I was fully present and fully experiencing the absence of sensory stimuli. It was a rich and full-bodied experience.

At 90 minutes, soft music started playing. I opened my eyes and I turned on the underwater lights. They were blinding. I sat there, head buried in hands, surrending to the inevitable exit from peace.

I gathered my things and exited the chamber, disoriented upon re-entry into ambient light and sounds. I had an awkward exchange with the polite attendant.

I pushed the front door open, and stepped out onto the street, back into the fully lit environment.

I felt a billion bits of information violently crashing into my eyes. I immediately shut them and took a step back.

As I adjusted and looked around at the billboards, cars, and storefronts, I felt like I was on a movie set. Suddenly, everything made sense.

In the following weeks, I had a simple technique to find my way back into that state of nothingness: close my eyes. Had a stressful call? Close my eyes. Tired of staring at the screen? Close my eyes. Overwhelmed by the sun? Close my eyes.

As the weeks have progressed, surprisingly, that is no longer effective. Perhaps my baseline of stimulation has risen too far above nothingness. Instead, I find that leaning on my back and staring at the ceiling gives me a similar relief. I empty my cache, I go on standby, and I’m just present with myself as I would be with a friend. Here we are, just staring at the ceiling.

Even more surprisingly, I haven’t felt the urge to go back. I’m sure I will in the future, but something in me doesn’t want to rely on those conditions to feel that experience. I want to bring that experience out into the well-lit environment.

You may know it by the more colloquial term “bell curve”.

Many things are (almost) normally distributed: population human height, population blood pressure, birth weights, annual rainfall, standardized test scores, shoes sizes, body temperature…the list goes on.

I’ve spent about a dozen or so hours across three or four Claude Opus 4.6 High conversations this year thinking about and understanding the normal distribution. This series will give me an opportunity to share what I’ve learned and by extension, how I’m using the Normal Distribution as a metaphor for my life and career.

While there are numerous mathematically heavy books on the topic, I wanted to find something more intuitive and accessible for non-mathematicians to guide my thinking and writing. Luckily, Claude found me this site: https://longintuition.com/2020/07/20/max-entropy-intuition.

In this article, I want to focus on logarithmic thinking, but instead of telling you, I want you to first experience it with me. It’ll feel silly but bear with me.