Last month I rebuilt my pool's plumbing system with the help of YouTube and GPT. I was quoted $6k and it ended up costing me about $1k of materials and a few days of work. This was my first exposure to any kind of outdoor plumbing, I'd never touched PVC before.
It's unlikely I would have had the confidence to do it from YouTube alone, specific diagnostic help and a full diagram to work off of was extremely helpful. I had it prepare an SVG of the whole assembly with all the measurements and parts labeled.
I'm glad it went well for you. But I run a not-for-profit pinball museum, and I have specifically had to ban volunteer repairpeople from using the chatbots because it will confidently propose idiotically wrong solutions. Which will then get proposed to me, or just directly applied, with equal confidence. At this point I'd rather hear, "My horoscope said..." than "ChatGPT said..."
I expect that it's good for common use cases that's well documented. But then, so are a lot of other approaches.
Yeah, I would imagine pinball machines are at least an order of magnitude more complicated to maintain than above-ground pools are. That situation sounds really annoying, I feel for you.
For sure, but I think the deeper problems are how much the industry changed over the decades and the extent to which pinball repair content comes from amateurs opining on forums.
Pool technology has been more stable, and there are more people out there writing well-informed content for LLMs to extract and present as their own.
Two years ago it would have been insane to say that you got help from ChatGPT to fix your outdoor plumbing, people here would have been frothing at the mouth for merely suggesting it. Four years ago it wasn't even on the radar of future possibilities.
The quality it is today, is the worst it's ever going to be.
Maybe? It's a plausible theory. But these operations are all wildly unsustainable financially at the moment, and it's not clear where they'll get future data from, having destroyed a lot of the incentives that generated their current source content.
Bubbles are not a great time to form intuitions. WebVan [1] and Kozmo [2] also seemed to herald a new age. Decades later, brick-and-mortar grocery stores and convenience stores are still doing fine.
> The quality it is today, is the worst it's ever going to be.
Someone in 2004:
A few years ago it would have been insane to say that you got help from Google’s I’m Feeling Lucky button to fix your outdoor plumbing.
The quality it is today, is the worst it's ever going to be.
It's still insane to do that today. Using any LLM to get info for things is a terrible idea, because they will slip inaccurate info into the mix and you have no way of knowing where it is.
I would still call you nuts for trusting ChatGPT with learning any construction or plumbing tasks. Atleast PVC is dead simple though since you basically just glue it together.
They already have all the data, all the money and all the chips. It’s actually the best it’ll ever be, as newer models will have to start paying off all that capex, newer models will be trained mostly on slop, and the SEO and influence op leeches will have begun their arms race to insert their products and values into the training data. We’ve seen this pattern before. AirBnB, Uber, social media, streaming video, et cetera didn’t get better once the VC money ran out and they needed to start turning a profit, they got much much worse.
I have schematics as early as 1937, but the odds that the manuals stayed with the machine are pretty low. (Usually these machines were owned by somebody who had a bunch, and I suspect the manuals tended to get collected centrally.) You can generally buy manuals on eBay, and some are being reprinted by people who own the original IP.
The internet has a lot of scans of wildly varying quality. The AI industry's hunger for data means there has been a lot of progress in OCR and data extraction, so I have notions of taking something like PaddleOCR and trying to turn the scans into a cohesive reference site that will work well on phones, etc. But I'm not sure when I will get to it.
If people out there are interested in working on a project like that, let me know. Email me at william@theflip.museum.
> This was my first exposure to any kind of outdoor plumbing, I'd never touched PVC before.
I find it telling that the highest praise for LLMs comes from people using it for something where they admittedly have very little domain knowledge. Domain experts usually mention major caveats. I've been testing them on subjects where I already understand the problem well, and I've yet to see any outputs that would make me trust them on things I don't already know.
When I apply LLMs to domains where I'm already an expert, I find it particularly lacking when I want it to do deep, difficult, novel work that requires precision. On the surface, the output looks pretty amazing at first. But when I turn a critical eye to every detail, I end up finding a lot of flawed "thinking", and the lengthy process of fully understanding what it generated and cleaning it up to my standards makes me question the entire value proposition. However, I find it does a great job being a low-level automaton sort of assistant.
For instance, in the domain of software engineering: I would not trust it to implement a major architectural change, or a groundbreaking, complex new feature. I would trust it more (but not completely) on something like a refactoring that may touch thousands of lines in a fairly mechanistic way, but that was a little too-complicated for simpler tools likes regexes. While that's kind of a nifty use, I think it's fair to say that non-LLM software purpose-built for such tasks can probably do the same thing more effectively for less real cost (meaning the currently-subsidized real cost of all the training and inference power burn, etc)
That's a good point - I've heard people make the same one whenever LLMs are brought up. I hope someday you're able to get more value from them.
Anyway, my pool's looking great and I gained some new skills. I probably could have gotten there with books and YouTube alone, but having another tool at my disposal made me a bit more confident.
That's the entire problem. The main thing LLMs gave you was confidence, and the thing to understand is that the confidence an LLM gives you is often utterly false and baseless.
Why were you not confident with literal how to videos and documentation, but became confident when a chatbot generated probable text?
I remember this was brought up about a month or so on HN, in the context of describing how one person's experience using LLMs can be so vastly different than another person's:
"LLMs seem good at things you are not good at."
So, if you've never touched PVC before, LLM sounds plausibly competent--it may actually be or it may not be, but you'll walk away from it thinking you learned something. If you are a professional plumber and ask an LLM the same thing, the output will more look flawed and possibly dangerous.
Same for software writing: If you're not a good software developer, you probably think an LLM is great and writes much better code faster than a human developer can. But if you are a good software developer, LLM output is slop and requires huge rework to be passable.
> So, if you've never touched PVC before, LLM sounds plausibly competent
well to be fair, it's sounds about as competent as your average homedepot employee. He's wasn't doing something super complicated, cutting and gluing PVC for an above ground pool is very common and doesn't require a plumber. I used some youtube videos to fix my dishwasher, i didn't need a professional service agent from the manufacturer I just needed some pointers.
as for software writing, for standard everyday enterprise app work which is typically just CRUD and moving data around it works fine. That kind of software does not need to be a highly tuned work of art to meet the requirements.
LLMs seem good at things you are not good at... primarily because you lack the skill to actually judge the goodness of their work, and they present their work confidently with an air of authority.
I am not a mechanic by trade, but I know nearly everything there is to know about working on an ICE car. I did paint cars professionally for a bit.
LLMs are absolutely full of garbage advice, when I try to use it for troubleshooting. However, people who don't know anything about cars are telling me it helped them fix issues. I am assuming their issues were maybe surface level and something I would just know without even looking at any manuals, because when I use it for complex problems it just doesn't work for me.
if it helped them fix issues then it helped them fix issues. Good for them. Maybe their problems didn't rise to your bar but at least they were able to get it solved. That's very useful and empowering to people without the direct knowledge and experience themselves.
Well, I guess another plausible explanation is that usually "domain experts" are people that get paid for their expertise, and have a vested interest in saying that LLMs cannot replicate what they want people to pay them for.
It’s likely it missed some crucial subtlety that will come to bite you in the ass down the line.
That’s the thing with AI - its responses sound plausible enough to non-experts but time and time again I see experts in any given field being able to identify AI content by pinpointing subtle but crucial errors. That’s one of its dangers - it gives you enough confidence to shoot yourself in the foot.
Human workers also wreck these jobs horribly that come to bite your ass in the end. I think an intelligent person equipped with AI and common sense and real stake in the thing being well done (because it's their own, so they care) is better than whatever is possible to pay for or book in a realistic timeline from another human.
Also, ask people in the trades to review each other's jobs. They will harshly criticize each other too for missing basic things and then go on to vehemently disagree. As an outsider it doesn't mean much that an expert found some fault. They always find something to nitpick.
> It’s likely it missed some crucial subtlety that will come to bite you in the ass down the line.
Maybe, but that's part of the experience of learning. I plan to maintain pools for the rest of my life, if I made an oversight which costs me down the line then the lesson will be that much more memorable.
This is an above-ground pool with a pump and a filter, the stakes are relatively low. In the absolute worst case I could rip it all out and pay a pro to do it for the price I was quoted.
A pool filter is for removing physical debris, it's not really for preventing bacterial or fungal infections.
I guess what you're talking about is sanitizer, in my case we use chlorine. I test it every time we swim, but I wouldn't have needed an LLM for that. It's very straight forward to maintain pool chlorine, my Dad taught me that when I was 13.
I feel like if you had wanted to you could have done it before. When I got a pool in 2020,there were still old school phpBB forums out there dedicated to DIY and maintenance. That same summer, my neighbour who in no way is techie or handy redid his plumbing too.
Yeah, absolutely. The YouTube guides have all been very helpful, a lot of them are done by actual paid professionals promoting their pool brand.
The fellow who'd done it originally is my neighbor and he's a retired teacher. There's a lot of ways to learn this stuff - my one takeaway has been that it's much easier than people make it out to be.
There are just boatloads of people who say "Oh LLMs are magically good because of democratizing access to information"
Except, for people who were gently motivated, that information was already pretty well democratized by libraries. You could trivially go to the library, and get whatever books were published on a topic, even if the only copy was on the other side of the country. Tons of the famous names from previous decades got their start teaching themselves things from a book in a library. It was very common in the technical churn of the 20th century that a new project at work meant you went to the library and grabbed books on a brand new topic and self-taught. This for example is how some programmers in the 90s developed 3D engines.
There was even a short period of human history where it was common to pay a few thousand dollars for a family encyclopedia. I got my start reading an 80s encyclopedia, focusing on the more technical tomes, before I found Wikipedia. The drive to access and learn information lead to me learning about computers in a time and place where a formal education on the subject was unavailable to me. I owe my career to it.
After the existence of Ebay, a few hundred dollars could populate a shelf with the standard reference books and material for nearly any interest. All it took was a willingness to look for books, buy them, and sit down and read them.
Similarly, the internet did the same since the 90s. Specifically, it allowed for non-physical clubs to supplement the fact that not everyone lived in Silicon Valley and could access those rich clubs on niche topics. But special interest magazines were already providing some of that functionality.
The primary filtering LLMs do is provide new access and ability to people who are far too lazy and unmotivated to do the real work necessary to learn about something without being literally spoon fed.
This helps explain why the primary thing LLMs have done is increase the noise floor of information, and explains differing sentiments. People willing to put in minimal effort to learn things had zero issue learning new information in the previous regime, so aren't that impressed when an LLM regurgitates the wikipedia intro paragraph or summarizes a popular reference. They already read that. They note that the LLMs confidence is often unwarranted, and they get reasonable results because they have foundational understanding of the domain and already know which pitfalls and problems to be concerned about, and how to prompt the LLM to make the right choices.
For people who largely are unwilling to take minimum effort to learn something new, of course LLMs feel magical, because Wikipedia level introductions to topics are magic to people who aren't already seeking them out. Of course, the question is, what in the world was previously stopping you from learning new things?
I think insecurity, over-estimating difficulty / effort, and risk aversion are all to blame. I think these things would improve a lot if people have DIY tasks as part of their upbringing or education, if only to get experience with it.
But the education system - at least for me, 20ish years ago - was very much tiered or broken up into classism: if you were highly intelligent or a good learner you'd go to advanced schools where you'd get higher level math, latin, etc. If you were "dumb" you'd get taught how to do woodworking and masonry. At best I was taught how to use a figure saw, drill press safety measures, and how to patch an inner tube (welcome to the Netherlands, this is very important. Or, was, it's much simpler and cheaper to just buy a new inner tube nowadays).
Still can (libraries still exist plus the internet) but finding the online resources gets more and more difficult; the fact the author mentioned youtube videos instead of someone's old-internet style "all about pvc pipes" website [0] or comic sans plumber sites [1] is already telling.
I'm not sure if the sites you linked would have been immediately very helpful in my specific case. You know this is for a pool, right?
The first is called "Everyday Uses for PVC Water Pipe" and has some cool ideas like using PVC for Wiimote holders or tridents, but I don't think that's super relevant for this project.
The second is a great forum which I'm already familiar with, but again, the topic is household plumbing and from a brief skim, none of the topics mention pools. I think mine would have been out of place.
Or they just defaulted to those options. People don't always use the best option, they stop looking when they find something that works good enough to their eye.
Definitely! My understanding is everything I did has been pretty well established pool maintenance for decades now. I'm sure there are loads of good books on the topic, though I'm a good 45min from the nearest library so it wouldn't have been my first choice.
I've love to know more about how you did this. Was it the underground plumbing or your pump system above ground. This is one of the few areas of the home I have little understanding of.
This was for a largish (40,000L) above-ground pool with no hookup to my home's plumbing system. The water was all pumped in from a water truck.
The previous system was also installed by a non-professional and was mostly tubes. It leaked to all hell and looked generally redneck and awful.
First step was draining the pool and doing a nice deep clean. Then I ripped out all the original plumbing until it was just the pool outlets, pump, and filter.
I arranged it all and measured the dimensions. I fed the figures along with a tonne of photos and explanation to GPT. I spent a while talking pros/cons and landed on a design which lined up with what I'd seen on YouTube. I had it prepare me a full shopping list of PVC, tools, cements, etc. all linked to a local pool dealer. I picked it up the next day.
The PVC was all cut with a chop-saw then primed and cemented together. I found this part easier than I would have expected. I put a layer of TigerFlex hose between the PVC manifold and the pump/pool/filter inlets so it had some tolerance.
We've been swimming in it all summer, no issues whatsoever so far.
> It's unlikely I would have had the confidence to do it from YouTube alone, specific diagnostic help and a full diagram to work off of was extremely helpful.
People have been DIYing swimming pools for decades with the help of… books:
The internet at large, including all those posts where someone responded 'that's wrong, it's actually' but as an algorithm can't actually make sense of that it will happily regurgitate misinformation.
I answered something similar above, but I'll paste it here too:
This was for a largish (40,000L) above-ground pool with no hookup to my home's plumbing system. The water was all pumped in from a water truck.
The previous system was also installed by a non-professional and was mostly tubes. It leaked to all hell and looked generally redneck and awful.
First step was draining the pool and doing a nice deep clean. Then I ripped out all the original plumbing until it was just the pool outlets, pump, and filter.
I arranged it all and measured the dimensions. I fed the figures along with a tonne of photos and explanation to GPT. I spent a while talking pros/cons and landed on a design which lined up with what I'd seen on YouTube. I had it prepare me a full shopping list of PVC, tools, cements, etc. all linked to a local pool dealer. I picked it up the next day.
The PVC was all cut with a chop-saw then primed and cemented together. I found this part easier than I would have expected. I put a layer of TigerFlex hose between the PVC manifold and the pump/pool/filter inlets so it had some tolerance.
We've been swimming in it all summer, no issues whatsoever so far.
There's plenty of value from LLMs, if you treat them as an advanced search engine and auto complete machine like they are. I've used them quite heavily to research things...but these are done best in the hands of skeptical people.
The problem as usual is the mass of rich people trying to profit off of them, not the technology itself.
I’ve been programming professionally for 29 years and with the right constraints (strongly typed language, strict linting, adversarial review styleguide, test coverage, clearly defined spec or requirements) it does indeed regularly write good code. All of the other stuff required is either set-once-and-forget or is also AI, and it does require supervision, but it writes good code the vast majority of the time.
Note that I am talking about frontier models and only the last 6-12 months. Opus was really the breakthrough point for me.
I’ve been programming professionally for 31 years. With all those guardrails it writes functional code - but I would still not call it good. But hey to each their own.
I think of the distinction here is about the domains we're coding within. If your software projects are product-y CRUD apps/sites... well, LLMs will tend to perform decently at that, because it's fundamentally pretty trivial work anyways, and there are so many examples to draw from. All CRUD apps are essentially isomorphic, with just some rules to plug in about data validation, business logic, etc. On the other hand, if your software projects involve more-difficult subject matter, you're going to find it struggles a lot more.
You don't have to let it design the architecture. Personally, when I'm using AI: I design the software, and the AI implements it. The classes and functions are typed the way I want them to be typed. AI does a great job.
Yeah if you just let it run wild, it will produce subpar results. But that is not much different from humans tbh.
When you discuss design and architecture first, and write that out in a design doc or something along those lines, it works quite well for the most part.
And 9 out of 10 times when it produces some poor results, just asking "is this really a good approach?" or just stating "This code makes me very sad" it will most of the time do a really good job of analysing why that code is bad and how to improve it.
I was talking about style and architecture. In all these cases the code is working as intended either way. There is no such thing as "correct" style and architecture. There are only tradeoffs. You should know that if you have 31 years of experience.
> There is no such thing as "correct" style and architecture. You should know that if you have 31 years of experience.
And you should know that this is not accurate. But at this point this has devolved into a dick measuring contest, which I refuse to do. Have a good day.
For well-scoped tasks I wouldn't say the code is bad, not brilliant for sure, but definitively good enough.
For a lot of problems "quick and good enough" is all that is required. I've used it a lot for managing my Home Assistant setup. Has saved me countless of hours.
In principle I could've done it myself, but I never would have, the time investment required to learn it wouldn't have been worth the value I get from it.
The point of good architecture is that when your requirements change, it's easy to change your code to accommodate them. But with AI, it's always easy to change your code to accommodate changing requirements. So the impact of bad architecture is low and you can focus on solving today's problems instead of tomorrow's problems.
AI can make fully functioning computing solutions. That's just a fact and denying it is like denying that a bicycle can roll and only four wheeled vehicles can.
Or saying that a car is a better vehicle than a bicycle. That is probably true, but many times for many people a bicycle is all they can get.
Glad you went for this analogy. It’s like an AI producing a board with a beach umbrella nailed on it and four wheels, and saying “meh it’s good enough for transportation, human car designers are doomed”. It’s functional, right?
It’s good at all kinds of stuff that is search adjacent, “find amesent parks with water feature within 100 miles of me that have rv parking nearby”, etc
select l.id, l.name
from locations as l
where l.type = 'amusement_park'
and exists (
select id from locations as l2
where l2.type = 'rv_parking'
and distance(l2.geo, l.geo) < $nearby_distance
)
But what we should have is a good map software where you could filter by type and distance (How many amusement parks can be in that circle?) and quickly check if there's a RV parking nearby.
I installed Ubuntu on an old laptop that I let Claude Code sysadmin. It makes it really easy to self-host open source stuff, and if there are issues it can fix them too.
Funnily enough, I've had a somewhat mixed-to-hostile response when trying to upstream the vibecoded fixes, so I suspect using an LLM to fix broken open-source software (that human maintainers don't have the time to fix themselves, nor the humility to accept an LLM-authored fix) will become more of a thing going forward too.
Oh, it's also been identifying a bunch of patterns in sales data for my business that has been increasing monthly profit consistently since last November (around $4,000 USD, every month, cumulatively so far with no sign of slowing down - could easily be $10k/mo in increased gains by end of financial year).
It can be very exhausting to receive a lot of PRs. especially when they're LLMs. We have to read what people don't often even read themselves. They're often very wordy. Then it hurts all the more when it's wrong.
I've had negative responses to small, isolated submissions that I've heavily QA'd and semi-positive responses to longer ones. I think it has less to do with the "realpolitik" of the code itself and more to do with the maintainer's viewpoints about whether AI as a whole is a positive or negative thing.
I tend to respond quite well to AI-authored or assisted PRs to my project, but to be fair we maybe only get 3-5 PRs in a good month.
I tend to value person-to-person interaction. I've received a lot of purely automated responses, which shows low value to me as a person, and ultimately the project. I'm not disagreeing that code matters, but we're losing person-to-person discourse and the community that comes with it.
Thinking it through a little bit is probably coming from having bounties on a few issues that might be contributing to my negative experience.
