Automation with LLMs is non-deterministic and jagged. Can that even be considered automation and is that useful?
LLMs are non-deterministic, the same inputs will lead to a different output every time. LLMs are also jagged, some scenarios work great, others are disappointing to say the least, and which is which can change even without the model number changing. A tricky foundation to build upon.
[...]
For automation to be automation it has to "reduce human intervention" and we have to "predetermine" what has to happen. What makes LLMs so novel is that "predetermine" no longer has to be hard rules that are specified ahead of time.
[...]
So can you build useful automation with LLMs? Yes, but first you need to spend the time to "predetermine" aka constrain what the beneficial actions are, which ones trigger alerts and which ones are plain refused. If you cannot do that for your domain, then you cannot automate it.
Second you need a domain that does not need reliability. That makes it a bad fit for medical or health decisions, financial or legal decisions, and basically any decision with serious consequences.
Third, be wary of using LLMs to generate "suggestions" for a human to "sign off". This will eventually lead to review fatigue where the human just approves whatever the LLM generated. This will also lead to skill atrophy where the human eventually loses the skills to make these decisions himself.
Fourth, it might be way cheaper to write your automation as deterministic code.
So can you build useful automation with LLMs? Yes, but first you need to spend the time to "predetermine" aka constrain what the beneficial actions are, which ones trigger alerts and which ones are plain refused. If you cannot do that for your domain, then you cannot automate it.
Second you need a domain that does not need reliability. That makes it a bad fit for medical or health decisions, financial or legal decisions, and basically any decision with serious consequences.
Third, be wary of using LLMs to generate "suggestions" for a human to "sign off". This will eventually lead to review fatigue where the human just approves whatever the LLM generated. This will also lead to skill atrophy where the human eventually loses the skills to make these decisions himself.
Fourth, it might be way cheaper to write your automation as deterministic code.
Lorand Kedves
First, 100% agree but I don't think you got the point.
The aim of LLMs is not automation but "intelligence", we do not check (or even, like) the question if it is useful, but cool and exciting and in some environments, dangerous (not the wasted resources, pollution etc. but the "emerging god" 🤦♂️ ).
This is how LLM lords "make profit" (the ultimate definition of being useful) - or to be more precise, fake profit. And as we go forward in this IPO game, this will be their problem, not the fact of just destroying the human civilisation. (John von Neumann, Can we survive technology? - 1955)
Otherwise, people would read the definition of the Turing test, not just recall what they felt when watching Her, Ex Machina, Terminator etc.
Just as predicted.
Richard Gross
You 100% agree with my point but I don't get the point? I think I am quite close to getting the point: https://richargh.de/posts/Fifty-First-Dates-with-AI
Lorand Kedves
I don't want to make this a competition (folks here report my comments and LinkedIn moderates me), but you define yourself as an IT Archaeologist - here is what it means to me.
https://lnkd.in/duJEbZty
The conclusion is that LLMs are futile, a catastrophic answer to an already bad question.
1: Transformable information (JCR Licklider, Libraries of the Future, 1964) is a graph.
2: An information system (including the DSLs, algorithms and state / transaction representation) IS transformable information.
-> working with them is NOT text but graph management.
https://lnkd.in/dTCD8dBv
I build this system. Transparent, deterministic, reliable, dirt cheap, platform independent, open source, Apache 2. The only problem is that this is "impossible", and against the business model of IT. ⬇️
'It is difficult to get a man to understand something, when his salary depends on his not understanding it.' (Upton Sinclair)
----
History
2011: I summarised my disappointment of 20+ years of coding with text.
https://lnkd.in/dy5Tnm3n
2018: Based on my experience using knowledge graphs while building real-life IT systems (startup, multinationals, government, academic research), tried PhD, rejected.
https://lnkd.in/ddue7rH3
Want more? 🙂
Richard Gross
I was not aware we were in a competition. Do you mind going back a couple of steps? You said you agree with me but then you added some words that make it seem like you don’t. So do you agree with my post, and if not, what exactly do you take issue with?
Lorand Kedves
I agree with the expectations against any information system: it should be deterministic, transparent, reliable. When we won Comdex'99 with DataScope, a data mining/AI tool, we wrote all machine learning components from math papers. The goal was "explainable AI" that we delivered.
LLMs can't do that.
The "human in the loop" is a patch for responsibility (because at some point, mistakes become legal issues, see how Copilot got its disclaimers, how Tesla sneak out of all lawsuits and then Musk personally destroyed regulatory systems in doggy etc.), they should use more AND less tokens, train their replacement, etc.
LLMs are dead end.
You are an IT Archaeologist, so let's step back 60 years, shall we?
Have you read and understood Man-Computer Symbiosis from JCR Licklider (1960), Augmenting Human Intellect from Douglas Engelbart (1962), or are you aware of Joseph Weizenbaum's or Neil Postman's problems with this technology?
Because there is nothing new in what either of us are typing here. Which begs the question: why do we spend time on this discussion?
At this point people think I take this as a competition, but to me, this is rather "do you know what you are talking about?"
Which is not better of course... 😊
Lorand Kedves
I may sound rude, but there is a reason for that. If you stop complaining and try to act, you should be aware that the whole system will be your enemy. This is not a game or a topic for a TED talk.
https://youtu.be/9vz06QO3UkQ
I am interested in this conversation if you are aware of these things and decided to continue. Otherwise, consider me as a troll and forget about this. A few days of LinkedIn guano covers everything... 😉
Cheers mate.
