I've been playing with ChatGPT 4 a bit. In roughly 70% of the cases, its output is either better or equivalent but much faster than what my best output would be. In around 20 to 25% of the cases, it is definitely worse. Sometimes obviously worse, sometimes perniciously worse, in the sense that its output is flawed, but not obviously so. In roughly 10% of the cases, it hallucinates or goes outrageously wrong. From a productivity point of view, that is not optimal. Yes, I can see huge gains in this 70% of excellent results. Yes, I can iterate or intervene to solve a lot of the 20-25% "worse" cases but the non-obviously wrong answers can be tricky to investigate. At this point, the lottery aspect is the most annoying in practice, but also the most reassuring for us humans.
As far as future progress is concerned, there is a serious concern - the topic of several recent academic papers - which is that we will run out of training data.
The most worrying thing in the long term (whatever that means if progress continues to accelerate), as far as I am concerned, is we don't understand why the process works so well and we start noticing unexpected emergent capabilities (see the widely publicized 154 pages or so "tech" report released by OpenAI.
Even if we don't exactly know how it happened, our current conscience, intelligence, and abilities ultimately emerged out of a soup of "organic" (by definition) molecules and an evolutionary process. We are plenty scary ourselves already. And, even if one dismisses the idea of our spontaneous emergence, for example for religious reasons, the problem remains identical: if divine intervention somehow bootstrapped us out of the soup what are we doing now if not bootstrapping something potentially as radical?
On the "plus" side for us humans, we can count on the tech industry to over-hype intensively. When and if the shit really hits the fan, I think (hope?) I will be already dead.