this is the key. Even the ancient Greeks knew that the most dangerous guy is the one that doesn't know what he doesn't know, and AI has no clue what it doesn't know. OK, I'm a Chem - I dabble in electronics and mechanical, and I could have generated that drawing for you, but I damn well know I don't know the ASME codes so that is the first place I know I have to go.
OK, so you train you next model on engineering codes....But you are always going to be one step behind the curve aren't you? I'll grant that doesn't mean you might not at some point be better than the average joe with a BS from State U, but how does the user ever know enough about the model (which is the ultimate black box) to have a level of confidence in it for things that are critical? To me that remains the missing link. Now I can believe profit motive may drive practice to accepting those outcomes without ever answering that question, but that's not a design world I want to live in - but since we are both old either way we probably won't have to!