And yet, they get shit on for releasing it at all (never mind in a way they knew perfectly well would leak), while no one ever seems to remember all of the other models which didn't get released at all... And ironically, Google is over there releasing Flan-T5 under a FLOSS license & free to download, as it has regularly released the best T5 models, and no one notices it exists - you definitely won't find it burning up the HN or /r/ML front pages. Suffice it to say that the developer community has never been noted for its consistency or gratitude, so optimizing for that is a mug's game.

(I never fail to be boggled at complaints about 'AI safety fearmongering is why we had to wait all these years instead of OA just releasing GPT-3', where the person completely ignores the half-a-dozen other GPT-3-scale models which are still unreleased, like most models were unreleased, for reasons typically not including safety.)

We needed scientists but we got parrots

Gonna check this out when I get home. Thanks!

I mean it's probably running on a single 4090 from one of the PhDs personal setup. Just wait for someone to replicate and release an actual model

If FB released this under a more permissive license they would've gotten so much goodwill from the developer community =/

I give you a simple solution: run GPT-3 and LLaMA in parallel, if they concur, then you can be sure they have not hallucinated the response. Two completely different LLMs would not hallucinate the same way.

It's not my work, so I can't answer your questions. Helpfully the authors see this post and can answer your questions.

>Won't it upset Ukranians that want to use such a model to help write about the war when they find out Russian law applies to their country?

Unless there's been a major movement in the war since I last checked the news, Ukraine is not part of Russia.

What you're describing sounds like a single universal AI that looks up local laws and follows them blindly.

I think what's going to happen is that each country will train their own AI that aligns with their local laws and values. A US or European AI would have no problem criticizing the Russian government or writing pro-LGBT text. But it would be banned in Russia and Saudia Arabia, and they would have their own alternative.

Very impressive work, thank you very much for sharing.

I have a few question regarding the training precedure:

• did you train using a next token prediction scheme or something else?
• do you think RLHF would further improve the model using your instructions?
• why did you choose to do the differentiation between Instruction and Input?
• How do you create the string the model is trained on? just concat Input and Instruction?

Thank you very much

In some countries pro-LGBT writing is illegal. When a censored model is released that can't write anything pro-LGBT because it's illegal somewhere, don't you think there would cause quite an uproar, quite a ruckus?

In Russia it's illegal to call their invasion of Ukraine a war. Won't it upset Ukranians that want to use such a model to help write about the war when they find out Russian law applies to their country?

Yes, they mean censorship. Nobody has ever provided a definition of what "safety" is in the context of a large language model. From use of other censored models not even the models know what safety means. ChatGPT happily described the scene from The Lion King where Scar murders Mufasa and Simba finds his dad's trampled body, but ChatGPT also says it can't talk about murder.

From what I have gathered from the vagueness on safety I've read from LLM developers, that scene would be considered unsafe to them.

Humans aren't going to have perfect laws everywhere, but it's still not the AI's place to decide what's right and wrong.

In practice, AI that doesn't follow local laws simply isn't going to be allowed to operate anyway.

In some countries it's illegal to say anything bad about the head of state. Should large lanague models be prevented of saying anything bad about heads of state because it breaks the law?

Squeeze that paper

It reminds me of the 90's when hardware became obsolete in under a year. Everybody moved so fast with large lanague models that they hit hardware limitations very quickly, and now they are working on efficiency. This also reminds me of computers when they moved to multi-core processors and increasing work per clock rather than jacking up the frequency as high as possible.

If I live to see the next few years I'm going to wonder how I managed to use today's state of the art text and image technology. That reminds me of old video games I used to love, but now they are completely unplayable.

Holding onto my papers!

Backpropagation requires a lot of accuracy so we need 16- or 32-bit while training. However, post-training quantization seems to have very little impact on the results. There are different ways in which you can quantize but apparently llama.cpp uses the most basic way and it still works like a charm. Georgi Gerganov (maintainer) wrote a tweet about it but I can't find it right now.

'So as your consumer electronics adviser, I am advising you to donate your current VCR to a grate resident, who will laugh sardonically and hurl it into a dumpster. Then I want you to go out and purchase a vast array of 8-millimeter video equipment.

... OK! Got everything? Well, too bad, sucker, because while you were gone the electronics industry came up with an even newer format that makes your 8-millimeter VCR look as technologically advanced as toenail dirt. This format is called "3.5 hectare" and it will not be made available until it is outmoded, sometime early next week, by a format called "Elroy", so order yours now.' -- Dave Barry, "No Surrender in the Electronics Revolution"

[deleted]

[removed]

Source code is also the same. Nothing changed.

Finally the instruct versions are prepended with "text-"

Claude is the informal name for Anthropic-LM v4-s3 (52B)

[removed]

https://blog.eleuther.ai/gpt3-model-sizes/