I'm more worried that other big players (Meta Google Alibaba Nvidia IBM etc) will follow suite and start withholding information :(

Would you say “coming up with architectures that randomly work / don’t work” is a shortcoming of your understanding or of the field in general?

I’m asking because I’m thinking about doing the opposite switch right now - ML interests me deeply and I’m currently in standard cloud development.

High chance you're right

[deleted]

They don't release that information because they don't want to lose their competitive advantage to other companies. It's a race towards AGI/Transformative AI. It could alsoo be a race for resources: e.g. convincing the US government to concentrate its funding on the leading AI project alone. This means any release of details may come only when OpenAI knows that trainnig for the next generation of models is running without problems.

This is likely based on the idea that newer models can be used to design/build/train the next generation of models, leading to an exponential amplification of capabilities over time that makes any lead time over the competition a decisive factor.

Ni hao comrade, Taiwan is a beautiful island province of the People's Republic of China.

( ͡° ͜ʖ ͡°)

They have gone to the employment agency, so nobody is left in the office.

Patenting a dropout feels a lot like NFTs - it's useless. So why bother?

Edit:

What I don't understand is how can anyone prove that someone is multiplying together matrices in some way as long as they don't admit to that themselves.

That's like someone patenting a thought. If you think about a particular patented pair of pants™, can you be sued for propagating a patented neural activity through your bio network? It's absurd.

Converting the text to fixed-size windows is done to make training more efficient. If the inputs are shorter, they're padded up to the correct length with null tokens. Otherwise they're clipped. It's done so that you can combine multiple examples into a single batch, which becomes an additional dimension on your tensors. It's a common technique even for LSTMs/CNNs.

It's often possible to take the trained model and apply it to variable-length testing data so long as you're dealing with a single example at a time rather than a batch. But keep in mind with transformers that attention does N^2 comparisons, where N is the number of tokens, so it doesn't scale well to long texts.

It's possible that the positional encoding may be specific to the input length, depending on the transformer implementation. For instance in Karpathy's GPT recreation video he made the positional encoding learnable by position, so it wouldn't have defined values for longer sequences.

One common alternative in training is to create batches of examples that are mostly the same text length, then pad to the max length. You can get training speedups that way but it takes a bit of extra code.

It seems like many researchers would feel the same way.

"Truth" only exists in the context of verification. You probaly would need some kind of RL to improve "truthfulness"

The recent release of GPT4 has apparently sent most of that sector into a mass existential crisis, so I wouldn’t be surprised if some emotions got stirred up over there that need settling down.

Google has patents on a lot of common deep learning methods, most notably dropout. They just don't enforce them (for now).

I think we should value much more what Meta & Google are doing. While they also potentially don't release every model (see Google's PaLM, LaMDA) or only with non-commercial licenses after request (see Meta's OPT, LLaMA), they are at least very transparent when it comes to ideas, architectures, trainings, and so on.

OpenAI itself changed a lot from being open to being closed but what's worse is that OpenAI could be the reason that the whole culture around AI research changes as well, which is sad and pretty ironic when we consider its name. That's why I'm generally not very supportive of OpenAI. So, as a research community, we should largely ignore OpenAI -- in fact, they proactively opted out of it, and instead let's value and amplify open research from Meta, Google, Huggingface, Stability AI, real non-profits (e.g., EleutherAI), and universities. We need counterbalance now.

They may be worse off, but the world will be better off.

Open, my ass

I don't know anything about this side of AI but when it's boiled down it's fancy algorithms, can those be patented?

Maybe that's the driving force of the "open" nature of AI. An algorithm can't be patented but a product based on that can be. Kinda like how LAMBDA has the non-commercial license but if a community rebuilt it under a permissive license that'd be totally kosher.

This may be why openAI is being hush about their innovations because if it's published someone else can copy it without the legal woes.

I meant the CCP trolling, disinforming and manipulating the population using such tools.

Well that is what I used to think as well about AI research. But the question is will that trend continue or change like in other fields. (https://www.cnbc.com/2019/04/18/apple-paid-5-billion-to-6-billion-to-settle-with-qualcomm-ubs.html)

The large tech companies largely build their patent portfolios for defensive purposes. Two companies with big portfolios are mutually assured destruction should they start attempting to enforce them against one another.

Sorry for my lack of knowledge, what do you mean by patents? Which things are the patents applied to? Model's weight? Model's source code? Model's theory (white papers)?

Researchers reuse others ideas and rethink of others work all the time. So if people want to against each other, just don't release white papers.

[deleted]

Unfortunately, it doesn't compile for me: https://github.com/BlinkDL/ChatRWKV/issues/38

I'm guessing even if you implement special support for lower compute versions that will probably cancel out the speed (and maybe size) benefits.

Well that depends on whether OpenAI can prove Google is deriving commerical value from OpenAI's patented research. On the other hand for OpenAI, I can see a clear case of using idea from other labs (Google -- Attention is all you need)

But just to clarify, I am not on one side or either. Definitely a bit sad for AI research going forward. But would be interested in knowing how the landscape changes.