How is one achieving long context lengths for LLM? Chatgpt has a length 32k? Is the transformer decoder "just" that big?

Pretty sure you need to verify your account with personal info (e.g. national ID card, and ID-binded cellphone number) to use their services, which in this case they can track you down if you dare to speak against the supreme leader.

[removed]

Have you never been peer reviewed before?

maybe but what about this? https://www.cnbc.com/2019/04/18/apple-paid-5-billion-to-6-billion-to-settle-with-qualcomm-ubs.html

OpenAI is not a small company either. It may be a "startup" but it's clearly backed by Microsoft and between those two there's probably quite a lot of patents that Google have used in the past too.

I don't think patent battles will go anywhere. DeepMind could simply stop releasing papers (or curtail it significantly) like they've already hinted they might do.

What do you say so? For Google its probably peanuts in terms of cost. And there is a clear case for them to make that transfomers originated with them.

It's probably not in their interest as they know they both will end up worse if they decide to follow that path.

I found this article to be extremely insightful. The best I have seen that explains in layman terms why large language models work, and what yheir limitations are.

https://fullstackdeeplearning.com/cloud-gpus/

Your best bet to reach 256Gb in the cloud would be Azure with 4x80GB A100 instances, however your 40k budget will only buy you 3000 hours of compute at best on demand, with spot instances stretching that a bit further.

If that's not enough for you then you will have to figure out how to make a server with RTX A6000 Adas with 48GB each. RTX4090 would be cheaper but there may be legal issues due to the gaming driver license, you would need to use multiple servers due to power usage or strongly limit the power limit, and Nvidia dropped P2P that may o may not matter depending on how much communication you need between the GPUs (https://discuss.pytorch.org/t/ddp-training-on-rtx-4090-ada-cu118/168366)

Not saying Google is better or OpenAI is better. But could they now be engaging in patent battles as it seems like now there is significant comercial interest at stake? And also OpenAI not releasing any details means for AI research going forward.

Yes - the Transformer is proof by demonstration that you don't need a language-specific architecture to learn language, and also that you can learn language via prediction feedback, which it highly likely how our brain does it too.

Chomsky is still he sticking to his innateness opinion though (with Gary Marcus cheering him on). Perhaps Chomsky will now claim that Broca's area is a Transformer?

Google has probably used stuff from OpenAI too (decoder only GPT-style training or CLIP or Diffusion or Dall-E ideas maybe?). Anyways, it's clear they (and probably every large tech company with big AI teams) are in an arms race at this point. Its definitely not a coincidence Google, OpenAI / Microsoft released on the same day, and we also heard Baidu is releasing sometime these days. Meta and others will be probably following suite. The publicity (and the market share for this new technologies) is worth too much.

This looks like a glorified blogpost. It even has sensationalistic bits that add nothing to the subject

>there is no possibility that Trump understands prime numbers

How can i train a model so that it detects severity of damage in a image. Which algo will suit for my need?

CCP trolling and disinformation capabilities just got a massive upgrade.

Doesn't matter to most reviewers. Little care for accessibility as well, remember the flak you'd get for not using MuJoCo in a RL paper when it wasn't opensource.

Ah. I see then you dont actually have many choices. LoL

Training for what? Classification, embedding, generation?

A lot of Chomsky ideas were refuted and his defence of Russia is simply awful. The man is more often wrong than right.

just got time to try it, but it doesn't load nor does it give error message :( Thanks anyways for your help!

There's a rent vs buy section in that article. It basically comes down to how much will you use the box, vs. how often / fast do you need to test things out. They go through energy costs and all that in the article. Just plug in your figures and see what the output is.

Yet when my sister submitted a paper before ChatGPT was released, she got complaints that she did not evaluate on ChatGPT literally days after it was released

Their pricing is definitely much better than AWS and google cloud, unfortunately they don't seem to be HIPAA compliant or this feature is not advertised.