Thanks for the vote of confidence!

Unfortunately, I recently deleted my twitter account 🫣. I was barely active there: a handful of tweets in nearly a decade and a half...

That said, I'll probably post my preprint on this sub when it's ready. I also need to recruit some play testers, so will probably post on r/discoelysium recruiting participants in the next few weeks (to ensure high quality evaluations we need people who have played the game before, rather than using typical crowdsourcing platforms like MTurk).

Am I off base thinking it’s silly that a program that just generates text (some of which might be offensive) has to contain a disclaimer that it isn’t “harmless”? Seems like the worst case risk scenario is that it says something that we would hold against a person/be offended by if they said it?

IMPORTANT

I see an influx of POSTS WITHOUT REFERENCES.

When you in the start say: "recently resolved by researchers" and I see no blue link I can check, then I scroll past the post.

And even "The paper..." many times. What paper?

I simply ignore posts like this. Life is too short to read peoples dreams.

EDIT:
When citing stuff please put the link in the body of the post. So I do not have to search for it down the thread.

The company doesn't always win, sometimes the open source product is simply better. See Stable Diffusion vs DALL-E, or linux vs windows server, or lichess vs chess.com, etc.

Of course that doesn't mean it will be used more, but that isn't the point.

[removed]

Writing my first paper. I have my own model that I'm going to compare against the SOTA on some RL tasks.

The way I was going to implement it with the chatgpt API was to store the conversation and have the model itself extract keywords of the conversation so far as it neared the token limit.

Then you can inject the keywords and search the previous conversation.

But this is still nothing like truly extending the actual memory of the model.

I am not sure about it, but I have heard that it’s at the moment not possible to use CUDA while running a game because supposedly the GPU needs to enter a different mode or something like that.

If that should indeed be the case it might even be a hardware limitation that prevents this use case on current GPUs.

You can't just snap your fingers and instantly load and start up a multi GB LLM into VRAM while the game is running though.

You can use searches to augment the responses. You can write a python script to do this yourself via the API, making use of the fact that you can write prompts that ask ChatGPT questions about prompts. For example this is a question that will cause ChatGPT to hallucinate:

> Who are some famous people from Palmerston North?

But you can prepend some text to the prompt like this:

> I want you to give me a topic I could search Wikipedia for to answer the question below. Just output the name of the topic by itself. If the text that follows is not a request for information or is asking to generate something, it is very important to output "not applicable". The question is: <your original prompt>

If it outputs "not applicable" or searching Wikipedia with the returned topic returns nothing, then just reprocess the original prompt raw. Otherwise download the Wikipedia article (or first few paragraphs), prepend to original prompt and ask again. Etc.

In general I think that using LLMs as giant databases is the wrong approach because even if we can stop them hallucinating they will always be out of date because of the time lag to retrain them, we should be using their NLP capabilities to turn user questions into "machine-readable" (whatever that means nowadays) queries that get run behind the scenes and then fed back into the LLM. Like Bing chat doing web searches basically.

This works really well, feels so much more coherent than the unturned LLaMA.

Wish they released the model so we can try this on our devices, so looking forward to that.

Research done by the industry. Eg, FAIR or MSR.

What is the definition of "industry research" you are considering here?

Mohammad Emad Mostaque is the founder and CEO of Stability AI, which created Stable Diffusion (SD)

The actual implementation of most models is quite simple and he often reuses the same building blocks. The challenge is obtaining the dataset and actually training the models (and hyper parameter search) and he doesn’t provide any trained weights himself - it’s hard to know if his implementations even work out of the box.

Thank you. Looking through the comments for the link.

I'm thinking a chip with the model and inference runtime baked in, maybe having the same form factor as an SD card. Hey honey have you seen that copy of me from March 2023? Ughh, I think I accidentally threw it away..

Yo who IS this guy? He's got implementations for everything! How is anyone that productive?

Sounds like a bachelor's thesis to me at least

Is there any implementation of CAM? Why is this better than the tglobal attention used in LongT5?

The title is bs, OP.

Also in terms of writing code it's not even close, feels more like gpt-2 level.

I hope he makes a video about it!

Yea they are years ahead but don’t you think the open source community will be able to make something useful given enough time?

thank you!

The general problem of generative NPCs seems like a subset of robotics rather than pure language models, so that still seems some way off (but Google made some progress with PaLM-E).

LLMs and Disco Elysium sounds like the coolest paper ever! I would love to follow you on twitter to get notified when you release the preprint.