I'm looking to be able to train a model that is suited to taking an image and reconstructing it with additional information, for example, taking R&G channels for an image and recreating it with the addition of the B channel. On first glance it seems like an in-painting model would be best suited to this, and treat the missing information as the mask, however I don't know if this assumption is correct as I've not got too much experience with those kinds of models. Additionally, I'm looking to progress from a really simple baseline to something more complex, so I was wondering if an architecture of a simple CNN or an autoencoder trained to output the target image given image missing information, but I may be way off here. Any help greatly appreciated!

How many companies could realistically replicate their results though? We already have a pretty good idea of what's going on under the hood, but would knowing the intricacies of GPT-4 help anyone smaller than Google?

> The recent release of GPT4 has apparently sent most of that sector into a mass existential crisis

I don't know where you got this from

I can tell you for sure that no one worth their salt would make claims like those for something that has been out for a day, and from what I've seen, still has the same problems you can get sued over. Might be torchkiddies larping NLP peeps and starting mass hysteria. The Andrew Ngang.

This is hearsay, but my understanding was that Hassabis' goal was for Deepmind to be winning one Nobel Prize per year or something like this, so I don't think he's personally up for the closed research model.

Anthropic is another good org to support for AI safety research: https://www.anthropic.com/research

I dislike Computer Vision. It just annoys me somehow.

The whole thing seems very bitter lesson-y and I suspect in the future we'll have a very general architecture that learns to reconfigure itself for the data.

Right, Google's use of this whole field has been limited to optimizing existing products. As far as I know, after all their billions in investment, it hasn't driven the launch of a single new product. And the viscerally exciting stuff -- what we're calling "generative AI" these days -- never saw the light of day from inside Google in any form except arguably Gmail suggested replies and occasional sentence completion suggestions.

> it's a different mode of launching with higher risks, many of which have different risk profiles for Google-scale big tech than it does for OpenAI

This is textbook innovator's dilemma. I largely agree with the summary but think basically the whole job of Google's leadership boils down to two things: (1) keep the good times rolling, but (2) stay nimble and avoid getting disrupted by the next thing. And on the second point, they failed... or at least they're a lot closer to failure than they should be.

> Example: ChatGPT would tell you how to cook meth when it first came out, and people loved it. Google got a tiny fact about JWST semi-wrong in one tiny sub-bullet of a Bard example, got widely panned and lost $100B+ in market value.

Common narrative but I think the real reason Google's market cap tanked at the Bard announcement is due to two other things: (1) they showed their hand, and it turns out they don't have a miraculous ChatGPT-killer up their sleeves after all, and (2) the cost structure of LLM-driven search results are much worse than classical search tech, so Google is going to be less profitable in that world.

Tech journalists love to freak out about everything, including LLM hallucinations, bias, toxic output, etc., because tech journalists get paid based on engagement -- but I absolutely don't believe that stuff actually matters, and OpenAI's success is proving it. Google's mistake was putting too much stock in the noise that tech journalists create.

There are workarounds though.

Dropconnect isn't patent encumbered (degrades feature detectors/neurons by dropping connections instead of disabling them) and is, I think better than dropout.

Similarly, with transformers, Google has a patent on encoder-decoder architectures, so everyone uses decoder-only architectures, etc.

Some companies are probably going to patent critical AI/ML things, but that hasn't really happened yet and I don't believe that any patent encumbered method is currently either critical or even optimal.

Staying up late to run another iteration sounds very familiar haha. Are you happy with your change or did it turn out that the grass is greener on the other side?

If you can’t replicate their results, then they’re not useful for research

Lesson is that to be successful you need to actually ship something. You can't stay in research land forever. Who would have thought?

It has nothing to do with semantics, it's basic corporate strategy.

When dealing with an imbalanced dataset, I have been taught to oversample on only the train samples and not the entire dataset to avoid overfitting, however this was for structured text based data in pandas using simple models from sklearn. However is this still the case for image based datasets that will be trained on a CNN? I have been trying to oversample only the train data by applying augmentations to the images. However, for some reason I get a train accuracy of 1.0 and a validation accuracy of 0.25 which does not make sense to me on the very first epoch, where the numbers dont really change as the epochs progress which doesn't make sense to me. Should the image augmentations via oversamping be applied to the entire dataset? (fyi I am using PyTorch)

With the ever increasing cost of training LLMs, I feel like we're entering a new phase in AI. Away from open science, back to aggressively protecting IP and business interests.

Microsoft, via OpenAI are taking big steps into that direction. We'll see if others follow suit. I hope not, but I think they will.

That's a bogus comparison, for a number of reasons such as:

1. These models are learning vastly more than language alone

2. These models are learning in an extraordinarily difficult way with *only* "predict next word" feedback and nothing else

3. Humans learn in a much more efficient, targetted, way via curiosity-driven knowledge gap filling

4. Humans learn via all sorts of modalities in addition to language. Having already learnt a concept then we only need to be given a name for it once for it to stick

This a very silly semantic game that i have no interest in engaging with

"Python 1.8 (deprecating Python 1.7)" links to "Deprecation of Cuda 11.6 and Python 1.7 support for PyTorch 2.0"

You can pivot to an adjacent role within ML like MLOps or Data Engineering. That’s what I did. I didn’t like waiting an entire day for a model to run and then having to fix it late at night so I can do another iteration throughout the night. So now I build machine learning platforms which is more traditional software engineering and comes with predictable outcomes. Your knowledge about ML is still valuable.

As a person that heavily relies on both CLIP (released 2 years ago) and Whisper (released just 6 months ago) in his research, I would disagree with the claim that "open research in AI [is] gone".

In addition, I've needed to run the usual benchmarking to compare my own work with several other models and was quite surprised when I was able to run my full benchmark on GPT-3 solely using the free credit provided by them.

Don't get me wrong, I criticize OpenAI for not completely sticking to the mission they built their foundation in (I mean it's literally in the name FFS) but I wouldn't say they completely closed off research from the public.

Precious Advices About AI-supported Audio Classification Model

Hello everyone,I'm Omer.
I am new in this group and writing from Turkey. I need very valuable advice from you precious researchers.
I am a PhD program student in the department of music technology. I have been working in the field of sound design and audio post-production for about 8 years. For the last 6 months, I have been doing research on AI-supported audio classification.My goal is to design an audio classifier to be used in the classification of audio libraries. Let me explain with an example as follows; I have a sound bank with 30 different classes and 1000 sounds in each class (such as bird, wind, door closing, footsteps etc.).
I want to train an artificial neural network with this sound bank. This network will produce labels as output. I also have various complex signals (imagine a single sound track with different sound sources like bird, wind, fire, etc.). When I give a complex signal to this network for testing, it will give me the relevant labels.I have been doing research on this system for 6 months and if I succeed, I want to write my PhD thesis on this subject. I need some advice from you, my dear friends, about this network. For example, which features should I look at for classification? Or what kind of artificial intelligence algorithm should I use?
Any advice you say you should definitely read this article or that article on this subject.I apologize if I've given you a headache. I really need your advice. Please guide me. Thank you very much in advance.

That was my first thought too. I'm about to finish my masters program and I spent the first half thinking that it was just hyperparameter tuning, until I sat down and learned the math and theory. Now it's so much more interesting and explainable. That random tuning is now much more calibrated from experience and understanding the theory. (As of now), I could easily make a career out of this, because it's not random and simple optimization. Plus, the field is so hot right now, that it's unreasonable to assume that what data scientists do now is what they will do in 5, 10, or 20 years

Many of the interesting developments in deep learning have in fact made their way to Google + FB products, but that those have not been "model-first" products. For example: ranking, personalization, optimization of all kinds, tech infra, energy optimization, and many more are driving almost every Google product and many FB ones as well.

However, this new trend of what I would call "Research Products" which are light layers over a model -- it's a different mode of launching with higher risks, many of which have different risk profiles for Google-scale big tech than it does for OpenAI. Example: ChatGPT would tell you how to cook meth when it first came out, and people loved it. Google got a tiny fact about JWST semi-wrong in one tiny sub-bullet of a Bard example, got widely panned and lost $100B+ in market value.

I was wondering if this has something to do with lack of experience. And from what I've heard indeed the more you experiment with the models, the better you understand what helps when, to some extent.

The thing is, a neural network still remains a black box, as the number of parameters is too big to fully understand what is happening. It is an empirical study mostly. So you choose your architecture, test, change hyperparameters, test, change the architecture, test, change some other parameters, test, and so on. You can't be sure your model will work properly right away and it may take lots of iterations. With larger models which take long to train it might be overwhelming.

Of course, it might be different in your case, you can start with some toy examples, and if you still like it, I'd recommend playing with larger networks.

There is Chrome extension, to rename all "OpenAI" mentions to "ClosedAI". It's open source