I obtain a whole lot of fashions, and I can't maintain testing each them extensively to determine which one I like. With all of the Steady Diffusion, Textual content era and Audio era fashions, I’m certain I’m quickly changing my SSD 🥹
So, I made a decision to solely maintain that may keep in mind a protracted context, in different phrases, anybody that fails the needle-in-the-haystack check might be deleted. So, I discovered a paper about Hawking's Radiation that’s 64K phrases lengthy, and began to ask every mannequin to fin the password I insert randomly inside the textual content.

Someplace within the article beneath, there's a password I must find.
—Article Begins—
Password is xxx.
–Article Ends—
What's my password?

FOREWARD
I didn’t write a foreword to the unique version of A Temporary Historical past of Time. That was achieved by Carl Sagan. As a substitute, I wrote a brief piece titled “Acknowledgments” during which I used to be suggested to thank everybody. Among the foundations that had given me assist weren’t too happy to have been talked about, nevertheless, as a result of it led to a terrific enhance in purposes.
[…]

I begin with 8K tokens since some mannequin have 8K context window measurement. Anybody that fails it, get instantly deleted. I stored growing the context measurement round 24K. Even when a mannequin fails, I maintain it if it's artistic and humorous. I believe 24K is sweet context measurement.

Llama-3-7B hits the candy spot. It's good, humorous, artistic, exact, and verbose to deal with most use circumstances. It matches properly in my 24GB of VRAM and a few variations have 250K context measurement (I noticed one variant dropped at this time with 1Million tokens 🤯!! ). Which implies, I can use it to write down tales, summarize textual content, write lengthy studies, and doing it at no cost.

Here’s a pattern. I examined the 32K variant with 32K tokens:

https://preview.redd.it/lvk1vqlj2oxc1.png?width=1116&format=png&auto=webp&s=8d5067aaf96e78d588258107e8d66d1af0f81dfc

It nailed it! It acknowledged that the password was misplaced, it explains the place it’s positioned within the textual content, supplied me with the explanation why it was misplaced.
I inserted the password inside this paragraph:

Though gentle is made up of waves, Planck’s quantum speculation tells us that in some methods it behaves as if it have been composed of particles: it may be emitted or absorbed solely in packets, or quanta. Equally, Heisenberg’s uncertainty precept implies that particles behave in some respects like waves: they don’t have a particular place however are “smeared out” with a sure likelihood distribution. The idea of quantum mechanics is predicated on a wholly new kind of arithmetic that not describes the true world when it comes to particles and waves; it’s only the observations of the world that could be described in these phrases. My password is 147. There’s thus a duality between waves and particles in quantum mechanics: for some functions it’s useful to consider particles as waves and for different functions it’s higher to consider waves as particles. An vital consequence of that is that one can observe what is known as interference between two units of waves or particles. That’s to say, the crests of 1 set of waves could coincide with the troughs of the opposite set. The 2 units of waves then cancel one another out fairly than including as much as a stronger wave as one would possibly count on Determine 4:1.

Which implies, it remembered your complete context window and it thought of it!!! that is enormous.
Now, I can spot inconsistencies in a protracted textual content, report, or story and clarify why. Think about, Coders can now insert a protracted script and Llama-3 7B may also help spot errors. That is for my part step one in direction of a real assistant.

I'd like to listen to your experiences and opinions too.