ScOp Venture Capital

We are hiring an analyst for ScOp Venture Capital. This is a great opportunity to work closely with entrepreneurs and investors at the intersection of investing and operating.

In our third episode on AI, we discussed bottlenecks that may hold AI back. Here's why: 🚄 Increasing Demands for ML: While early models were trained on gaming PCs, recent models require extensive GPU power in sophisticated data centers. The cost of next-generation models like GPT-5 is expected to increase significantly. 📈 Data Challenges: Larger models require more data, and we're approaching a saturation point with the current volume of available data, particularly for text information. 🚀 Opportunities for Growth: Despite challenges, there's still room for growth, especially in training models from videos and other media formats. Technical advancements are necessary to fully leverage these opportunities. 🐿 Specialized Chips and Algorithmic Improvements: Semiconductor companies are developing specialized chips for running specific types of neural networks. Additionally, algorithmic improvements are underway to reduce computational complexity. 🕹 Alternative Learning Methods: There have been significant advancements in reinforcement learning, with applications like teaching a robot to play soccer. There's potential for further development, such as using debate between agents to learn from their own arguments. Additionally, videos contain a lot of under-exploited implicit knowledge and world models. For example, a model should be able to infer the laws of gravity by watching enough videos with falling objects. Ep.3: How bottlenecks have slowed down AI's progression https://lnkd.in/gBDzkbTR

Too many CEOs underestimate the importance and difficulties of marketing. Here's a quick read for early-stage company CEOs. https://lnkd.in/gwAnQnbr

Second video in the AI series. PS: doing some A/B testing with a YouTube link instead of direct video upload.

Check out Ivan's thoughts on AI.

I'm extremely excited to share that Rogo has raised $7 million in funding from AlleyCorp with participation from BoxGroup, Company Ventures and ScOp Venture Capital to bring generative AI to the financial services industry. We’re hiring fast and working on extremely exciting technology at the intersection of generative AI and finance. If you're super talented and like to work hard please reach out! Also, check out our new website :) https://lnkd.in/e5M4Jeid

Word2Vec was a pivotal paper published a decade ago by researchers at Google. They showed that by attempting to predict a word from their neighbors (or the neighbors from the word), the resulting model acquired compelling semantic capabilities. The main element of this model is a n x m matrix, where n is the vocabulary size and m is the dimension of the vector to encode each word. A typical value from m is 300. Rather than having an identifier for each of the roughly 200k words in the English language (in addition to proper names and numbers, which are also words), each word can be represented with a coordinate in 300-dimensions. This is known as an embedding, and is a primordial type of language model (and a component of most language models since). The implication is that if two words are close to each other in this 300-dimensional space, they are similar. Hence, "dog" and "cat" would presumably be closer to each other than "airplane" is to either of them. This is the so called cosine similarity, enabled by vector databases like Pinecone. Furthermore, each of the 300 axes can be thought as having a meaning (although it might not be monosemantic). So for example "dog" would score high on the axis intended to represent "animalness", whereas "plane" would score low. It's important to understand that this sort of distance calculation wouldn't be possible if each word was identified with a sequential ID, since that's equivalent to having a single dimension. Having higher cardinality allows many words to cluster together around central concepts, like "animals", "computers", "politics", and so on. The shocking result is that this structure is conducive of performing a sort of arithmetic with words. The canonical example being "king" - "man" + "woman" = "queen". Performing this arithmetic operation on the vectors for the terms in question will yield a new vector within a short proximity of "queen". Proximity, as opposed to identical, is an important characteristic. The relationship between "Paris" => "France" and "Rome" => "Italy", will be similar but not identical, given that it was learned from statistical properties of a text corpus. So the inexactness is a feature. To learn, I implemented word2vec from scratch (following Olga Chernytska ). The code is here: https://lnkd.in/gV5J4Ugx . Run word2vec_py and play with the results using Inference_ipynb Take a look at this interactive visualization. Try to interpret what is different about the two green clusters: https://lnkd.in/guRHnwNY Note: "king" - "man" + "woman" doesn't always work as expected. When embeddings are trained, dimensions acquire different meaning, due to randomness. In order to work, the embeddings have to learn the right properties (e.g. male vs female). I suspect a larger training set would improve results. After several runs, I ended up with these n-best results: king: 0.782 queen: 0.535 woman: 0.516 monarch: 0.515