publication croisée depuis : https://lemmy.pierre-couy.fr/post/2152233

I've been playing with deep reinforcement learning for a while. I originally started with a simple DQN, added all improvements from the Rainbow paper, and finally changed C51 for a quantile regression (and plan to swap it for an Implicit Quantile Network).

After implementing C51 (which was my first time with distributional RL) I started playing with policies that take advantage of the learned distributions : By independently taking N samples from each action-value distribution, scoring actions by averaging the samples, and picking the greedy action with respect to these scores, I was able to make the agent learn faster than similar agents using only NoisyNets or an epsilon-greedy policy (I'm still using NoisyNet, this is done on top of it). In the limiting cases, N=1 is just Thompson Sampling and N=+Infinity is just a plain greedy policy.

Finding an optimal value for N proved to be a challenge, so I decided to pick a random value for it at the start of each episode (N = 2**rng.uniform(8,12) for a QR-DQN with 32 quantiles/action works well in my experiments), which led to even better results.

I later found out about DLTV which made the agent discover new behaviors, but performed worse than previous experiments overall. Inspired by it, I tried something I did not find in previous works and got the best results out of all my previous experiments :

At each time step, compute an exploration_score as the ratio of "intra-action variance" over "inter-action variance" (rendered latex equation). I then take N/exploration_score samples from each distribution, and pick an action as described above. (more details at the end of this post)

For anyone reading this, I have a few questions :

1. Are you aware of any previous work I missed that tries similar exploration policies with distributional RL (interpolating between Thompson sampling and the greedy policy)
2. Most papers I found about learning from multiple exploration policies seem to be in the context of multi-actor parallelization. Is there any novelty in randomizing the policy parameters at the start of each episode, especially in the single-actor case ?
3. Is any part of what I'm doing worth the time it would take to quantitatively evaluate it ? I've been doing it mainly for learning and fun and have only qualitatively evaluated it so far. However, if there's a chance I can contribute to the field, I'll gladly make some time to compare it to published papers on ALE.

---

A few more details

I actually track a moving average and standard deviation of the exploration score, which lets me shift/rescale its values to a target average and standard deviation, and divide N by the shifted/rescaled value. I initially started with a target average of 1 and standard deviation of 1 as well (which gave good results), then tried randomizing these parameters at the start of each episode as well. This led to a lot more diversity in the policies and even better results.

Since this worked so well, I additionally randomized the noise strength in the NoisyNet layers.

Overall, this made the agent a lot more robust to deviating from what it considers to be the optimal trajectory, and allowed it to learn complex behaviors previous iterations were never able to learn (e.g. taking a few steps back to gain momentum, waiting for good cycles, or dodging hammer bros)

---

Watch it learn

For anyone interested, I made a live stream of the training in progress with graphs and some more details on the experiments I'm running. The current training run was started ~2.5 days ago. The agent has finished and unlocked levels up to 5-1, and is currently learning 5-2.

---

A lot more details

Long text hidden, click to expand

Available actions : The agent does not have access to the up and down buttons, the available actions only use left, right, A and B.

Adding the down button would double the total number of actions (because down can be pressed on top of all available actions).

Reward function : It mainly consists of reward(t) = max(0, x(t) - previous_best_x) + a larger reward for beating a stage. I had to tweak the scaling of both components.

I initially had penalties for time and death, but one made the agent suicidal in front of hard-to-overcome obstacles, while the other made it fear them too much and hug the left side of the screen. Removing both proved to increase the performance.

One trick that seems to help with most '*-3' levels (which have a lot of void to fall into) was to hold the reward while the vertical velocity of Mario is negative (meaning it is falling). Without this trick, the agent would sometimes get stuck learning to jump the farthest it can into the void.

Stage scheduling : Each episode is one attempt on one level. At the start of each episode, a stage is randomly picked with probability proportional to 1/(number of times the stage was beaten) among the unlocked stages. Each stage is unlocked after the previous one has been beaten 30 times, with only 1-1 unlocked at the start of the training.

Available stages : The first iterations of the agent were unable to learn maze castles (4-3, 7-3 and 8-4), so I removed them all. The reward function will give rewards for the first path the agent tries, then the agent will be teleported back by the game and no reward is received until it finds the right path and gets past the point where the game teleported it back. I plan to test newer (better) versions of the agent on these stages only and see if mazes can be re-added to the pool.

I've also removed underwater stages (2-2 and 7-2). The agent can learn them fine, but the game dynamics are really different from all other stages and they're really boring to watch. Since I already removed a bunch of stages, I figured I could remove these as well but I may re-add them with mazes because beating every level is cooler than beating a cherry-picked selection.

Since 8-4 is the only stage that requires going down a pipe, I considered it was not worth it to add the down action and will likely never re-add it to the pool, which would unfortunately be really anti-climactic...

Replay buffer warm-up : After initially using the standard approach of filling the buffer with transitions sampled from a random policy before training the neural net, I came-up with a "soft warm-up" scheme in which the first gradient updates happen after only 2000 transitions, but initially happen every few thousand transitions and gradually become more frequent until the replay buffer is full. Together with my custom exploration policy, this works very well : the agent very quickly starts behaving similar to a "right + random button" policy before learning to actually jump and run.

Custom n-step bootstrapping : When I initially implemented n-step bootstrap targets, I initially used n=3 from the Rainbow paper, noting the same instabilities as the paper did for higher n values. I then found the Retrace(\lambda) paper which seems to successfully address this by increasing n until the online network disagrees with the action choice from a stored transition. This makes n larger where the replay buffer data is on-policy, and smaller when it becomes off-policy. Since my GPU is already maxed and the training is already slow (20.8t/s when real-time is 20t/s) I could not afford the additional computations (building a training sample (s(t), a(t), sum(r(t+0..n)), s(t+n)) needs up to n_max transitions to go through the online network).

I'm trying to achieve similar sample efficiency gains by using cheaper alternatives as proxies for "how off-policy is a given transition" : I'm using the number of times a transition has been sampled, with n = int(max(n_min, n_max * k**times_sampled)) ; 0<k<1. The currently running experiment uses n_max=14, n_min=1 and k=1/1.3. I'm pretty sure it helps early in the training, and it does not collapse like a constant n=14 does

Stream setup : As I said, this is something I do for my own fun, and I really wanted to be able to see the agent learn in real time. The code runs a separate process, to which frames from training episodes are sent in a queue. The process then sends the frames as raw RGB24 to an local UDP socket, to which GStreamer connects and encodes the stream. With a simple MediaMTX configuration, I can manage the Gstreamer process and have the stream available through WebRTC on my LAN.

Then I figured someone else might have fun watching this, so I added a line to my MediaMTX config to send the stream to twitch and youtube. The overlay is a headless browser displaying custom HTML/JS (using d3.js for the graphs) piping raw frames to ffmpeg. GStreamer handles compositing the two streams together into the side-by-side view.

We are Student One Causal Networks. We have built an exhaustive statistical enumeration engine that runs full parameter-space searches across technical indicators, timeframes, and asset classes. The output is not a signal or a strategy recommendation — it is a structured dataset of statistically validated configurations. A neutral compute layer, not an opinion. The current build is functional for human users. What we are doing now is making it natively consumable by AI agents — proper tool definitions, OpenAPI spec, machine-readable outputs, the full stack so an agent can invoke our engine mid-conversation and return a real computed answer instead of a guess. We are in advanced discussions with one of the top three AI companies in the world to integrate this infrastructure into their agent ecosystem. We cannot name them publicly at this stage, but we will share the full picture privately with anyone serious enough to reach out. We are looking for engineers who have worked with agent-tool protocols — MCP, function calling, GPT Actions, tool-use APIs, or anything that lets a model call external compute during inference. This is a founding-stage equity role. No salary. The kind of opportunity that either makes sense to you immediately or does not. Interns with relevant skills are also welcome to apply. studentone.tech Drop a comment or message directly. Tell us what you have built.

The AI boom is more likely a marker of the end of the 50-year digital boom than the start of a new wave of innovation.

Highlights include: GIF decode and encode for imgcodecs, improved PNG and Animated PNG files handing, animated WebP Support, and especially the new HAL for RISC-V RVV 1.0 platforms.

With Massed Compute’s flexible-term GPU service, you can tap into NVIDIA H100, A100, and L40 instances for top-tier performance and predictable costs without rigid commitments. Email "QUOTE" to jack@massedcompute.com for a custom quote at a discounted rate.

hi

Open sourcing this project I made in just a weekend, planning to continue this in my free time, with synthetic data gen and some more modifications, anyone is welcome to chip in, I'm not an expert in ML. The inference is live here using tensorflow.js. The model is just 1.92 Megabytes!

cross-posted from: https://infosec.pub/post/23608906

I'm trying to train a machine learning model to detect if an image is blurred or not.

I have 11,798 unblurred images, and I have a script to blur them and then use that to train my model.

However when I run the exact same training 5 times the results are wildly inconsistent (as you can see below). It also only gets to 98.67% accuracy max.

I'm pretty new to machine learning, so maybe I'm doing something really wrong. But coming from a software engineering background and just starting to learn machine learning, I have tons of questions. It's a struggle to know why it's so inconsistent between runs. It's a struggle to know how good is good enough (ie. when should I deploy the model). It's a struggle to know how to continue to improve the accuracy and make the model better.

Any advice or insight would be greatly appreciated.

View all the code: https://gist.github.com/fishcharlie/68e808c45537d79b4f4d33c26e2391dd

cross-posted from: https://eventfrontier.com/post/177049

I keep getting an error ValueError: perm should have the same length as rank(x): 3 != 2 when trying to convert my model using coremltools.

From my understanding the most common case for this is when your input shape that you pass into coremltools doesn't match your model input shape. However, as far as I can tell in my code it does match. I also added an input layer, and that didn't help either.

I have put a lot of effort into reducing my code as much as possible while still giving a minimal complete verifiable example. However, I'm aware that the code is still a lot. Starting at line 60 of my code is where I create my model, and train it.

I'm running this on Ubuntu, with NVIDIA setup with Docker.

Any ideas what I'm doing wrong?

from typing import TypedDict, Optional, List
import tensorflow as tf
import json
from tensorflow.keras.optimizers import Adam
import numpy as np
from sklearn.utils import resample
import keras
import coremltools as ct

# Simple tokenizer function
word_index = {}
index = 1
def tokenize(text: str) -> list:
    global word_index
    global index
    words = text.lower().split()
    sequences = []
    for word in words:
        if word not in word_index:
            word_index[word] = index
            index += 1
        sequences.append(word_index[word])
    return sequences

def detokenize(sequence: list) -> str:
    global word_index
    # Filter sequence to remove all 0s
    sequence = [int(index) for index in sequence if index != 0.0]
    words = [word for word, index in word_index.items() if index in sequence]
    return ' '.join(words)

# Pad sequences to the same length
def pad_sequences(sequences: list, max_len: int) -> list:
    padded_sequences = []
    for seq in sequences:
        if len(seq) > max_len:
            padded_sequences.append(seq[:max_len])
        else:
            padded_sequences.append(seq + [0] * (max_len - len(seq)))
    return padded_sequences

class PreprocessDataResult(TypedDict):
    inputs: tf.Tensor
    labels: tf.Tensor
    max_len: int

def preprocess_data(texts: List[str], labels: List[int], max_len: Optional[int] = None) -> PreprocessDataResult:
    tokenized_texts = [tokenize(text) for text in texts]
    if max_len is None:
        max_len = max(len(seq) for seq in tokenized_texts)
    padded_texts = pad_sequences(tokenized_texts, max_len)

    return PreprocessDataResult({
        'inputs': tf.convert_to_tensor(np.array(padded_texts, dtype=np.float32)),
        'labels': tf.convert_to_tensor(np.array(labels, dtype=np.int32)),
        'max_len': max_len
    })

# Define your model architecture
def create_model(input_shape: int) -> keras.models.Sequential:
    model = keras.models.Sequential()

    model.add(keras.layers.Input(shape=(input_shape,), dtype='int32', name='embedding_input'))
    model.add(keras.layers.Embedding(input_dim=10000, output_dim=128)) # `input_dim` represents the size of the vocabulary (i.e. the number of unique words in the dataset).
    model.add(keras.layers.Bidirectional(keras.layers.LSTM(units=64, return_sequences=True)))
    model.add(keras.layers.Bidirectional(keras.layers.LSTM(units=32)))
    model.add(keras.layers.Dense(units=64, activation='relu'))
    model.add(keras.layers.Dropout(rate=0.5))
    model.add(keras.layers.Dense(units=1, activation='sigmoid')) # Output layer, binary classification (meaning it outputs a 0 or 1, false or true). The sigmoid function outputs a value between 0 and 1, which can be interpreted as a probability.

    model.compile(
        optimizer=Adam(),
        loss='binary_crossentropy',
        metrics=['accuracy']
    )

    return model

# Train the model
def train_model(
    model: tf.keras.models.Sequential,
    train_data: tf.Tensor,
    train_labels: tf.Tensor,
    epochs: int,
    batch_size: int
) -> tf.keras.callbacks.History:
    return model.fit(
        train_data,
        train_labels,
        epochs=epochs,
        batch_size=batch_size,
        callbacks=[
            keras.callbacks.EarlyStopping(monitor='val_accuracy', patience=5),
            keras.callbacks.TensorBoard(log_dir='./logs', histogram_freq=1),
            # When downgrading from TensorFlow 2.18.0 to 2.12.0 I had to change this from `./best_model.keras` to `./best_model.tf`
            keras.callbacks.ModelCheckpoint(filepath='./best_model.tf', monitor='val_accuracy', save_best_only=True)
        ]
    )

# Example usage
if __name__ == "__main__":
    # Check available devices
    print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))

    with tf.device('/GPU:0'):
        print("Loading data...")
        data = (["I love this!", "I hate this!"], [0, 1])
        rawTexts = data[0]
        rawLabels = data[1]

        # Preprocess data
        processedData = preprocess_data(rawTexts, rawLabels)
        inputs = processedData['inputs']
        labels = processedData['labels']
        max_len = processedData['max_len']

        print("Data loaded. Max length: ", max_len)

        # Save word_index to a file
        with open('./word_index.json', 'w') as file:
            json.dump(word_index, file)

        model = create_model(max_len)

        print('Training model...')
        train_model(model, inputs, labels, epochs=1, batch_size=32)
        print('Model trained.')

        # When downgrading from TensorFlow 2.18.0 to 2.12.0 I had to change this from `./best_model.keras` to `./best_model.tf`
        model.load_weights('./best_model.tf')
        print('Best model weights loaded.')

        # Save model
        # I think that .h5 extension allows for converting to CoreML, whereas .keras file extension does not
        model.save('./toxic_comment_analysis_model.h5')
        print('Model saved.')

        my_saved_model = tf.keras.models.load_model('./toxic_comment_analysis_model.h5')
        print('Model loaded.')

        print("Making prediction...")
        test_string = "Thank you. I really appreciate it."
        tokenized_string = tokenize(test_string)
        padded_texts = pad_sequences([tokenized_string], max_len)
        tensor = tf.convert_to_tensor(np.array(padded_texts, dtype=np.float32))
        predictions = my_saved_model.predict(tensor)
        print(predictions)
        print("Prediction made.")


        # Convert the Keras model to Core ML
        coreml_model = ct.convert(
            my_saved_model,
            inputs=[ct.TensorType(shape=(max_len,), name="embedding_input", dtype=np.int32)],
            source="tensorflow"
        )

        # Save the Core ML model
        coreml_model.save('toxic_comment_analysis_model.mlmodel')
        print("Model successfully converted to Core ML format.")

Code including Dockerfile & start script as GitHub Gist: https://gist.github.com/fishcharlie/af74d767a3ba1ffbf18cbc6d6a131089

I created a Lemmy community specifically for TensorFlow! Check it out and subscribe if you're interested.

Declaration

We, the undersigned members of the Open Source community, assert that Open Source is defined solely by the Open Source Definition (OSD) version 1.9.

Any amendments or new definitions shall only be recognized if declared by clear community consensus through a transparent process to be determined.

Unsplash has a ton of free content and a great api - so I used the hugchat api to generate search queries based on some user input text and fetched images from there.

Buggy test site (using all free api's so will break frequently, free Unsplash* api is 50 pics/h) here: http://aisitegeneration.devsoft.co.za/ *sorry Unsplash I haven't added the attribution for photo's yet, I will soon ok?

Thoughts on this approach vs generative AI?

I am currently trying to get a bit more into ML, for me that means playing with it in some context I know already or applying it to something interesting - either way I am aware this whole endeavor is a bit of stretch and having a good grasp on machine learning requires a good mathematical knowledge.

In my off time I have been recreating a digital copy of a table card game called Scout: For The Show and I had the great idea to try and make an autonomous agent based on Machine Learning for it (definitely the best starting idea /s but there is still a lot to learn in failures).

First, I did the naive thing, imagined the inputs and outputs from a players perspective - current hand, amount of turns taken, count of cards in other player hands, ... but my intuition tells me this is in some way very wrong (?), the "shapes" of these inputs/outputs are weird - I don't think the model would respond with a valid move anytime soon during training like this, if ever.

Second, I've then searched far and wide for card games and machine learning and found some resources where they usually reduce the problem space as much as possible and apply the model only on a subset of the information (often represented in completely different formats/dimensions - Markov Decision Process).

Obviously I am not asking for the mathematical analysis of the game in question, in broad sense I am looking for any kind of pointers that might apply here, I am aware this is a very brute-force approach for something that should be carefully mathematically analyzed and from that a model could be derived.

Thanks for any pointers, wisdoms or ideas!

Notes:
I am coming from a software development background - Python mainly, so it's not that far for me programming wise, and I have already played with YOLO models though only as user.

The Scout card game has 45 cards with a number (1-10) on the top and bottom, the main objective is to capture points by playing stronger card combinations, either pairs/triples/x of a single number (1-1-1, 9-9, ...) or sequences/straights (2-3, 5-6-7-8, ...).
The twist is that cards in hand can't be moved or flipped around, only the top side number is important for most of the game (and each variation of the top/bottom numbers is contained only once, 1/10 and 10/1 is the same card, only flipped).
Players take turns in either playing a new hand on the table (Show - capturing the remaining hand, scoring) or taking a one card from the table (Scout) and putting it anywhere in their hand, even flipping top/bottom)

Resources I have found:
https://www.youtube.com/watch?v=IQLkPgkLMNg (Great explanation of the problems with solved/unsolved games, minimax, MCTS etc)
https://www.youtube.com/watch?v=vXtfdGphr3c (Reinforced Learning)

cross-posted from: https://slrpnk.net/post/5501378

For folks who aren't sure how to interpret this, what we're looking at here is early work establishing an upper bound on the complexity of a problem that a model can handle based on its size. Research like this is absolutely essential for determining whether these absurdly large models are actually going to achieve the results people have already ascribed to them on any sort of consistent basis. Previous work on monosemanticity and superposition are relevant here, particularly with regards to unpacking where and when these errors will occur.

I've been thinking about this a lot with regards to how poorly defined the output space they're trying to achieve is. Currently we're trying to encode one or more human languages, logical/spatial reasoning (particularly for multimodal models), a variety of writing styles, and some set of arbitrary facts (to say nothing of the nuance associated with these facts). Just by making an informal order of magnitude argument I think we can quickly determine that a lot of the supposed capabilities of these massive models have strict theoretical limitations on their correctness.

This should, however, give one hope for more specialized models. Nearly every one of the above mentioned "skills" is small enough to fit into our largest models with absolute correctness. Where things get tough is when you fail to clearly define your output space and focus training so as to maximize the encoding efficiency for a given number of parameters.

I am looking to build a PC where I can run some LLMs and also pytorch and maybe some video encoding and I was wondering what is the best price wise GPU with at least 16GB of VRAM that I can buy right now.

For the record I really hate NVIDIA and I would prefer not to give them my money, but if it is the only viable option I would probably wait till they release the 4070 Ti Super, but what about the AMD (7800 XT) GPUs or the Arc A770? Are they going to work? In the past I had an AMD GPU and making ROCm work with it was a pain in the ***.