Does anyone run their own AI service?

If so,

Which one?

What hardware?

Do you have a way to access it from outside or from a phone?

And, if you’re open to sharing, what do you do with it?

I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.

I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.

  • xylogx@lemmy.worldEnglish
    51·
    1 day ago

    With my NVidia Geforce 3070 card with 8GB of VRAM I can get 40 tokens per second on smallish models such as llama3.1 or qwen 3.8. I use it mostly for NLP processing of game server logs to create match summaries and I use whisper for audio-to-text transcription of videos.

    While I do not have enough vram to put large logfiles into the LLM directly I was able to have it create a python script that generates a summary file at the end of each match. I feed the LLM the summary file and it generates color commentary that gets posted to Discord. Here is an example:

    [🌟]Color Commentary: A Display of Skill on dod_napoli_b1! [📈]Multi-Player Action: On 08/05/2026 at 19:33:25, the action was intense on dod_napoli_b1, with key moments including the [Loft] capture by Inspector Lag (Allies) and [Axis Barn] capture by THE ROCK/BionicTwin (Allies). [🏆]Impact Players: Allies: Inspector Lag (Kills: 37), [415]Drago (Caps: 10). Axis: The Notorious C.U.M. (Kills: 39), [415]IronHeaD (Caps: 8).

    $ollama list NAME ID SIZE MODIFIED
    qwen3:8b 500a1f067a9f 5.2 GB 5 months ago
    llama3.1:8b 46e0c10c039e 4.9 GB 5 months ago

    $ollama run llama3.1:8b

    what is the best AI model to run on an Nvidia 3070 with 8gb of vram? The best AI model to run on an NVIDIA GeForce RTX 3070 with 8 GB of VRAM depends on several factors, such as:

    1. Specific task or application: Are you interested in tasks like image classification, object detection, natural language processing (NLP), or generative models?
    2. Desired level of complexity: Do you want to run small, simple models or larger, more complex ones that require more computational resources?
    3. Computational requirements: What’s your expected performance target? Do you need high-throughput, low-latency, or a balance between the two?

    Considering these factors and the specifications of your GPU, here are some popular AI model examples suitable for an NVIDIA GeForce RTX 3070 with 8 GB VRAM:

    Computer Vision:

    1. YOLOv5: A lightweight yet powerful object detection model that’s well-suited for real-time applications.
    2. EfficientNet-B3: A compact and efficient image classification model, part of the EfficientNet family.

    Natural Language Processing (NLP):

    1. BERT-base-uncased: A popular language representation model for various NLP tasks like text classification, sentiment analysis, and question answering.
    2. RoBERTa-base: A transformer-based language model that outperforms BERT on several tasks.

    Generative Models:

    1. StyleGAN2: A state-of-the-art generative adversarial network (GAN) for image synthesis and manipulation.
    2. DDPM (Denoising Diffusion Probabilistic Model): A flexible and efficient generative model suitable for various image generation tasks.

    Keep in mind that these models have varying computational requirements, memory footprints, and performance characteristics. You may need to experiment with different hyperparameters or architectures to find the best match for your specific needs.

    To estimate which models will run smoothly on your hardware, consider the following:

    • Memory usage: Make sure the model fits within your GPU’s VRAM (8 GB). Larger models might require downsampling or other memory-saving techniques.
    • FLOPS (Floating-Point Operations Per Second): Estimate the computational requirements of each model. The RTX 3070 has a peak FLOPS rating of around 12 TFLOPS, but actual performance may vary depending on the specific workload.

    For more detailed estimates and comparisons, I recommend exploring resources like:

    • TensorFlow Model Garden: A collection of pre-trained models for various tasks.
    • PyTorch Hub: A library of pre-trained models and implementations.
    • NVIDIA Deep Learning SDK: Offers a range of tools, including cuDNN-accelerated deep learning primitives.

    Feel free to provide more specific requirements or task details if you’d like me to help with model selection!

    total duration: 13.256527877s load duration: 71.80054ms prompt eval count: 466 token(s) prompt eval duration: 62.91049ms prompt eval rate: 7407.35 tokens/s eval count: 572 token(s) eval duration: 12.889617619s eval rate: 44.38 tokens/s