STACKIT AI Model Serving Tested: Fully Sovereign AI Solutions for Enterprises

Digital Sovereignty in Challenging Times

We live in an era characterized by rapid technological advances and global uncertainties. In such challenging times, digital sovereignty is becoming increasingly important, especially for companies in Europe. While global hyperscalers like AWS, Google, and Azure undeniably offer powerful services, many companies face the challenge of operating their data and applications in an environment that is both secure and compliant with European data protection standards. Data protection can currently be maintained with Google Cloud, Microsoft Azure, and also with AWS, but there are still industries like hospitals or social insurance that cannot simply use a hyperscaler. This is where STACKIT comes in – a sovereign cloud solution from Germany that represents an attractive alternative to global providers and helps maintain digital independence.

What is STACKIT? A Sovereign Cloud Alternative from Germany

STACKIT stands for an independent and data-sovereign cloud infrastructure specifically tailored to the needs of European companies. Unlike global providers, STACKIT offers a 100% European cloud that meets the strict requirements of the General Data Protection Regulation (GDPR) and other standards like C5. This means companies retain complete control over their data and can ensure it is not stored or processed outside the EU. By using open-source technologies, STACKIT avoids vendor lock-in, ensuring flexibility and autonomy. STACKIT’s infrastructure is regularly audited by independent institutions to meet the highest security standards. As part of the Schwarz Group, STACKIT offers economic solidity and reliability, even in times of crisis.

STACKIT Model Serving: An Innovative Solution for Artificial Intelligence

STACKIT Model Serving is a new service that enables companies to efficiently and securely integrate Artificial Intelligence (AI) into their applications. This fully managed hosting environment is specifically tailored to Large Language Models (LLMs) and offers advantages for companies that want to use AI technologies without having to worry about the technical details of model hosting. We already know similar products from Azure with Azure OpenAI, Google with VertexAI LLM Serving, and also from AWS with Bedrock.

Unlike the three mentioned above, STACKIT Model Serving enables the deployment, scaling, and maintenance of AI models in a secure and data-sovereign environment. Companies can thus focus on developing their AI-driven applications while STACKIT manages the infrastructure. This facilitates access to powerful models and accelerates the implementation of AI projects.

Currently Available Models

In the beta phase, STACKIT Model Serving offers a selection of powerful and open-source-based models that meet the specific requirements of different use cases. Available models include:

• neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8: A highly developed language model based on the Meta-Llama architecture, optimized for precise and contextual instructions.
• neuralmagic/Mistral-Nemo-Instruct-2407-FP8: Another powerful model that stands out for its compatibility with OpenAI standards.
• intfloat/e5-mistral-7b-instruct: A model particularly suited for embedding tasks and offering flexible application possibilities.

In the future, the portfolio will be further expanded so that even more models will be available to meet growing requirements.

Using STACKIT AI Models with the OpenAI SDK

To practically demonstrate the performance of STACKIT Model Serving, we would like to present a simple example that shows how to create a simple “Hello World” program with the OpenAI SDK. This example is API-compatible with OpenAI SDKs and is also suitable for less technical users.

Prerequisites

Before you start, make sure you meet the following requirements:

1. Python installed: Make sure Python is installed on your system.
2. API key from STACKIT: You need a valid API key from STACKIT, which you can generate in your account. As a STACKIT partner, innFactory is happy to support you with onboarding.

Step 1: Set Up Project

First, create a new directory for your project and change to this directory:

`mkdir stackit-hello-llm

cd stackit-hello-llm`

Step 2: Install Packages

Create a file named requirements.txt with the following content:

OpenAI

Install the required packages with pip:

pip install -r requirements.txt

Step 3: Create Python Script

Create a file named hello_llm.py and insert the following code:

import os from OpenAI import OpenAI client = OpenAI(api_key=os.getenv("STACKIT_API_KEY"), base_url="https://api.OpenAI-compat.model-serving.eu01.onstackit.cloud/v1") def list_models(): response = client.models.list() return response def create_completion(model_id, prompt): response = client.completions.create(model=model_id, prompt=prompt, max_tokens=50) return response def main(): # Step 1: Retrieve list of models models = list_models().data print("Available models:") for model in models: print(model.id) # Step 2: Perform test call with "Hello World" model_id = "neuralmagic/Mistral-Nemo-Instruct-2407-FP8" prompt = "Hello World" print("Test call with model:", model_id) completion_response = create_completion(model_id, prompt) print("Response:\n", completion_response.choices[0].text) if __name__ == "__main__": main()

Step 4: Set Environment Variable

Set your STACKIT API key as an environment variable. This may vary depending on the operating system.

For Unix/Linux/macOS:

export STACKIT_API_KEY='Your_Stackit_API_Key'

For Windows (Command Prompt):

set STACKIT_API_KEY=Your_Stackit_API_Key

Step 5: Run Script

Run the Python script:

python hello_llm.py

You should see a list of available models and receive a response from the LLM responding to the “Hello LLM!” prompt.

Code Explanations

• Importing Libraries: The code imports the necessary libraries, including os for environment variables and OpenAI for interacting with the STACKIT API.
• Client Initialization: An OpenAI client is initialized with the API key and BASE_URL from STACKIT.
• Model Listing: The list_models function retrieves and returns the available models from STACKIT.
• Creating a Completion: The create_completion function sends a prompt to a specific model and receives a response.
• Main Function: When running the script, the available models are listed and a test call is made with the “Hello LLM!” prompt.

This simple example shows how easy it is to work with the OpenAI SDK and STACKIT Model Serving.

innFactory as an Official STACKIT Partner: Competent and Committed

innFactory is an official partner of STACKIT and supports companies in optimally leveraging the benefits of the STACKIT Cloud and Model Serving. As an experienced AI consultancy, we offer comprehensive advice and technical support to plan and develop customized AI solutions that meet the individual requirements and security standards of your industry.

STACKIT Model Serving – A Powerful and Sovereign Solution

STACKIT Model Serving is a powerful and simultaneously data-secure solution for companies that want to use Artificial Intelligence efficiently and independently. Especially in highly regulated industries, STACKIT offers a unique combination of technical excellence and complete data sovereignty that is often not achievable with global providers AWS, Google, and Azure. With the available models and continuous expansion of the portfolio, STACKIT Model Serving is suitable for a variety of use cases that have high security and compliance requirements.

By working with innFactory as your official STACKIT partner, you benefit from comprehensive expertise and customized solutions that strengthen your digital sovereignty and position your company for the future. Discover the possibilities of STACKIT Model Serving and contact us for more information. Together we lay the foundation for an innovative and independent digital future for your company.