Operating multiple containers with kubernetes for job processing

Currently, I am facing a challenge with processing multiple files in s3 using k8s. To address this, I have set up a job on k8s that consists of around 500 containers, each with different environments. However, the job is running very slowly and has failed multiple times.

I am utilizing the Kubernetes API Python to submit the job as follows:

Provided code snippet...

I have attempted to configure parameters such as completions=10, concurrency=2, but with my current setup, the job executes multiple times resulting in 10*500=5000 executions.

Can anyone suggest a more efficient way to create a job on k8s with multiple containers?

  1. 1 Job -> 1 Pod -> 500 containers
  2. 1 Job -> 500 Pods (Each pod 1 container).
  3. Are there any other alternatives?**
pythondockerkubernetes

Answer №1

Your query may have some subjectivity to it, as there can be various solutions based on your specific circumstances and approach. One insightful point made by David Maze is worth considering:

If you already have the scripting infrastructure in place, why not consider using 500 Jobs? (In general, it's advisable to avoid multiple containers per pod.)

If you are developing a script, this could prove to be an effective solution. It doesn't need to be overly complicated - simply create one Job for each pod with one container in each.

Alternatively, you could potentially define your own Custom Resource and build a controller akin to a Job controller that supports multiple pod templates.

For further insights, check out this related inquiry.

Similar questions

If you have not found the answer to your question or you are interested in this topic, then look at other similar questions below or use the search

The HighChart export server is unable to generate static images when using the Docker file for exporting

My current setup includes OS-Windows 11 Enterprice, NodeJS 14.0.0, NPM 6.14.4, and Highchart export server 2.1.0 I am attempting to generate/export static images through Docker. Here is my Docker File: # Pulling image from node FROM node:carbon ENV ACC ...

node.jsdockerexpresshighchartsserver

Python Selenium, Difficulty encountered with the find_element_by() function

Could someone please help me with this issue? I'm encountering errors with the find_element_by_() commands in my code and have tried reinstalling everything, as well as changing the Python version. Unfortunately, nothing seems to be working. Any sugge ...

pythonseleniumselenium-webdriverselenium-chromedriverselenium4

Is it possible to customize the text in a label by tracking mouse movements in Python using Tkinter?

I am having trouble updating my label based on the position of the mouse. Currently, it only works once when I run the code. How can I make sure the label updates every time my mouse changes position? The following code snippet is what I have so far, but i ...

pythonuser-interfacetkinterlabel

In the latest version of Qiskit 0.24.0 tutorial on Max-Cut and Traveling Salesman Problem, learn the method to tackle the TSP problem involving more than three nodes

I recently experimented with qiskit's Traveling Salesman Problem example, starting with 3 nodes and running it on IBM's simulator_statevector backend. The execution went smoothly and the results were as expected. However, I decided to challenge ...

pythonquantum-computingqiskit

To resolve the SyntaxError in matplotlib when trying to swap axes in a 3D wireframe plot, make sure to specify named arguments instead of using expressions

Using Python 3.4.3 and unable to update the system, I encountered an issue while trying to create a 3D wireframe plot using matplotlib and mpl_toolkits.mplot3d. ax.plot_wireframe(*a,b, rstride=1, cstride=2) >> SyntaxError: only named arguments may f ...

pythonmatplotlibplotpython-3.4argument-unpacking

Mastering the art of typing authentically like a human using ActionChains' key_down and key_up

I am developing an online automation tool. However, this specific website evaluates the user's typing speed, causing issues with input commands. The error message is triggered by two factors: the timing of key presses and releases. This means that th ...

pythonselenium

Accessing a webpage using python requests to authenticate users

I'm attempting to access a website using python 3, requests, and lxml for logging in. However, despite sending a post request to the login page, I am unable to access the pages that are meant to be visible after logging in. What could be the issue? i ...

pythonpython-requestslxml

Python ModuleNotFound error when invoking Python code from system() function in R

I have a scenario where I have an R script running on my Raspberry Pi that calls a Python script using the system() function. Let's call this R script tweetsomething.R # first some data cleaning etc. system("python3 tweetImage.py var1 var2" ...

pythonrpython-3.x

Enhancing Django's login form validation with AJAX

I'm trying to implement an AJAX login feature in Django. The goal is to check if the user has provided the correct username and password. Take a look at my current code setup: urls.py urlpatterns = [ url(r'^$', views.home_view, name=&a ...

javascriptjquerypythonajaxdjango

The compatibility between Selenium Chromedriver and PHP Python is not functioning properly

I've been working on a Python code to take screenshots of websites. import time from PIL import Image from selenium import webdriver options = webdriver.ChromeOptions() options.add_argument("--no-sandbox") options.headless = True driver = webdriver. ...

pythonphpseleniumselenium-webdriverselenium-chromedriver

The functionality of hatches in Matplotlib cycler is not functioning properly

Consider the sample code snippet below: %matplotlib notebook import matplotlib.pyplot as plt from cycler import cycler bar_cycler = (cycler(color=["#E69F00", "#56B4E9"]) + cycler(hatch=["/" , "o"])) p ...

pythonmatplotlibplotgraphvisualization

Encrypt Data Using RSA Public Key

Imagine I've created a private and public key RSA pair using the following command: ssh-keygen -t rsa -C "<a href="/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="0f76607a7d506a626e66634f6a216c6062">[email protected]</a& ...

pythonrsa

retrieving information from the surrounding context within a class

In an attempt to avoid rewriting a new function for each case, I am interested in using DeleteView for various scenarios. My idea is to pass the model as an argument in the URL and then select which delete operation to execute by customizing the get_contex ...

pythondjangoview

selenium and behat: Provider not found for session

I'm currently working on setting up selenium in a docker container to use with Behat. However, when I check the status of the hub at http://localhost:4444/status, it shows that it is not ready: { "value": { "ready": false, ...

dockerseleniumsymfonybehat

The connection between NodeJS and a postgres database is not functioning properly

Here is my app.js file: var express = require('express'); var app = express(); var uuid = require('node-uuid'); var pg = require('pg'); var conString = process.env.DB; // "postgres://username:password@localhost/database& ...

node.jspostgresqldockerdocker-composedockerfile

What are the steps to switch end-of-line conventions?

My task seemed simple - create a text file using the Unix LF convention at the end of each line. However, when I use pandas .to_csv to accomplish this, it defaults to CR LF instead of just LF. This becomes problematic as my code needs to be used by a cowor ...

pythonpandasnewlinepython-2.xlf

When a node or edge is clicked in Cytoscape, display its attributes in a table format

Hello, I am new to using cytoscape.js and currently working on rendering shallow networks in a Flask App. At the moment, I have managed to display node and edge attributes when hovering over them, but the information is shown in a format that is not very ...

pythonflaskcytoscape.js

How can I generate a PDF file directly from byte data in Python without having to write it to a binary file first?

This Python code retrieves data from a database table and converts it into a PDF file without saving it to disk, then displays it in the browser. l_statement = "select srno,attachment from dbo.attachments where srno = 1" sqlcursor.execute(l_stat ...

pythonpdf

Encountered an issue in Docker for Windows OS: "ERROR: Unable to retrieve https://gcr.io/v2/: x509: certificate signed by unknown authority"

I'm in the process of setting up an application that has Python and GraphQL as the backend, along with Redis. While trying to build Nginx using ' docker-compose --profile backend --profile frontend up --build ' it keeps pulling Redis but ...

pythonnode.jsdocker

python Using urllib2 to retrieve JSON data

I need help retrieving a JSON object from a specific URL. I've provided the correct API key, but when I attempt to do the following: data=json.loads(a.read()) print data I encounter this error: Traceback (most recent call last): File "C:\Pyt ...

pythonjson