For the past year, I've been successfully running a python script within a Docker environment without any major issues.
However, after updating to Docker version 4.25.0 about a month ago, my python script stopped working properly.
I managed to fix several problems, but I am struggling with this persistent file access issue!
Below is an excerpt of the problematic code:
for f in os.listdir(taskOutputFolder):
if not f.startswith('.'):
if f.endswith('.zip'):
print("\n")
print("***************************************")
print(f"Zip File Found! {f}")
print("***************************************")
src_file = self.fileCheck(taskOutputFolder, f)
dest_file = os.path.join(nextTaskInputFolder , f)
# Copy File To Task's Output Folder
print(f"Copying file {f} from {taskOutputFolder} to {nextTaskInputFolder}")
try:
shutil.copyfile(src_file, dest_file)
print(f"Successfully copied {f} to {nextTaskInputFolder}.")
except Exception as ex:
print(f"Failed to copy {src_file} to {dest_file} with error:")
print(ex)
Prior to the above code, I utilize:
with ZipFile(zip_out_filepath, mode="w", compression=ZIP_DEFLATED, compresslevel=9) as archive:
for file_path in Path(zip_in_filepath).iterdir():
archive.write(file_path, arcname=file_path.name)
To generate a .zip file, and during these functions, I verify the existence and accessibility of files using the following function:
def fileCheck(self, fileFolder, fileName):
file = os.path.abspath(os.path.join(fileFolder, fileName))
print(f'Checking File Status for {file}...')
# Check for bad file naming characters and remove any whitespace if found
badFileNameChars = ['(', ')', ' ', ',', '\t']
res = any(ele in fileName for ele in badFileNameChars)
if res:
print(f'Found Bad File Name Characters In: {fileName}')
file = self.cleanFileName(file)
# Check If File Copy Was Successfull
retries = 0
file_found = False
while not file_found:
if not os.path.exists(file):
retries += 1
print(f'\nWaiting For File To Exist: {file}...')
print(f'Retry: {retries}...\n')
time.sleep(5)
if retries > 6:
file_found = True
else:
file_stats = os.stat(file)
print(f'File Found: {file}')
print(f'File Stats: {file_stats}')
file_found = True
return file
To ensure that the file exists and is accessible...
Upon executing the script, I encounter the following error:
Checking File Status for /mnt/preprocessing-out/preprocess-pipe-local.zip...
File Found: /mnt/preprocessing-out/preprocess-pipe-local.zip
File Stats: os.stat_result(st_mode=33279, st_ino=60, st_dev=40, st_nlink=1, st_uid=0, st_gid=0, st_size=380537, st_atime=1700345226, st_mtime=1700345215, st_ctime=1700345215)
Checking Zip Output Again For Delayed Completion Check...
Checking File Status for /mnt/preprocessing-out/preprocess-pipe-local.zip...
[Sat Nov 18 22:08:16 2023] [CRITICAL_FAILURE] - Missing output file after running ZIP Function
The code snippet leading to the aforementioned error is here:
with ZipFile(zip_out_filepath, mode="w", compression=ZIP_DEFLATED, compresslevel=9) as archive:
for file_path in Path(zip_in_filepath).iterdir():
archive.write(file_path, arcname=file_path.name)
zip_out_filepath = self.fileCheck(Path(zip_out_filepath).parent, os.path.basename(zip_out_filepath))
print('Checking Zip Output Again For Delayed Completion Check... ')
time.sleep(5)
zip_out_filepath = self.fileCheck(Path(zip_out_filepath).parent, os.path.basename(zip_out_filepath))
assert os.path.isfile(zip_out_filepath), 'Missing output file after running ZIP Function: '
I have tried various methods to resolve the file accessing issues mentioned above, including switching from command line zip functions to Python ZipFile module functions, renaming files, etc., all to no avail.
This dilemma seems linked to the Docker update to version 4.25.0, which is what changed when the problem arose. Since my expertise with Docker is limited, I am uncertain how to troubleshoot effectively. Any guidance towards a remedy or insights into Docker changes causing random file access errors would be greatly appreciated.