Problem with parallelized LaTeX runs after FNDB refresh

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Problem with parallelized LaTeX runs after FNDB refresh

Patrick Lainer
Dear MiKTeX users,

I have come across a peculiar problem with multithreading: right after  
refreshing the FNDB and rebuilding formats, calls to TeX & friends  
fail when executed in parallel because each process attempts to  
refresh .fndb files residing in the UserData directory tree (the  
output of initexmf --report is attached as report.txt) at the same  
time. Subsequent runs are not affected. Here is an almost minimal  
working example Windows batch file to reproduce the problem:

@echo off
call :timestamp clean up from the previous run
if exist "test*.*" del "test*.*"
call :timestamp refresh MiKTeX
initexmf --admin --verbose --update-fndb --dump=lualatex 2>nul
initexmf --update-fndb --verbose --dump=lualatex 2>nul
luaotfload-tool -vvv --update
call :timestamp generate test batch file
echo @echo off > test.bat
setlocal EnableDelayedExpansion
for /L %%i in (1, 1, %NUMBER_OF_PROCESSORS%) do (
   :: affinity for a processor core has to be given in hexadecimal
   call cmd /c exit /b %%i
   set hex=!=exitcode!
   echo start /max /affinity !hex! cmd /k lualatex  
--interaction=nonstopmode --jobname=test%%i  
\documentclass{article}\begin{document}Test\end{document} >> test.bat
)
endlocal
call :timestamp first round of tests after refreshing MiKTeX
call test.bat
pause
call :timestamp second round of tests after partially failed first round
call test.bat
goto :EOF
:timestamp
:: log time for comparison with log files
echo.
echo -- %DATE% %TIME% -- %* --
echo.
:EOF

After the first round of tests, new shells pop up with some of them  
showing an error. Apparently it's a statistical process -- in my case,  
0 to 5 out of 8 will fail, so maybe you have to run the script more  
than once. The error messages point to the log files for more details,  
so I attached the relevant sections of lualatex.log and initexmf.log.  
As said before, no errors appear in the second round without an  
intervening database refresh. A sample output of my script  
corresponding to the log files is attached as output.txt.

Note that luaotfload-tool is not relevant and only shows up for the  
sake of completeness. I first noticed that it sometimes calls initexmf  
when no FNDB refresh was made beforehand. I also inspected the access  
to files with Process Monitor from Windows Sysinternals and noticed  
that on the first try each process attempts to lock the .fndb file for  
writing at the same time.

I'm not entirely sure about the source code, but the function  
Application::Init in  
https://github.com/MiKTeX/miktex/blob/master/Libraries/MiKTeX/App/app.cpp#L139 
looks like every MiKTeX executable checks the FNDB and refreshes it  
when necessary. It appears that after my manual call to initexmf the  
executable called next refreshes the FNDB again anyway.

I've had a lot of trial and error to arrive at the above example and  
just now found a workaround: a "sacrificial" call to lualatex after  
the refresh and before parallelization will repair the FNDB without  
the conflict inherent in parallelization. Please correct me if I made  
a mistake; otherwise it would be interesting to know why the call to  
initexmf doesn't work as expected.

Best regards,
Patrick


--
Basisgruppe NAWI Physik (http://physik.htu.tugraz.at)
Referat für Bildungspolitik (http://bipol.htu.tugraz.at)
Curriculakommission Bachelor/Master ([hidden email])
Technische Universität Graz


------------------------------------------------------------------------------

_______________________________________________
Q: How can I leave the mailing list?
A: See http://docs.miktex.org/faq/support.html#leavingml

output.txt (3K) Download Attachment
report.txt (682 bytes) Download Attachment