two nasty production problems, a hypothetical solution, and next steps

alex 4th August 2024 at 11:55am

The very last few problems I had to solve in alpha deployment (there are still a few things that must be taken care of!) were two very pesky situations that almost embarass me; for the sake of transparency they shall be disclosed, along with what I believe would have allowed for a quicker fix of both issues.

Issue number one: production server was not displaying student content. This happened because the static student content lies outside the project directory, and it did not occur to me that (of course) the container would have on access to outside; this was fixed quickly by mounting some files and directories when running the container:

podman run [...] --mount type=bind,source=THROVE_LOCAL_DIR,target=THROVE_CONTAINER_DIR [...] localhost/my-django-app:latest

...but the time it took me to realise this was the issue? Let's not mention that.

Issue number two: after properly displaying the files on the front-end, the (secure) serving of the files was not working; 404 errors all over. Now, in this case I had one very useful error appear at the beginning of the debugging process, but I did not find it for lots of time after — I suppose some git magic happened, reverting some important changes; and I spent some time trying to understand why my nginx was failing, while wholly ignorant of the fact that the backend was not routing properly. So that was very embarassing too.

Common culprits? Aside for a lack of proper testing — I might have stopped at the unit level, and some higher level of integration tests could have helped —, the absence of logs really do feel like the biggest flaw. And so, lesson (hopefully) learned. My logging is properly setup now, and it should help me shorten the feedback loop between hammering code and analysing the results.


Anyway, as of now, the application is live and some students already have access to it. There are a few problems: the front-end must be improved, logging in takes long, file serving must improve; these shall be addressed as time goes by. The important thing is the MVP is out and (mostly) functional; it has a real-life purpose; it brings me some joy. And I'm quite happy? So that's it, for now.