It would be good to share some technical details on the servers, the auth, the databases etc.
for example the rocketchat application crashed this morning and i had to figure out how it was running internally so it could be restarted. i would like to write this down somewhere.
was wondering where this should go ? it definitely is meta , probably a wiki ? i also considered putting it under staff - which makes sense - although it doesnt need to be private - would rather anyone who is interested could peak behind the curtain…
also could be a problem if the guide to restarting the discourse application is on the discourse application …
at 9am here (around when i woke up and maybe? opened the app?) the server started reading around 150mb every minute for 20 minutes. which kept the cpu more busy than usual. then the memory falls to 20% which is when the app crashed i guess. i restarted the vm and then it crashed again straight away.
i then upgraded to a larger machine and restarted - which worked and is why the memory is on 40% rather than 80% now. maybe it crashed due to high memory usage, but not sure what the root cause is. one thought is that the haircut image i uploaded directly from my phone yesterday was probably pretty big. maybe it got in some kind of loop serving this image back to me in the morning…
didnt dig into any logs, but will keep an eye on this. maybe run some more experiments uploading images etc. if this is the problem we could also put some file size limit there maybe
word, i hope it was a fluke and/or that the upped server specs help. rocket chats logs seem pretty good too, they can be surfaced thru the admin UI as well and you can change levels of verbosity. and thanks for documenting the issue! so that way hopefully if it happens again i or someone else can help restart it too.
edit: looked into it and i guess the logs that appear in the admin panel dont actually get saved to the disk anywhere by default…so i guess the ones from before the crash are gone now. it looks like at least some of it ends up in /var/log/syslog tho, and i do see “out of memory” messages in there.
