Somewhere to share the technical details of maintaining this site?

It would be good to share some technical details on the servers, the auth, the databases etc.

for example the rocketchat application crashed this morning and i had to figure out how it was running internally so it could be restarted. i would like to write this down somewhere.

was wondering where this should go ? it definitely is meta , probably a wiki ? i also considered putting it under staff - which makes sense - although it doesnt need to be private - would rather anyone who is interested could peak behind the curtain…

also could be a problem if the guide to restarting the discourse application is on the discourse application …

any ideas ?

maybe we can create a github thing, and link to it from here. just a repo with .md docs would work altho i guess they have wiki features and stuff too

also, thanks for fixing the chat. i wonder why it crashed in the first place?

1 Like

github is a good idea - yes i think some markdown files would do. i will create this and link it back here.

regarding the crash - yeah it is weird:

at 9am here (around when i woke up and maybe? opened the app?) the server started reading around 150mb every minute for 20 minutes. which kept the cpu more busy than usual. then the memory falls to 20% which is when the app crashed i guess. i restarted the vm and then it crashed again straight away.

i then upgraded to a larger machine and restarted - which worked and is why the memory is on 40% rather than 80% now. maybe it crashed due to high memory usage, but not sure what the root cause is. one thought is that the haircut image i uploaded directly from my phone yesterday was probably pretty big. maybe it got in some kind of loop serving this image back to me in the morning…

didnt dig into any logs, but will keep an eye on this. maybe run some more experiments uploading images etc. if this is the problem we could also put some file size limit there maybe

word, i hope it was a fluke and/or that the upped server specs help. rocket chats logs seem pretty good too, they can be surfaced thru the admin UI as well and you can change levels of verbosity. and thanks for documenting the issue! so that way hopefully if it happens again i or someone else can help restart it too.
edit: looked into it and i guess the logs that appear in the admin panel dont actually get saved to the disk anywhere by default…so i guess the ones from before the crash are gone now. it looks like at least some of it ends up in /var/log/syslog tho, and i do see “out of memory” messages in there.

they will go here https://github.com/langolierz/scanlines-technical-details

2 Likes

How did you setup rocketchat? If it’s from snap it sometimes updates and then crashes. =D

it wasn’t set up through snap - but it does seem like it crashed itself when doing some kind of scheduled auto maintenance.

chat.scanlines.xyz