I run my own Lemmy instance at corndog.uk and recently migrated pictrs to object based storage. I didn’t have any problems until I attempted to upgrade from pictrs:0.4.0-rc.14 to pictrs:0.4.1. When I updated that container I noticed images were no longer loading in a new tab so in an attempt to fix it I restarted the lemmy and lemmy-ui containers.

After doing that my entire instance seems to be broken. Reverting back to pictrs:0.4.0-rc.14 doesn’t seem to work either and I can’t get the frontend to come back up at all.

In the lemmy-ui logs I get this error:

Error: Input bit Array is empty
    at Sharp._createInputDescriptor (/app/node_modules/sharp/lib/input.js:56:13)
    at new Sharp (/app/node_modules/sharp/lib/constructor.js:350:29)
    at Sharp (/app/node_modules/sharp/lib/constructor.js:173:12)
    at /app/dist/js/server.js:2:1191814
    at tryCatch (/app/node_modules/@babel/runtime/helpers/regeneratorRuntime.js:44:17)
    at Generator.<anonymous> (/app/node_modules/@babel/runtime/helpers/regeneratorRuntime.js:125:22)
    at Generator.next (/app/node_modules/@babel/runtime/helpers/regeneratorRuntime.js:69:21)
    at asyncGeneratorStep (/app/node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
    at _next (/app/node_modules/@babel/runtime/helpers/asyncToGenerator.js:22:9)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5

The pict-rs container doesn’t have any errors in it and I’m not sure if the errors I’m seeing in the lemmy container are relevant to my issue.

2023-07-20T21:43:52.935210Z  WARN lemmy_server::root_span_builder: Fetched remote object https://mastodon.vlaanderen/users/richrtwitt which was deleted
   0: lemmy_server::root_span_builder::HTTP request
           with http.method=POST http.scheme="https" http.host=corndog.uk http.target=/inbox otel.kind="server" request_id=eff855c7-474a-436e-8c98-ca1aae3e4de3
             at src/root_span_builder.rs:16
LemmyError { message: None, inner: Fetched remote object https://mastodon.vlaanderen/users/richrtwitt which was deleted, context: SpanTrace [{ target: "lemmy_server::root_span_builder", name: "HTTP request", fields: "http.method=POST http.scheme=\"https\" http.host=corndog.uk http.target=/inbox otel.kind=\"server\" request_id=eff855c7-474a-436e-8c98-ca1aae3e4de3", file: "src/root_span_builder.rs", line: 16 }] }

Can anyone help me troubleshoot or point me toward a better resource for getting help? This is the second time I’ve run into this issue (I’m taking regular postgres database backups now) but I can’t seem to find a safe way to keep my containers up to date without running into this issue and I’m not sure what I’m doing wrong.