UPDATE: The latest RC version of Lemmy-ui (0.18.2-rc.2) contains fixes for the issue, but if you believe you were vulnerable, you should still rotate your JWT secret after upgrading! Read below for instructions. Removing custom emoji is no longer necessary after upgrading.
Original post follows:
This post is intended as a central place that admins can reference regarding the XSS incident from this morning.
What happened?
A couple of the bigger Lemmy instances had several user accounts compromised through stolen authentication cookies. Some of these cookies belonged to admins, these admin cookies were used to deface instances. Only users that opened pages with malicious content during the incident were vulnerable. The malicious content was possible due to a bug with rendering custom emojis.
Stolen cookies gave attackers access to all private messages and e-mail addresses of affected users.
Am I vulnerable?
If your instance has ANY custom emojis, you are vulnerable. Note that it appears only local custom emojis are affected, so federated content with custom emojis from other instances should be safe.
I had custom emojis on my instance, what should I do?
This should be enough to mitigate now:
- Remove custom emoji
DELETE FROM custom_emoji_keyword;
DELETE FROM custom_emoji;
- Rotate your JWT secret (invalidates all current login sessions)
-- back up your secret first, just in case
SELECT * FROM secret;
-- generate a new secret
UPDATE secret SET jwt_secret = gen_random_uuid();
- Restart Lemmy server
If you need help with any of this, you can reach out to me on Matrix (@sunaurus:matrix.org) or on Discord (@sunaurus)
Legal
If your instance was affected, you may have some legal obligations. Please check this comment for more info: https://lemmy.world/comment/1064402
It seems to me that the scope of this could have been mitigated with a simple privilege separation policy for admin server accounts but I see a lot of (what looks like) server admins using that account as their daily driver.
Also, lemmy-ui should post a security advisory to their github.
Welp, that is some simple lapse of separation-of-duties principles that shouldn’t have been.
I don’t pretend to know the demographics of lemmy server admins, but my gut says it’s predominantly hobbyists and devops types, rather than grizzled system admins.
People in DevOps should know better than reusing their accounts. It doesn’t take a seasoned system admin to know about basic OPSEC and separation amongst profiles.
Agreed. I’m pointing it out in hopes that becomes one of the takeaways of this incident.
(Unrelated, why are you marked at a bot account?)
Wait, why am I marked as a bot account?
This is the first time I’ve heard about this. Whom should I ask for help?
It’s a checkbox in the setting for your account. You should be able to change it.
Hi, could you check now?
I still see it.
Yes and no. Admin accounts often remain logged in as a practical matter. They can’t see incoming reports, registration applications, etc. if the account isn’t logged in. And there is no “middle tier”/sitemod or customizable permissions allowing for anything between community mod and instance admin that would mitigate the need to use admin accounts day to day.
I still wouldn’t use the admin account as my daily driver. Leave it open in another browser/private tab specifically to perform admin actions (as noted) but not for browsing/posting/community modding. I understand how that’s a pain but given how early days we are with this platform and the high probability of more issues surfacing, it’s a necessary pain.
I’d really like to see the devs add some tools to mitigate future risk and further protect admin accounts. The least of which being that admin actions require stronger validation than a browser side cookie, and frequent re-validation to perform admin actions.
i’ve thought about switching to a different account for non-admin tasks but I was a normal user here first and I want to keep using this account for normal posting too. I think using a different account is kind of pointless from a security standpoint, because performing mod and admin actions requires looking at user-generated content anyway and there is no way around that (except for maybe moving to a very clipboard-heavy workflow which would make everything take much longer).
There is work underway to fix these XSS bugs in a more comprehensive way (better CSP, and HttpOnly cookies…) which will hopefully land soon.
(i am not a lemmy developer but i’m reading the github and matrix chat…)
I mean. You could still be a normal user and have a separate admin account.
I get your point. But it’s a huge risk.
Using a separate account doesn’t substantially mitigate the risk. It might reduce the chances of getting randomly exploited, but it’s easy to post things admins need to see to do their job so any attacker wanting to target admins would still be able to even if we used separate accounts.
Systemically fixing the XSS problems is necessary either way :)
edit: actually I guess most common admin activities could be separated from the rarely used ones, at least… or the infrequent actions could simply require re-authenticating. That wouldn’t be a bad idea.
To be clear I’m not talking about separate mod accounts. I’m talking just admin.
that wouldn’t have necessarily stopped this attack I don’t think, but yeah, probably a good idea on multiple levels.
If the separate admin window was open, and a tagged reply or PM was sent to the admin account I think that would render the emote in the notification and trigger the exploit
Fair point. I know that I don’t have the answers. I do think that admin actions need to be more stringently scrutinized. Maybe something like a “sudo” model where your a normal acct 99% of the time and admin actions require a temporary elevation.