Quick search surfaced the following for Linux:

k3b, where the source repo states bluray burning capabilities.
xfburn also mentions bluray burning capabilities.

For Windows, albeit old and unupdated, I know the following still works for other purposes (never tried bluray burning/writing though):

ImgBurn mentions bluray burning/writing capabilities, but never tried it.

Bonus: not capable of bluray burning/writing but just fun to mention for any still into ripping/writing to discs on Windows:

InfraRecorder, simply a classic, and it’s open source!

if this isnt a troll, then you are one of a dying (pretty much never existed) breed

burning dvds or copying vhs tapes made sense years ago bc you needed a converter to plug your computer into a tv, and the quality was iffy. now most machines and tvs are compatible with hdmi, so going from digital to physical is like printing a mapquest route and taking a picture of it with your phone

edit: sorry i didnt provide a soln

There really isn’t any good software for making your own custom Blu-rays.

If you don’t care about menus and just want to put your .mkv on a Blu-ray disc, then you can use tsMuxeR to convert your .mkv file into a Blu-ray compliant ISO or BDMV folder structure.

Keep in mind you’ll need to use the proper video and audio codecs if you want it to properly turn your video file into an ISO - it’s recommended you use AVC/H.264 for video and AC3 (lossy) or TrueHD (lossless) for audio.

The term you’re looking for is “bluray authoring”.

Search for that. Industry tends to use Scenarist but it would be too expensive for an individual and they probably wouldn’t sell to you anyways. There are semi-pro tools, most paid, some hundreds of Euros.

Edit: here’s a list of some of the better ones with prices: https://www.videohelp.com/software/sections/authoring-bd-hd-dvd