Are there any actual rules of how to write music? I mean there're guides everywhere in the Internet like "so you have scale, shapes of scale, chord progression over scale - let's go, do whatever you want with it until it sounds good". But of course it's not sounds good. You can play random notes and it just not sound out of place, but that's all. There's nothing specific about how to chose next note in riff, which chord better suit to chord progression, how to chose between different chord types etc. Or I just don't know what exactly to look for. So the question is there any real rules or should I experiment with things until I find what suits to me because it's the only way? And if there is, what should I read about?
In the sense that music is art, there aren't rules. There are no music police who will put you in jail for parallel fifths.
But, to the extent that musical art is about style, there are rules and conventions within styles. For example, you avoid parallel fifths in chorale harmonization, but in heavy metal parallel fifths are de rigueur.
what should I read about?
You should read music scores or song lead sheets, or whatever notation is used for the styles of music that interest you. The music itself is the primary textbook.
You didn't mention a particular style, but you used terms like riff and chord progression so I have a general idea. For basic technical knowledge get a book about how to read staff notation and a book on music theory. That will give you a technical, musical vocabulary. If you can, try your local library first to peruse many books before investing in purchases. Music books are sometimes expensive.
...do whatever you want with it until it sounds good...play random notes...there's nothing specific about how to chose next note...
Really?!? That's someone's songwriting advice?!? It sounds like a recipe for lots and lots of wasted time!
Most musical styles are about patterns and not randomness. Those patterns are the "rules." Randomness as a songwriting process is either a specific process, like cut-ups, or randomly groping around, because you simply don't know the patterns. Don't justify the latter through appeals to the former.
So the question is there any real rules or should I experiment with things until I find what suits to me because it's the only way?
There are rules. Do with them what suits you. Follow them, break them, extend them, re-write them, make new ones, etc. etc. Random guess work is not the only way. In fact, I would say this instead: how did the great musicians, the innovators you admire learn? I'm sure you will find most of them studied (formally or informally), applied common patterns (rules), and then made incremental changes to what they learned.
There are no hard and fast 'rules', unless you want to write a fugue; waltzes need to be 3/4, but as far as writing something is concerned, I can't see anyone picking up a 'rulebook' and going systematically through them, applying as many as possible!
Knowing scales, chord families, rhythms, and listening are all good things, particularly as a starter. Any 'rules' that exist came from 'that sounded rubbish - don't use it again', to 'that sounds good - use it again'. Most people's first efforts end up in the bin. So don't expect miracles - yet.
Music is not merely a sequence of disconnected tonal events placed in time (at random) is the a temporal painting (I've heard this quote attributed to other musicians, like Zappa), it is the cumulative sum of all the events that have occurred for the duration of the song. As such, even if you had a set of rules, without the understanding what a song is the rules won't help you.
Having a mathematical background I like to use the analogy of derivatives. Music is not so much about what has been played at t = 1, and t = 3, and etc., about the "change" in time from 1 to 3, etc. It's the derivative that counts. Another comparison might be a movie. A movie tells a story and it is not a bunch of things that happen. In fact the viewer cannot really appreciate the story until it is all over and they replay it in their mind to "see" the juxtaposition of this or that plot twist to how it was set up an hour ago. Music works in a similar manner. We set up cycles, and melodic themes, and through that build a background in the listener's ear. Even the untrained listener will remember what they heard, maybe even be able to hum it. So when you solo or improvise you want to draw on the themes that were presented to the audience, at least a little bit, before going down the rabbit hole of free exploration. It takes time and training to understand this fully. The best training is listening. Listen to music you like and try to identify patterns. The opening lick of Ornithology bears a striking resemblance to Einen Kleinen Nacht Musik by Mozart (perhaps more so after some lubrication), the group of notes is the same.
Yes, there are rules of a sort in music but they are not like rules in other disciplines. I wrote about this in another stack exchange question. Artists study light, shading, color, and perspective, all very geometric, mathematical, and physics based elements of art. But the artist is not bound by any "rules" to use them in a specific way. Look at Picaso, Pollak, etc. The education is there, the expression is unbounded. Music has similar structures in it that are to some extent based in the physics of sound and vibration and physiology of how our ear works (and our brain too). And yes it is true that movements of chords like V7-->I or IV-->I create certain feelings that ii-->vi would not create, or that it makes sense to use the C major scale over a C maj chord but at the end of the day none of that is a rule in the sense that you are looking for.
There are no rules to creativity. There are guidelines (which you have found and seem to frustrate you) and there are musical styles in which certain rhythms, grooves, changes, and melodies are favored. It is culture more than anything that sets up rules. I would recommend taking the time to learn some of these guidelines and try to identify them in action in music you listen too.
Classical Western Harmony theory is very much worth learning. Starting perhaps with multi voice homophonic harmony, i.e. where all voices follow that same rhythmic expression in time but sing or play different notes. This is heard in choirs and guitar chord melodies. When it comes to composing you may want to familiarize yourself more with the genre of music in which you want to compose. Composers are often called upon to write music in a variety of styles. Learn what's common among the styles and what distinguishes them. One hint in this regard might be that they all use the I, IV, V progression in some fashion or another but have different phrasing or inflection, or different tempo. When it comes to improv, especially in Jazz, a particular school of thought has emerged that teaches one to match a mode to each chord in a progression. This is grossly misunderstood by beginners as an indication that (1) only those notes can be played over those chords and (2) all those notes must somehow be played over those chords, neither of which makes sense. In reality the process of playing through these scales while the chords drone in the background (like on a Jamey Abersold album) is to acclimate one's ear to the harmonic texture present within those scales and how they sound against the chord. In fact one can do this with the chromatic scale too. A better approach, advocated by Jerry Coker, is to keep a journal of licks and phases that you think sound cool. He recommends thinking of 5 new licks a day for some extended period of time. Then work on fitting these lick into tunes, perhaps right into the melody. After a while you've rewritten the song, and that's a solo. Improv is variation on a theme so it stands to reason that if you're soloing over someone else's tune the best place to get ideas is the song itself.
On the topic of fitting a scale or mode over each chord there is a classic example of a scale played over a chord where one has "out of tune notes" and it works great, the Blues. The chords of the blues are not in one specific key, and the scale used is not one of the modes that would classically fit over any of them. That's all the proof I need that the rules were meant to be broken. What counts is setting up the musical story, like a movie, where the lick you are going to play in a minute is set up by the lick you're playing now. To this end I don't mind telling you my favorite starting notes over a major 7th chord are the #5 and b9. I'm not going to drone on those notes (maybe a little) but they will lead to a melodic idea that is dark and heavy, but eventually resolves.
"But of course it doesn't sound good... You can play random notes in a scale and they just sound compatible, not good." (I paraphrased a little. :D )
Sounding "good" is somewhat relative. We can probably agree on songs that we'd both consider good or bad, but there will also likely be differences.
I think a lot of folks are internally driven by a compass, pointing in the direction of what's "good". I think it's an important thing to have and one should listen to it, but keep in mind it's also subject to influence and can change over time.
Learning your favorite songs or how things that sound "good" are played will help improve your relationship with that compass. However, exploring music, breaking things apart, gluing pieces back together in different ways and/or changing them altogether are also fruitful parts of that process. You might even be doing this already when you hear songs that are 'almost amazing/perfect' (or that you don't like), just making substitutions left and right in your mind for how a lyric or melody 'should' go.
The answer so far has been about musical expression and developing an inner voice because I think that's likely what's underlying the question. For the question itself though, I agree with others, "The code is more what you'd call guidelines than actual rules." Due to this, the writer often still makes musical decisions of what is "good" in these systems.
Also, there's not really a single "code", as the thought processes and motivations of the people that practiced their style of music during their time are often considered. For example, chapter 2 of Kennan's 'Counterpoint' is about how to write "good" melodic lines (for counterpoint). A modern take on creating and adding a melodic line might be something like Friedland's 'Building Walking Bass Lines'.
So the question is are there any real rules or should I experiment with things until I find what suits me because it's the only way?
I would say there are guidelines to help your musical decision making that are accessible, but they are not really rules intended to make those decisions for you. It is very fruitful to learn pre-written things like scales/shapes/songs AND you'll want to try experimenting with those things and make your own musical expressions (improvise on top of songs, use scales to come up with melodies that sound and feel "good", learn different small licks and try to put them together (glue them by adding additional notes or chords or some musical expression if they don't fit together at first), etc...)
And if there is, what should I read about?
Before purchasing books and things, you might be interested in sections 6,7, and 9, of this music theory site which are for 'Triads', 'Roman Numerals and Cadences', and 'Harmonic Progression and Harmonic Function'. I imagine the takeaways are to get a sense of how chords are built, how they relate to scales, where the harmony of the 'Roman Numerals' comes from, and some examples of popular progressions. At a least, the site as a whole looks like it has some breadth and is skimmable, so maybe poke around and familiarize yourself with the notions/terminology there to help with future searches.