Don't feel bad about that. "Who" and "Whom" continue to challenge people who've spoken English since infancy; and more than a few people who actually teach English at the University level.
I will note that, for all its popularity as a de facto universal language, English can be an extremely complicated and often baffling language. Even for the native speaker. English is one of very few languages where you can have crossword puzzles that depend on a person's understanding and knowledge of a word's multiple meanings. For instance a recent NYT Crossword clue: "Figure in a dark suit", and its correct answer: "Spade" depend on the solver knowing that one of the many meanings of the word "Suit" (in addition to a set of clothing; a verb meaning to agree with or be appropriate to; and a legal action) - is that of a subset of a deck of playing cards. And that "Spade" - in addition to referring to a digging tool can also refer to a type of playing card.
How non-native English speakers deal with this, I don't know. I can tell you that German and French crosswords are, by comparison, relatively tame affairs, more on the lines of trivia quizzes.
On the other hand, English is a language that dispenses, almost entirely, with the baffling gendering of nouns that makes a German Mädchen (girl) neuter or a French wardrobe (garde-robe) feminine.
I applaud anyone who works to improve their English grammar.
I think I have been reading to many comic books lately, because the first thing I thought of was "Batman".