In older Portuguese grammars gender was sometimes considered an inflection; but that is a (over)simplified model, because it's a requirement of inflection that it be regular -irregularity being an exception to the rule- and gender is not regular.
A complete (modern) model of gender in Portuguese morphology would start by introducing it as one of the grammatical categories which are:
Having said that, lets introduce the technical term gender contrast before we look at the possible groupings for gender as a grammatical category:
A. Without gender contrast (not morphological nor syntactic)
1. Neutral gender
A rare group in Portuguese, most obvious examples are from the closed (function) group of words:
- "isto" and "aquilo" (not masculine nor feminine, nor do these words allow gender contrast).
B. Syntactic gender contrast (not morphological)
2. Sobrecomum
Words that have one gender but do not allow gender contrast, they only admit a determinant of one gender in front of them:
a testemunha (as an adjective, preceded by the singular feminine determined article) . You can't say "o testemunha" , preceded by the singular masculine determined article; and if you were to say "o testemunho" it would no longer be an adjective but a noun - but, what's more, it would be a different lexeme because altough sharing the same radical (testemunh-) it would have a different semantic meaning).
The same as in the previous example for um individuo that can't be said uma individuo nor uma individua).
3. Comum de 2 géneros
Words that have no determined gender by themselves but allow gender contrast to be made by the preceding determinant, as in:
C. Morphological gender contrast
4. Epiceno (gender contrast through compound)
Words that only have one gender, e.g. (o crocodilo, not a crocodilo) but allow gender contrast through compound (composição) notice that's it's not by inflection nor by derivation because the right side is itself a word and not a suffix:
- crocodilo-macho
- crocodilo-fêmea
5. Words that allow gender contrast through índice temático
-
- falador+∅ (m)
- falador+a (f)
-
-
6. Words that allow gender contrast through derivation
-
-
- profet+a (m)
- profet+isa (f)
-
-
-
- impera+dor (m)
- impera+triz (f)
-
-
-
- perdi+gão (m)
- perd+iz (f) (the only example of this contrast in the portuguese lexicon)
Notice that all the suffix pairs in the above lists are not repeated.
Having written the above, it becomes easy answering the examples in the question:
But how do I know when something is addressed as "female" or "male", only by the last letter of the word?
You don't and you can't. Unlike inflection (be it verbal or nominal) you can't deterministically deduce from the suffix what the values of the grammatical categorias are, for example in the imperfect tense 1st person plural of the 1st conjugation (1ª conjugação do pretérito imperfeito na 1ª pessoa do plural) of the verb:
knowing it's a verb, you can immediately segment it and identify the value of each part:
The segments being deterministic allow us to deduce:
- fal - the radical.
- á - the vogal temática of the 1ª conjugação (-ar) with a diacritic.
- va- the tense-mood-aspect morpheme allows to know it's pretérito perfeito.
- mos- we immediately know it's plural and 1ª person.
But as previously shown, gender contrast aren't produced by a single word formation or inflection process. Instead they are produced by several (compound, derivation, índice temático contrast and lexicalization). What's more, sometimes the contrast is established on a morphological level, other times on a syntactic level, and lastly in the lexicon. Which takes us to your next question:
How would I identify a word ending with E or O for example?
The main idea of @tchrist's post above can be boiled down to a single word in linguistics: it's lexicalized. Meaning the gender information can't be deduced from any process or context but must be retrieved from the lexicon together with the lexeme itself.
However, you can establish "classes temáticas" depending on the word endings' "constituinte temático" . The general notion of "constituinte temático" is divided into "vogal temática" (-ar, -er, -ir) for the verbs, and "índice temático" for the remainig Parts-of-Speech. The division of the "índice temático" into "classes temáticas" depending on the ending can be seen in chapter 18.2. Classes temática of Gramática da Língua Portuguesa.
References:
Villalva, Alina. Estruturas morfológicas: unidades e hierarquias nas palavras do português. Textos universitários de ciências sociais e humanas. Lisboa: Fundação Calouste Gulbenkian Fundação para a Ciência e a Tecnologia. (2000)
Inês Duarte, Isabel Hub Faria e M. Helena Mira Mateus, et al. Gramática da Língua Portuguesa. Editorial Caminho. 6ª edição (2003). (The revised Part V authored by Alina Villalva has been made available online.)